DAMEX: Dataset-aware Mixture-of-Experts for visual understanding of mixture-of-datasets
Authors: Yash Jain
Harkirat Behl
Zsolt Kira
Vibhav Vineet
Venue: NeurIPS 2023 poster (2023)
Keywords: mixture-of-experts
moe
object detection
mixture of datasets
multiple datasets

Abstract: Construction of a universal detector poses a crucial question: How can we most effectively train a model on a large mixture of datasets? 
    The answer lies in learning dataset-specific features and ensembling their knowledge but do all this in a single model.
    Previous methods achieve this by having separate detection heads on a common backbone but that results in a significant increase in parameters.
    In this work, we present Mixture-of-Experts as a solution, highlighting that MoE are much more than a scalability tool. 
    We propose Dataset-Aware Mixture-of-Experts, DAMEX where we train the experts to become an `expert' of a dataset by learning to route each dataset tokens to its mapped expert.
    Experiments on Universal Object-Detection Benchmark show that we outperform the existing state-of-the-art by average +10.2 AP score and improve over our non-MoE baseline by average +2.0 AP score. We also observe consistent gains while mixing datasets with (1) limited availability, (2) disparate domains and (3) divergent label sets.
    Further, we qualitatively show that DAMEX is robust against expert representation collapse. Code is available at https://github.com/jinga-lala/DAMEX

DAMEX: Dataset-aware Mixture-of-Experts for visual understanding of mixture-of-dataset
Yash Jain'
Harkirat Behl²
Zsolt Kira!
Vibhav Vineet²
Georgia Institute of Technology
Microsoft Research
Abstract
Construction of universal detector poses a crucial question: How can we most effectively train model on a large mixture of datasets? The answer lies in learning sataset-specific features and ensembling their knowledge but do all this in single model. Previous methods achieve this by having separate detection heads on a common backbone but that results in significant increase in parameters. In this work, we present much more than a scalability tool. We propose Dataset-Aware Mixture-of- Exxture-ff-Expetts. DAMEX where we train the experts to become 'expert' of a dataset by learning to route each dataset tokens to its mapped expert. Experiments on Universal Object-Detection Benchmark show that we outperform the existing state-of-the-arl by average + +10,2 AP score and improve over our non-MoE bascline by average +2.0 AP score. We also observe consistent gains while mixing datasets with (1) limited availability, (2) disparate domains and (3) divergent label sets. Further, we qualitatively show that DAMEX is robust against expert representation collapse. Code is available at httttss: /tt///gtttut/..////////ee com jinga- www.o//iinaallllaaaaaaa lala/DAMEX
oftturreof-Experre as solution, highlighting that MoEs are
Introduction
Human visual perception has naturally evolved to excel at recognizing and localizing objects in a wide range of real-world scenarios 225 33. However, the field of computer vision has traditionally
concentrated on training and evaluating object detection models on specific datasets 42
3 43
20
28 34
These models have shown limited adaptability new environments [1
32 leading
to decreased accuracy and practicality in real-world settings [30 27 Recognizing the challenges
posed by the single dataset models and the availability of diverse datasets collected over time, unified models are being developed that merge information from multiple datasets into single framework
35 45
24 15).
The process of combining datasets presents several challenges that need to be addressed. Firstly, these datasets have been collected over time [5] and for different purposes, resulting in variations across domains. For example, one dataset may contain images from indoor environments [ while another may be focused on the medical domain [38]. Secondly, imbalances in training images and tail distributions may exist across the datasets. This means that certain datasets and classes might have substantial number of training examples, while others have only a few training examples. To illustrate, the popular Universal Objject-Dttcctinn
Benchmark (UODB) dataset for robust object detection demonstrates this diversity: the Clipart [13] data subset comprises 500 training images, whereas the COCO dataset (22) includes 35,00 55,000 training images. Figure I I]shows such diversities in the real world data, encompassing various aspects such as domains, the number of training images, camera views, scale and label set, taken from the multiple data sources. Can universal object detection method be designed that effectively addresses the above issues associated with mixing multiple datasets?
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Watercolor (Style)
Clipart (Style)
Comic (Style)
DOTA (Aerial)
Deeplesion (Medical)
KITTI (Traffic)
LISA (Traffic)
Widerface (Natural)
Kitchen (Indoor)
voc (Natural)
COCO (Natural)
Figure I: Samples from Universal Object Detection Benchmark (UODB): UODB set comprises of group of 11 datasets which vary in domains, labels, camera view and scale.
To tackle multi-datasets based object detection problem, several approaches have been proposed. manually unifies taxonomy of 7 semantic segmentation datasets using Mechanical Turk whereas our objective is learn a single universal detector on any mixture of datasets. (45) provides a way to combine multiple separate detectors by learning unified label space for multiple datasets. Their work proposed a new loss term to unify the label space but requires training separate partitioned detectors heads which are later unified with drop in performance. [5 trains universal detector on 11 datasets of Universal Object Detection Benchmark (UODB) and gain robustness by joining different sources of supervision. However, their method requires knowledge of the input dataset domain during test time whereas our objective universal detector should be agnostic to the input data domain making the problem more challenging.
[6
In this work, we propose Mixture-of- Experrs (MoE) based object detection method that can handle all the desired properties of a universal detector. Traditionally. MoEs have been used as scaling tool in Natural Language Processing 6
and image classification problem propose that Mixture-of-Expert are more than just scalable learners, instead MoEs are an effective and efficient solution to build a universal model for mixture of datasets. Building over vanilla MoF, we introduce novel Dataset-aware Mixture-of- -xperts model, DAMEX which learns to disentangle dataset-speciffc features at the MoE layer while pooling information at non-MoE layers. DAMEX learns to route tokens to their corresponding expert so that during inference it automatically chooses the best pathway for the test image through the network. We use DINO 42 based detection architecture for developing our approach.
18 31 4
11 29]. However,
We perform our evaluation on UODB dataset which comprises of 11 diverse datasets as shown in Figure I] We compare against previously reported state-of-the-ar on UODB [35] and recent multi- dataset object-detection method (55) Moreover, we also compare our method against individual dataset trained DINO detectors along with complete UODB trained DINO model. We report an average +10.2 AP score improvement over previous baseline and +2.0 AP gain over non-MoF
based method. Additionally, mix dataset pairs that highlight that MoE based detectors are better

---

in handling, (1) Mixing with limited data availability, (2) Domain Adaptation, (3) Mixing divergent label sets within same domain, and show consistent gains on DAMEX. We qualitatively show that DAMEX results in better expert utilization and avoid representation collapse which is a common problem with MoE training (2).
Our contributions can be summarized as:
We introduce a novel Dataset-aware Mixture-of-Expert, layer. DAMEX which disentangles dataset-specific features within the MoE layer, effectively tackling mixing of heterogeneous datasets. We experimentally show that DAMEX facilitates superior expert utilization and avoid the common issue of representation collapse in vanilla MoE.
Compared to the baselines, DAMEX does not require test-time dataset labels as it learns to route appropriate input dataset its corresponding expert during training itself, which is more challenging setup.
To the best of our knowledge, we are the first work explore the potential of MoE as being more than a scalability tool but an effective learner for a mixture of datasets by serving as knowledge- wwwlledgeensembling strategy within the model architecture with marginal increase in number of parameters w7r. dense architecture.
We establish new state-of-the-ar on UODB set by beating previously reported baseline by an average of +10.2 AP score and outperforming non-MoE baseline by +2.0 AP score on average. Further, we observe consistent improvements in various dataset mixing scenarios, like (1) Limited data availability, (2) Disparate domains, and (3) Divergent label sets.
2 Related Work
2.1 Mixing Multiple Datasets in Object-detection
Training on multiple datasets has emerged as a robust way to improve task performance of a deep learning model as shown in stereo matching 33),, depth estimation [1]]0] and detection performance
[35 16 45). Previously, [16] manually unifies taxonomy 7 semantic segmentation datasets using Mechanical Turk whereas 44 manually merge the label mapping of datasets at the detection head.
However, our objective is to build a universal object-detection model which should be able to adapt to any mixture of detection datasets. [37 trains partitioned detector three datasets and combine their information through an inter-dataset graph-based attention module. 451 goes a step forward and provides a way combine multiple separate detectors by learning a unified label space for multiple datasets. Their work proposed a new loss term to unify the label space but requires training separate partitioned detectors heads which are later unified with drop in performance. Our approach, on the other hand does not rely on unification of labels and does not need specialized partitioned detectors as utilizes the advantages of transformer based end-to-end detection pipeline and simply increases the number of classes in the last layer, resulting in marginal increase in parameters. Similar to (37), [35] trains universal detector on II datasets of Universal Object Detection Benchmark (UODB) by having separate detection heads for each dataset with domain attention module conditioned on datasets for supervision of backbone during training. However, like [37] their method also requires knowledge of the input dataset domain during test time whereas our objective universal detector is agnostic to the input data domain making the detection more challenging.
2.2 Mixture-of-Expert.
Mixture-of-expert (MoE) [14] is a machine learning concept that employs multiple expert layers, each of which specializes in solving specific subtask. The experts then work together to solve the entire task at hand. Recently, MoE has been widely applied to large-scale distributed Deep Learning models by using a cross-GPU layer that exchanges hidden features from different GPUs 21 4 18 BII The MoE approach is differentiated from existing scale-up approaches for DNNs, such as increasing the depth or width of DNNs, terms of its high cost-efficiency. Specifically, adding more model parameters (experts) in MoE layers does not increase the computational cost. per token. Thus, MoE has been studied for scaling for the models of trillion-size parameters in NL.P
[12 6 19
66 22 18].
3

---

primarily showed benefits of MoE on ImageNet classification task. In this work, we further pursue this direction by modelling MoE on object detection task and highlight the effectiveness, and robustness of MoEs in learning from mixture of datasets. Moreover, we improve over expert utilization problem of MoEs [2 in mixture of dataset setting by introducing a novel Dataset-Aware of its corresponding dataset while pooling shared information between the datasets the rest of the network.
In machine vision, 12 29
Cetture-f--xpertt, DAMEX layer that learns to make each expert an 'expert'
3 Preliminaries: Mixture-of-Experr (MoE)
An MoE layer consists of two parts: (1) a set of experts distributed across different GPUs, and (2) gating function (router). The distribution across GPUs accelerate the inference speed of MoE while keeping the parameter count same. The gating function or router determines the destination GPU of each input token, then the tokens are dispatched their target expert. These processed token representations are then combined such that the tokens are returned to their original location.
3.1 Routing of tokens
Let us denote the input tokens with x E R², set E of experts by {4] and router variable with Wr E RXD,, The probability of each expert p; will then become:
Ix Wr
(1)
exp (gzz)
pi(x))
Σ j=1 exp (gz,)
After calculating the probability of each expert (x),,,
B1 used top-k experts for routing the token.
33 calculates the output y as weighted combination of the processed tokens with their selection probability as:
Σ (e(x))x(x)
y/
iEtop-
We use & = 1 as it results in a marginal degradation in performance but a huge gain in throughput [6] and [1]. The output y is then passed forward to the remaining layers of the neural net.
3.2 Load balancing among the experts
Sparsity in the activation of experts desirable for the MoE layer with respect to given token. simplistic approach would be to select the top-k experts based on the softmax probability distribution. In practice however, this approach causes majority of the tokens to be assigned to a small number of busy experts causing a load imbalance during training. This leads to significant input buffer for few busy experts, causing a delay in the training process, while other experts remain untrained [6]. Additionally, many experts may not receive adequate training. Therefore, better gating function design is needed to distribute the processing burden equally across all experts.
To ensure better distribution of tokens, an auxiliary loss on experts is added along with the main task loss [18 [2]. The load balancing auxiliary loss Laand-halancing can be written as an average of Limportunce and Liaad-
The importance loss fimportance for expert e, is calculated by
Σ (x(x)
(4)
Em Var(1 ([)
Limportance
(5)
Mean(/)²
Let us denote the normal distribution as (0, I), where o gate noise The CDF of this normal |P]]
distribution is denoted as (.). For load L; for expert et, sample the CDF at probabilities p.(x)

---

Vanilla Mixture-of-Experts
DAMEX:
MoË Transformer module
Expart
Dipertid
Experto
Eapariol
Output Tokers
PPN
Expert-1
Expert-2
Expert-3
Expert-N
Add Hom
PFN:
FFN:
FFNg
FFN
MoE
Roster
Load belancing emong excerts
Matt-ilee Alkerkkn
Input images in batch
COCO tokens routed
Watercolor tokens routed to expert-N
to expert-1
Imput Tokens
Router
COCO
Watercolor
Dataset-aware expert selection
The dense FFN of transformer block replaced by an MoE layer. We show the difference between functioning of vanilla MoE layer DAMEX. Given two images from COCO and Watercolor dataset, the vanilla MoE router tends to balance the distribution of input tokens to each expert while DAMEX learns route dataset tokens to their corresponding expert (Here expert-1 is assigned to COCO dataset and expert-N to Watercolor dataset). During inference, the trained router choose the appropriate expert depending on the input token without any information of dataset source.
Figure 2: Overview of DAMEX:
(x)) and sum it over all the tokens in the image Im. The load L; denotes the number of assignments among the experts. The loss Lisad becomes
L
Σ (p)( æElm
(6)
Var T.)
Lioad
(7)
Mean L)2
Finally, the load balancing auxiliary loss Lloud-balancing is calculated by
Limportance + Lirad
L balamcing
(8)
2
Method
Our method Dataset-aware cotturr--of-xxpeere DAMEX is based Mixture of Experts (MoE) applied over transformer-based object detection pipeline DINO (DETR with Improved denoising anchor boxes) 42 such that each expert is trained to align to corresponding dataset, subtly making an actual 'expert' of that dataset-specific features. this section, we will explain our method in detail.
4.1 Setup
We replace alternate non-MoE transformer modules from decoder of DINO architecture with MoE transformer modules. A common problem that occurs with mixing of datasets in object-detection is to have separate detection heads for each dataset which increases the number of parameters significantly in two-stage detection pipeline [35 45). We leverage the transformer architecture of DINO in handling the number of classes by increasing the dimension of last classification layer which results in marginal increase in number of parameters (Rs 10, 000) compared to total model parameters, 46... In all experiments, we use one expert per GPU which keeps our parameter count same as that of non- Mon method (with addition of tiny router linear layer).
5

---

In vanilla MoE unlike image classification problem where every visual token is used for loss calcula- tion [ 29), we apply the -aad-balnnning loss (oustton 8jonyy on the foreground tokens in the detection task. We find that the foreground-only loss balancing results in better gradient updates and mitigate expert representation collapse on detection task.
4.2 DAMEX: Dataset-aware Mixture-of-Experts
Vanilla MoE balances the experts usage across GPUs to utilize maximum benefit from ensembling features from input tokens. However, can lead to inefficiency in knowledge sharing among experts due to distribution shift. For e.g. the expert domain knowledge required for a natural images dataset like COCO very different from a surveillance-stlle dataset like DOTA and we would want to learn their -ataset-specific characteristics. Thus, to solve this inefficiency with using MoE on cexture-of-dataseee we introduce Dataset-aware Mfxture-of-Expert (DAMEX) where we train the MoE router to route tokens depending on their dataset origin.
Given a set D of datasets {dmm ) such that input token x E dm. We define mapping function h: {D} + {E} such that each dataset dm is assigned specific expert 6;, then auxiliary loss LAMME is calculated by cross-entropy loss between logits p: (probability of selection of expert e:) and labels h(dm) (target expert for token x from dataset dm).
|
DAMEX = Σ ((h(dm) =i) (x)
DAMEX trains the MoE router dispatch all the visual tokens of the dataset its corresponding expert as shown in inFigure 2, Selecting specific expert ensure efficient usage of MoE and in-turn avoid their representation collapse.
5
Experiments
Datasets. We evaluate the presented approach on Universal Object Detection Benchmark (UODB)
[35] UODB comprises of 11 datasets: Pascal VOC [5], WiderFace 400, KITTI (8, LISA 26)..
DOTA 36 COCO [22). Watercolor, Clipart, Comic [3], Kitchen [ and Deepl Lesions
38 shown
Figure Pascal VOC and COCO have natural everyday object images while Kitchen features indoor kitchen items. WiderFace is huge human-face web images dataset covering a large variety of regions across the globe. Both KITTI and LISA datasets are based on traffic scenes captured with cameras mounted moving vehicles. KITTI includes categories such vehicles, pedestrians, and cyclists, whereas LISA is primarily composed traffic signs. DOTA is a surveillance dataset with aerial images annotating planes, ships, harbors etc.. Watercolor, Clipart and Comic cross- cross-domain datasets in their separate styles of images. Comic have the same label set as Pascal VOC, while Watercolor and Clipart are subset of that label set. DeepLesions is medical CT images dataset highlighting lesions. Altogether, the UODB offers a wide array of domains, label sets, camera views and image styles, providing comprehensive suite for evaluating multi-dataset object detection models.
Implementation details. Following recent end-to-end object-detection pipelines 23 19
adapted MoE on current state-of-the-art, DINO [42] We use pre-trained ImageNet ResNet-50 backbone with 4 scale features input across all methods. For hyper-parameters,
in DINO, use a 6-layer Transformer encoder and a 6-layer Transformer decoder and 256 as the hidden feature dimension. We altered DINO's decoder with alternate MoE layers following [12] using TUTEL library [12]. We use a capacity factor ff of 1.25 and an auxiliary expert-balancing loss weight of 0.1 with top-l selection of experts. We kept one expert per GPU and train 8 RTX6000 GPUs with batch-size of 2 per GPU, unless mentioned otherwise. Note that the number of parameters of model remains the same (except the addition of marginal router parameters) as each expert re- side on separate GPU replacing the existing feed-forward layer. For DAMEX, we keep the 1:1 dataset-mapping by assigning single expert to each dataset, except for full UODB run where we assign single expert to COCO, VOC and Clipart dataset while KITTI and LISA shared expert to fit 8 experts (on 8-GPU machine) on 11 datasets. We use learning-rate of 1.4e-4 and kept other DINO-specifis hyperparameters same as 42

---

Table Multi-dataset object detection results on UODB benchmark: We report mean AP score on UODB set. Mixing refers to whether the model is trained with all the datasets mixed together or individually. The top two rows denote single domain models which are separately trained on each set. Observe that MoE hardly increases the number of parameters but show jump in performance.
Mining
Pirams
KETT
Kicchen
IXTA
Deept
Clipan
Waterolor
DINO
66,6750
57.6
46.4
45.1
4017
39.5
10.1
40.6
DINO-Met
16,0800
58.0
47.11
45.4
41.0
10.0
Wang eral
44,7500
21.5
51.4
23.0
59.2
49.8
25.5
30.1
26.0
25.7
29.7
30.7
34.1
Zara al
70.1854
47.5
30.7
26.3
60.7
42.6
15.5
28.8
22.4
20.4
24.4
21.2
31.1
DINO
46.7353
59.4
53.9
36.7
73.0
48.6
39.9
48.7
42.9
23.2
27.8
20.1
42.3
DINO Mol:
66.2633
57.9
16.4
35.0
74.0
49.8
401.2
45-1
63.0
26.1
37.3
430
DAMEX(OUS)
46.7433
56.0
74.9
49.4
41.3
29.7
44.3
Baselines and metries. We compare our approach against [35 that is the previous state-of-the-art on UODB benchmark. [35 gains a further advantage as during inference, the object detector knows from which dataset each test image comes from and thus makes predictions in the corresponding label space. On the other hand, our setup is more challenging as during inference the method is agnostic to the source dataset. We also compare our method against Partitioned detector of 45 as their work is complimentary to our approach and establish a universal detector with a common backhone. ResNet-50 is the common backbone across all the methods in this paper, including baselines. For fair comparison on the architectural gains, we prepare competitive baselines by running DINO on each dataset individually as well as mixing all the datasets together. All the reported numbers this work are mean Average Precision (AP) scores evaluated on the available test or val set of corresponding dataset.
5.1 Multi-dataset Object Detection on UODB
In this section, we highlight our main results where we compare our DINO baseline against MoE DINO setup along with our proposed approach DAMEX on complete UODB datasets. Moreover, we re-evaluate the AP scores of (35] by running their best released model against COCO AP evaluation. Table IJshows the AP scores on individual dataset and average performance on UODB. We observe that the proposed method improve accuracy by an average of +00. AP score with previous state-of- the-art and shows an average improvement of +2.0 AP on Mixed DINO baseline. Individually, we observe increase in performance with DAMEX out-of-domain datasets, like DOTA (surveillance aerial images) and DeepLesion (medical CT images) due to experts learning dataset specific features. While, KITTI suffer from performance drop due to the reduced representation in the batch. Overall, dataset-aware MoE, DAMEX outperforms vanilla-MoE by +1.3 AP score highlighting the benefit of our proposed dataset-specific experts in multi-dataset training.
5.2 Analysis
We analyze the benefit of DAMEX in mixing datasets by quantitatively and qualitatively evaluating it for the desired properties associated with a universal object detector such as, limited data mixing, domain adaptation and same domain but different annotation set mixing. Further, visualize the deeper routing decisions made by DAMEX and vanilla MoE across mixed dataset classes to highlight the efficiency and effectiveness of DAMEX.
Limited data mixing. In this setup, demonstrate that the MoE based object detection model is better in handling imbalanced data. For this, we take KITTI and Kitchen dataset from the UODB set and create several few-shot scenarios by taking all the original images of Kitchen dataset and pair it against 50-shot, 100-shot, 1000-shot and complete KITTI dataset. The results have been presented in the Table 2 We demonstrate that the DAMEX is better in handling imbalanced even
decrease the number of training images with a relative percentage gain of 15.7% in 50-shot and 24.3% in 100-shot setting over Mon-MoE bascline. We observe that the gap between DAMEX and vanilla MoE decreases with increase in data points. We believe that vanilla MoE is able to find a similar optimal distribution at the class-level with large amount of data.
Domain adaptation. Another common scenario of mixing dataset is when there are two datasets with the same annotation set but different domains. To replicate this setting, we conduct our analysis on mixture of Watercolor and Comic dataset. Both Watercolor and Comic have the same label set
7

---

Table 2: Limited Data Mixing: We mix Kitchen dataset with n-shot examples of KITTI dataset. On right, we can observe the relative percentage gain of MoE and DAMEX over non-MoE method. In low-resource setting, DAMEX outperforms vanilla MoE due to efficient expert distribution whereas the gap diminishes as MoF found a similar distribution at the class-level with large amount of data.
vanila MoE DAMEX
of examples
50.bbo
100-shot
1000-shot
Full
Percantage Gain
Method
KITTI
Kitchen
KITTI
Kitchen
Kitchen
KITTI
Kitchen
Mixed DINO
14.6
46.3
18.9
48.1
40.0
48.5
63.0
48.1
Mixed DINO-MoE
16.4
47.6
20.9
46.3
45.2
47.0
68.7
48.3
DAMEX(OUs)
47.7
48.3
Shots (Log Scale)
but have very different styles of images. Watercolor has unique characteristics of watercolor art, such as transparent, fluid color washes, and visible brush strokes while Comic have bold, outlined objects and figures, with high-contrast color fills. The performance comparison of DAMEX, vanilla MoE, and non-MoE baseline is demonstrated in Table 3 We note a significant +2.9 AP increase on Watercolor and Comic datasets, underscoring that specialized experts can effectively learn dataset-specific traits, while shared network components can aggregate shared knowledge of the datasets.
Table 4: Addressing Divergent Label Sets in Mixed Datasets: KITTI and LISA, though both consist of traffic imagery. possess en- tirely disparate annotation sets, leading po- tentially conflicting gradient updates. Never- theless, DAMEX effectively circumvents this issue by learning dataset-specific features.
Table 3: Domain adaptation: Watercolor and Comic both have same label sets but different style image domains. By learning -ataset-specific features, DAMEX efficiently adapts to both dataset compared to vanilla MoE and non-MoE baseline.
Method
Watercolor
Comic
Method
KITTI
LISA
Mixed DINO
15.3
10.7
Mixed DINO
67.7
76.6
Mixed DINO-MoE
16.6
12.9
Mixed DINO-MoE
69.2
77.9
(AMEX(Ours)
18.2
13.6
(AMEX(Ours)
69.4
78.5
Table 5: Effect of dataset-expert mapping in DAMEX: We compare the effect of dataset-expert mapping on UODB set. We observe that random assignment is still better than vanilla MoE which is trained on load-balancing loss while DAMEX experts learns dataset-specific features. Further, adding a human prior (or domain knowledge) in mapping can lead to further performance improvements in DAMEX.
Method
KITTI
VOC
Widerfacc
LISA
Kitchen
COCO
DOTA
DeepLesion
Comic
Clipart
Wisercole
Mean
Mixed DINO-MoE
57.9
56.4
35.0
74.0
49.8
402
45.2
43.0)
24.1
27.3
1999
43.0
DAMIX: Readom magping
60.8
36.2
35.2
75.9
48.8
418
46.11
43.6
243
28.5
23.6
43.7
DAMEX: Human-grior mapping (ours)
61.8
36,0
35.1
74.9
49.4
41.
46.5
43.3
25.4
29.7
23.5
44.3
Having two datasets within the same domain but having different annotation set is a common problem with mixing datasets. A universal detector should be robust against such missing labels scenario. We conduct our analysis on the mixture of KITTI and LISA datasets. Both are traffic images dataset, however while KITTI label set contains pedestrians, cars and cyclists, LISA label set contains only traffic signs such speed limit, stop, no turn and warning sign. Note that these classes are present in both the datasets which make this a more challenging and realistic scenario for universal detector. Table 4 depicts the performance of DAMEX against vanilla MoE and non-MoE baseline. We observe that datasct-aware MoE outperforms the baseline with +1.7 AP score KITTI and +2.9 AP score LISA dataset, highlighting that universal detector can benefit from dataset-specific experts.
Mixing divergent label sets but within same domain.
Deeper routing decisions. We analyze the expert utilization of MoE layer in vanilla MoF against DAMEX. Figure illlsttrasss the distribution of expert selection weights over mixture of datasets. The plots were produced by running Mixed DINO-MoE and DAMEX from Table on UODB set.

---

The datasets belonging to same domain are mapped to same expert. Figure 3 show that experts are equally distributed across datasets with shallow MoE layer suffering from poor expert utilization. Vanilla MoE fail to specialize in discriminating among datasets. DAMEX the other hand efficiently use experts across datasets and conclusively learn select appropriate daaaset-specific expert during inference. The router selection improves in deeper MoE layers due to dataset-specific features learnt by earlier MoE layers. Moreover, DAMEX ensures fair expert utilization across all MoE layers.
Effect of human-prior in dataset-expert mapping. A central part of DAMEX is the dataset-expert mapping h {D) + {E} that the router learns during training. However, a question remains on how this mapping assignment affect the visual understanding of the model overall? We conduct an experiment where we randomly assign UODB datasets to experts and found a decrease in performance, shown in Table DAMEX allows the user to incorporate human-prior domain knowledge in mapping datasets to experts. This in-turn helps in sharing similar features from larger datasets, COCO and VOC belong to the same domain, natural images. In our experience, assigning datasets with similar domains to same expert tend to help performance while keeping disparate domains to separate experts.
Molf Vanilla MoE Expert Utilization
DAMEX Expert Utilization
Molf
Mel Layer
Export
Export
Expert
Expori
Figure 3: Expert Utilization in DAMEX across datasets: We show distribution of expert selection weights across dataset assignments. Here, Dn y-axis refer to datasets mapped to expert ID while r-axis correspond to 8 experts. The (expert e, dataset d) pair denote the average routing weights of the tokens corresponding bounding box of examples in dataset d. We can observe that DAMEX learns to route tokens to their mapped experts and is able to replicate it during inference without the knowledge of dataset-source. On the other hand, vanilla MoE struggles efficiently utilize experts and suffer with mode collapse at Layer 1.
Effect of Number of Experts on DAMEX. We study the effect of number of experts on DAMEX by experimenting on mixture of four disparate domain datasets, namely KITTI, DOTA, VOC and Clipart to maximize the effect of dataset-aware experts. Table shows the variation of mean AP score across the four datasets. best performance is obtained when number of experts are same as number of datasets. This observation different from vanilla MoEs where performance increases with increase in number of experts 12 29] It
Table 6: Effect of number of experts on DAMEX
# Experts
Clipart
DOTA
KITTI
VOC
Mean
can observe that the
34.0
45.3
62.0
57.7
49.7
32.6
44.9
56.5
48.7
highlights that DAMEX an efficient method for mixing datasets both in terms of compute and number of parameters as less number of experts require lesser compute. Note that for 8 experts, are mapping two experts per dataset with equal probability during training while expert selection during inference done by the trained router.
Conclusion
In this work, we have presented novel Dataset-aware Mixture-of-Fxpertt. (DAMEX) layer for the object-detection task. We demonstrate the potential of MoE models for mixing datasets on object- detection task. The introduced DAMEX model leverages the MoE architecture to learn dataset-specific features, thereby enhancing expert utilization and significantly improving the performance on universal object-detection benchmark with marginal increase parameters. Our experimental results establish a new state-of-the the-art performance on UODB by achieving an average increase of +10.2 AP score over the previous baseline and +2.0 AP score over non-MoE method.

---

7 Limitations & Social Impact
One limitation of our approach is that we are currently concatenating the classes of each dataset which results in duplicate recognition of overlapping classes. A future work in unifying these labels over our setup can be an interesting avenue.
Going forward it is important that we ensure that the universal object detectors are unbiased and maintain fairness in their predictions. Here unbiased refers classes with very few training examples or domain shift which can be called underrepresented classes. The detectors should be able to work on both classes with large, medium and few training samples in any environment. Similarly, for fairness, a universal object detector should be able to handle objects from different geographical regions, races, genders, etc. These are issues that we believe are ameliorated by methods focused on learning under imbalance or domain shift.
As result, we feel DAMEX could be method that takes step towards this where each dataset, even with very few samples can be represented by different experts. We can potentially divide a large dataset into different sets which cater to least one aspect of fairness and bias. However, we understand the need for further analysis and research before we have a truly unbiased and fair universal object detector and implore the community do so too.
References
[1] Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly to small image transformations? arXiv preprint arXiv: /2177, 2018.
[2] Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, et al. On the representation collapse of sparse mixture of experts. Advances in Neural Information Processing Systems, 45:34600-34613, 2022.
[3] Xiyang Dai, Yinpeng Chen, Bin Xiao, Dongdong Chen, Mengchen Liu, Lu Yuan, and Lei Zhang. Dynamic head: Unifying object detection heads with attentions. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 7373-7382. 2021.
[4] Nan Du Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph. Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstenn Toju Duke, Lucas Dixon, Kun Zhang, Quoe Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. GLaM: Efficient scaling of language models with mixture-of-experts. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 5547-5569. PMLR, 17-23 Jul 2022.
[5] Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111:98-1366 2015.
[6] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1):5232-522 5270, 2022.
[7] Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. Yolox: Exceeding yolo series in 2021. arXiv preprint 1070022107.08430, 2021.
[8] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pages 3354-3361 IEEE, 2012.
[9] Georgios Georgakis, Md Alimoor Reza, Arsalan Mousavian, Phi-Hung Le, and Jana Košecká Multiview rgb-d dataset for object instance detection. In 2016 Fourth International Conference on 3D Vision (3DV), pages 426-48 434. IEEE, 2016.
[10] Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omnivore: single model for many visual modalities. In Proceedings of the IEFF/VV Conference Computer Vision and Pattern Recognition, pages 16102-16112 12, 2022.
10

---

[11] Tanmay Gupta, Amita Kamath, Aniruddha Kembhavi, and Derek Hoiem. Towards general pur- pose vision systems: An end-to-end task-agnostic vision- Saiinnlan architecture. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16399-16409, 2022.
[12] Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang. Rafael Salas, Jithin Jose, Prabhat Ram, Joe Chau, Peng Cheng, Fan Yang. Mao Yang. and Yongqiang Xiong. Tutel: Adaptive mixture-of-experts at scale. CoRR, abs/2206.03382, June 2022.
[13] Naoto Inoue, Ryosuke Furuta, Toshihiko Yamasaki, and Kiyoharu Aizawa. Cross-domain weakly-supervised object detection through progressive domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5001-5009, 2018.
[14] Robert A Jacobs, Michael Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
[15] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan- Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02644 2023.
[16] John Lambert, Zhuang Liu, Ozan Sener, James Hays, and Vladlen Koltun. Mseg: A composite dataset for multi-domain semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2879-2888, 2020.
[17] Katrin Lasinger, René Ranftl, Konrad Schindler, and Vladlen Koltun. Towards robust monoc- ular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. arXiv preprint 1rXiv:1907.01344, 2019.
[18] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
[19] Feng Li. Hao Zhang. Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Du-detr: Accelerate detr training by introducing query denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13619 13627, 2022.
[20] X Li, W Wang, Hu, J Li, Tang, and Yang. Generalized focal loss v2: Learning reliable localization quality estimation for dense object detection. 2020 ieee. In CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, pages 13-19, 2020.
[21] Junyang Lin, An Yang, Jinze Bai, Chang Zhou, Le Jiang, Xianyan Jia, Ang Wang, Jic Zhang, Yong Li, Wei Lin, et al. M6-10t: A sharing-delinking paradigm for efficient multi-trillion parameter pretraining. arXiv preprint arXiv:21 2021.
[22] Tsung- Yi Lin, Michael Maire, Serge Belongie. James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 12, 2014, Proceedings, Part V 13, pages 750-755. Springer, 2014.
[23] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. DAB-DETR: Dynamic anchor boxes are better queries for DETR. In International Conference on Learning Representations, 2022.
[24] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.08916, 2022.
[25] David Marr. Vision: A computational investigation into the human representation and processing of visual information. MIT press, 2010.
[26] Andreas Mogelmose, Mohan Manubhai Trivedi, and Thomas B Moeslund. Vision-based traffic sign detection and analysis for intelligent driver assistance systems: Perspectives and survey. IEEE Transactions on Intelligent Transportation Systems, (3(4):1484-1499) 2012.
11

---

[27] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389-5400. PMLR, 2019.
[28] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[29] Carlos Riquelme, Joan Puigeerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583-8595 2021.
[30] Amir Rosenfeld, Richard Zemel, and John K Tsotsos. The elephant in the room. arXiv preprint arXiv: 1808
2018.
[31] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrre Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv: DrxXiv//70..00 /701.065.8, 2017.
[32] Antonio Torralba and Alexei A. Efros. Unbiased look at dataset bias. In CVPR 20/1, pages 1521-1528, 2011.
[33] Shimon Ullman, Michel Vidal-Naquet, and Erez Sali. Visual features of intermediate complexity and their use in classification. Nature neuroscience, 5(7):682-687, 2002.
[34] Chien- Yao Wang, Alexey Bochkovskiy, and Hong- Yuan Mark Liao. Scaled- ccaled-yolov4; Scaling cross stage partial network. In Proceedings of the IEEF/evf conference on computer vision and pattern recognition, pages 13029-13038, 2021.
[35] Xudong Wang. Zhaowei Cai, Dashan Gao, and Nuno Vasconcelos. Towards universal object detection by domain attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7289-7298, 2019.
[36] Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie. Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang. Dota: large-scale dataset for object detection in aerial images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9974-3983, 2018
[37] Hang Xu, Linpu Fang, Xiaodan Liang, Wenxiong Kang, and Zhenguo Li. Universal-renn: Universal object detector via transferable graph r-cnn. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12492-12499, 2020.
[38] Ke Yan, Xiaosong Wang, Le Lu, Ling Zhang, Adam P Harrison, Mohammadhadi Bagheri, and Ronald M Summers. Deep lesion graphs in the wild: relationship learning and organization of significant radiology image findings diverse large-scale lesion database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9261-9270, 2018.
[39] Gengshan Yang, Joshua Manela, Michael Happold, and Deva Ramanan. Hierarchical deep stereo matching on high-resolution images. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 5515-5524, 2019.
[40] Shuo Yang, Ping Luo, Chen-Change Loy, and Xiaoou Tang. Wider face: A face detection benchmark. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5525-5533, 2016.
[41] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2 1432, 2021.
[42] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. 2022.
arXiv preprint
12

---

[43] Haoyang Zhang, Ying Wang, Feras Dayoub, and Niko Sunderhauf. Varifocalnet: An iou-aware dense object detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8514-8523, 2021.
[44] Xiangyun Zhao, Samuel Schulter, Gaurav Sharma, Yi-Hsuan Tsai, Manmohan Chandraker, and Ying Wu. Object detection with a unified label space from multiple datasets. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28. 2020, Proceedings, Part XIV 16, pages 178-193. Springer, 2020.
[45] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühhl. Simple multi-dataset detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7571-7580, 2022.
13


Reviews:
[{"review_id": "D69zb3NpX3", "reviewer": "Reviewer_UzKA", "summary": "This paper tackles the problem of mixture-of-datasets training for object detection. The authors propose a mixture-of-expert-based (MoE) model that utilizes dataset-specific features to tackle mixing of heterogeneous datasets. The backbone is based on DINO transformer and one expert is assigned to one dataset. Results on the UODB dataset shows the efficacy of the proposed method.", "questions": "1. The line of work on mixture-of-datasets for video understanding should also be discussed [1*, 2*, 3*] in related work. Also, open-vocabulary object detection based on large vision-language models (VLMs) [4*] should also be considered.\n2. A inference speed comparison is needed. What is the FPS, for example, of the proposed method vs. others?\n3. Other minor comments:\nThe citation format makes some sentences strange: “Firstly, these datasets have been collected over time Everingham et al. [2015] and …” (Line 29)\nThe text in Figure 2 is too small to read.\n\n[1*] Akbari, H., Yuan, L., Qian, R., Chuang, W. H., Chang, S. F., Cui, Y., & Gong, B. (2021). Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. Advances in Neural Information Processing Systems, 34, 24206-24221.\n[2*] Duan, H., Zhao, Y., Xiong, Y., Liu, W., & Lin, D. (2020, August). Omni-sourced webly-supervised learning for video recognition. In European Conference on Computer Vision (pp. 670-688). Cham: Springer International Publishing.\n[3*] Liang, J., Zhang, E., Zhang, J., & Shen, C. (2022). Multi-dataset Training of Transformers for Robust Action Recognition. Advances in Neural Information Processing Systems, 35, 14475-14488.\n[4*] Yuan, L., Chen, D., Chen, Y. L., Codella, N., Dai, X., Gao, J., ... & Zhang, P. (2021). Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432.\n--------------------------Post rebuttal\n\nI have read the author rebuttal and other reviewers. The authors have addressed my concerns and questions. I'm increasing my score.", "limitations": "The authors have discussed the limitation and potential issues of the research.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea is straightforward and the intuition of using one expert per dataset makes sense. The method seems easy to implement and the authors promise to release code so the reproducibility is good.\n2. The proposed method is better than the previous SOTA by a large margin.", "weaknesses": "1. The design of assigning one expert for each dataset seems limited. The MoE model learns to route the tokens to a specific dataset expert, but in the real world there are many more than 11 datasets/styles. For images that are watercolor-comic etc., would the model get confused? Also, please discuss the need for a universal object detector trained on limited supervised datasets when large vision-language foundation models (VLMs) like [4*] can already get 60+ mAP (vs. 41 in this paper) on COCO. It would help to see the performance of the VLMs on these lesser-tested datasets like DOTA and Watercolor.\n2. The performance on common object detection benchmarks (MSCOCO, VOC, etc.) is lower than baselines. This may suggest that some features are difficult to learn under the MoE setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and investing their time in understanding our work. \n\nQ. “The design of assigning…”.\n\nWe would like to thank the reviewer for this question that prompted us to run a new experiment and improve our manuscript further. DAMEX allows the user to incorporate human prior in mapping datasets to experts which in-turn helps in leveraging similar features from larger datasets. Table A2 (rebuttal pdf) shows that DAMEX w/ random expert-dataset mapping performs worse than DAMEX w/ human-prior mapping but is still better than learned experts through load-balancing loss as in vanilla MoE. This shows that mapping plays an important part in the performance and having a human-prior in mapping can boost model performance. Further, DAMEX isn’t limited to one dataset/expert as we show in Table 1, where Expert 1 shares COCO, VOC and Clipart dataset. We will add a section on mapping in the camera-ready manuscript based on these results.\n\nQ. “For images that are watercolor-comic etc.,...?”\n\nFor the images that are watercolor-comic, we believe it will be routed to any of the watercolor or comic expert and should get equally good representations from both experts.\n\nQ. Need for VLFM.\n\nThank you for the question. We have answered this question under common answers.\n\nQ. “A inference speed …”\n\nThank you for the question. We find that the inference speed of DAMEX (2.99 FPS) is very similar to MoE (3.46) and believe that it can further be improved through code optimizations as the goal of our work is not higher inference speed but better performing mixture-of-datasets object detector.  Also, our implementation library (Tutel) is catered towards vanilla MoE not DAMEX.\n\n\nQ. “The performance on common…”\n\nYes, we agree with the reviewer in pointing this out. We observe that single dataset detectors of COCO, VOC outperform DAMEX. This is a trend that we notice in all baseline dataset-mixing papers too [1,2] which have non-MoE architectures as well. Our understanding is that natural image dataset distribution is significantly impacted with other dataset distribution types due to lighting, camera views and scale.  Further, most of the parameters are shared between the datasets and there is only a fraction of dataset-specific parameters,\n\nQ. Writing suggestions.\n\nThanks for the suggestions. As suggested, we will add a section on mixture-of-datasets for video understanding in related works. We acknowledge the issue with the citation style and fig 2. font size. We will fix the manuscript and incorporate the changes.\n\n\n[1] Xudong Wang, Zhaowei Cai, Dashan Gao, and Nuno Vasconcelos. Towards universal object detection by domain attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7289–7298, 2019\n\n[2] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Simple multi-dataset detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7571–7580, 2022.\n"}, {"review_id": "8DBeD1m9iF", "reviewer": "Reviewer_2x26", "summary": "This paper introduces a DAMEX layer based on the idea of assigning samples conforming to the charactersitiscs of a dataset to the corresponding expert. The underlying thought is to build dataset-relevant modules and ensemble them all together. Previous approaches leverage Mixture of Experts to scale their model while maintaining approximately the same inference speed. This work deals with the scenario of training multiple datasets together for object detection.", "questions": "See weaknesses please.", "limitations": "No. The authors mentioned the negative societal impact but does not have a solution yet.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The writing is clear. Results seem to prove the effectiveness of the proposed approach. The added number of parameters is small.", "weaknesses": "1. There is no introduced novelty in the proposed approach. It is good to know dataset specific experts are good for improving performances on object detection but the proposed DAMEX is in principle a linear routing layer that maps the input to different experts. The training procedure is also very straightforward: training different datasets on different GPUs. Essentially, this is equivalent to model ensemble a per-block router. Most of credits should go to GShard and Tutel. \n2. More clarification is needed for the improvement of performances. Throughout the results part, clarifications of why the proposed method is better than the others are missing. For example, I miss why DAMEX could deal with imbalanced datasets? Why it has better domain adaptation than DINO? \n3. Scalability is limited. From the experimental setting and the results, the maximum number of experts is 8, which is pretty small. It seems that it is highly correlated with the number of GPUs as well. From Tab. 5, we can see that when multiple experts are distributed to a single dataset, the performance drop is significant, which demonstrates that the approach is not scalable.\n4. Important experimental details are missing. For example, in Tab. 5, which two datasets are distributed together when # Experts = 2? The number of iterations needed for training is not given. \n5. From the visualization in Fig. 3, DAMEX has more condensed cluster such as Expert 1. This may indicate that other experts are not exploited well and it might work to delete some experts with no obvious decrease on performance. This potentially would have robustness issues. Again, it is more like doing majority voting from the ensemble perspective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their valuable feedback.\n\nQ. “There is no introduced novelty…”\n\nWe believe that performance gain and novelty are inter-related. \n\nSparsely activated networks are composed of components (experts), each of which learn to handle a subset of the complete set of training cases.\nThe key here is the division of training set into different subsets, or in other words, routing to different experts.\n \nGShard [1], Tutel [2] and Switch [3] learn the expert division at train time.\nWe take a different approach. We utilize the extra information that is available in the form of dataset/domain to guide the division. The training involves letting one expert handle samples from one data set.\n\n-\tAs noted above, the idea is different from what is being done commonly in the form of MoEs. And thus, it is novel in the context of this problem. \n\n-\tWhile it seems simple from the outlook, it is quite perplexing to see why it works so well, as noted by the reviewer. \n\n-\tThe fascinating part is that no such dataset label is provided at test time. But our training enables the network to recognize the dataset/domain. Thus, our novel loss function and model design allows the network to aggregate the knowledge across datasets in the shared parts and learn dataset-specific features in the dataset experts.\n\n- The dataset-expert mapping adds human-prior in the expert training which further helps performance. This can be seen in Table A2 (rebuttal pdf)  where DAMEX mapping is compared against a random assignment and vanilla MoE. We would like to extend our gratitude to R1 for suggesting this experiment and making our manuscript better.\n\n-\tTo summarize, this seemingly simple form of work division allows us to learn a good multi-dataset object detector. This finding is very useful for the community. This could lead the community to explore other kinds of expert division rather than just learning them at train time.\n\nQ. “More clarification is needed…”\n\nThank you for your feedback. DAMEX is better than baselines in domain-adaptation and imbalanced datasets because dataset-aware experts are able to learn dataset-specific features while the shared part of the network aggregate common information from the datasets. This again connects to our contributions: (a) human-prior expert-dataset mapping, (b) dataset-specific experts training, (c) correct expert selection during inference. We will further improve our writing in the corresponding sections of the manuscript to emphasize these points.\n\nQ. “Scalability is limited…”\n\nWe thank the reviewer for their question. We believe scalability can be achieved from two perspectives (a) Compute and (b) Datasets. We focus on the latter as the focus of our work. In DAMEX we know the number of datasets at training, hence, we can scale the number of optimal experts by matching it with the datasets. Instead of having 1 expert/gpu we can have 2 experts/gpu to accommodate higher expert load on the same compute. However, if we have less training datasets (say 4) then as shown in Tab 5, the optimal setup is using same number of experts as datasets. Having more experts (say 8, Tab 5) does not help in performance. \n\nQ. “Important experimental …”\n\nDuly noted. In Tab 5, VOC and KITTI share the same expert following same strategy as Tab 1. Other hyperparameters including number of epochs were set to be same as DINO (36 epochs) as mentioned at L223. We accept reviewer’s suggestions in writing and will clarify it in the camera-ready version.\n\nQ. “From the visualization …”\n\nWe understand reviewer's concern for Fig 3 and have clarified it in common answers. We provide an updated Fig 3 as Fig A1 in rebuttal pdf.\n\n[1] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi tional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\n[2] Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, Joe Chau, Peng Cheng, Fan Yang, Mao Yang, and Yongqiang Xiong. Tutel: Adaptive mixture-of-experts at scale. CoRR, abs/2206.03382, June 2022.\n\n[3] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1):5232–5270, 2022.\n\n"}, {"review_id": "BdTUz9RlRr", "reviewer": "Reviewer_1BaQ", "summary": "This work, motivated by the goal of developing a “universal detector,” seeks to understand how best to train a model across a large set of existing curated, labeled datasets that might differ in collection strategy, labeling standards, categories of interest, etc. They posit that the best strategy is to train dataset-specific features within a model that can be used to jointly predict on new images via mixture-of-experts, where dataset tokens are used to route images through the network based on the “expertise” of the learned features effectively and with many fewer parameters than prior multi-headed approaches. They show nice improvement using their method on the Universal Object-Detection Benchmark, and demonstrate that their method reduces representation collapse, a common issue with MoE training.\n\nAfter reading the authors rebuttal and discussing with them, I will maintain my score of 6.", "questions": "How would you add an additional dataset into the mix with this method? Would it require training from scratch?\n\nIt seems in figure 3 that despite the load balancing, expertise is heavily relying on “expert 1” for many of the classes. How is this related to the class distribution across datasets?", "limitations": "More nuanced evaluation and discussion of failure modes of their method on some datasets but high performance on others would help the reader gain intuition as to when this method would be worth implementing and applying. What about the individual dataset structure contributes to when this dataset-specific routing helps vs hurts?\n\nI do appreciate the discussion of label unification as a current limitation. \n\nThe comment “Going forward it is important that we ensure that the universal object detector are unbiased and maintain fairness in their predictions” feels like a throwaway. How would this work possibly contribute to that or how could it be adapted to contribute to that goal? How is the current paradigm possibly entrenching bias?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This work, which extends MoE-based dataset mixing to object detection, is differentiated from prior work using MoE for dataset mixing in image classification via the inclusion of specific per-dataset-expertise-routing layers within a larger shared-across-datasets architecture. They  distribute expertise via a load balancing loss (similar to prior work), which they adapt to the detection setting by applying it only to foreground tokens (reducing the influence of background on expertise routing).  They use explicit dataset tokens to route information to specific experts during training and inference, which requires train- and test-time dataset labels. DAMEX particularly seems to do well over vanilla MoE in a few-shot setting, as seen in the figure in Table 2.", "weaknesses": "The claims of performance improvements of 10% on average are slightly misleading, as there are probably lots of additional contributing factors that are not attributable to their method that led to prior reported numbers being lower. They see only a 2.3% improvement on average over vanilla DINO, and their contribution of a dataset-specific routing layer only improves performance over DINO-MoE by 1.3%. On many test sets their method underperforms by a significant margin. I would appreciate a more nuanced and representative claim, and some additional investigation and analysis as to why their dataset performs better on, e.g., DOTA, and much worse on Watercolor or LISA.\n\nSince the dataset tokens are explicit, I believe the model would need to be completely retrained to add a single additional source of data, which is (perhaps prohibitively) inefficient. Maybe an extension of this work would be to enable add-on routing/expertise generation for new mixing datasets without end-to-end retraining across all datasets.\n\nNit: \n\nI’m not sure the citation style matches the NeurIPS standard?\n\nGrammar is inconsistent and frequently in error throughout, with too many small mistakes to reasonably capture in this review. I would recommend significant text editing to increase the readability of a camera-ready manuscript for this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time in understanding our work and providing feedback to improve the manuscript. \n\nQ. “The claims of performance…”\nDuly noted. As mentioned in common concerns, we will change our writing to better convey the performance gains to the readers. \n\nWe understand reviewer's concern regarding why some baselines are better on few datasets and we believe it is because: \n\n-  Individual detectors tend to overfit on datasets and have lot more parameters. \n\n- We choose DINO as our detection pipeline as it is SOTA in unimodal object-detection method. With DINO as base architecture, we observe that DAMEX improves further on the base numbers, and we see consistent gains across datasets with the best overall performance. Note that, DAMEX is an architecture agnostic idea of using MoE in a dataset-aware setting.\n\n- Individual detectors and Wang et al. have knowledge of dataset-source during test time which is absent for DAMEX (L105-107). \n\nWe thank the reviewer for this insightful question and will update the writing to share these nuanced reasons.\n\nQ. “Since the dataset tokens…”\n\nYes, this is a very good point by the reviewer. In our current setup the addition of a new dataset will require re-training from scratch. We agree that an extension of this work would be addition of a new dataset in a continual learning environment by introducing a new expert with some frozen layers in the shared network. We avoided delving into this scenario as it would have diluted our core idea of DAMEX. We motivate the community to pursue this as future work.\n\nQ. Question regarding Fig 3.\n\nWe understand the issues with fig 3 and have cleared them up under common answers by providing a new figure (Figure A1, rebuttal pdf).\n\nQ. “What about the individual …”\n\nWe agree with the reviewer's thinking. Individual dataset structure is important in assessing dataset-expert mapping. In our experience, assigning datasets with similar domains to same expert tend to help performance while keeping disparate domains to separate experts. We will add this in the camera-ready manuscript.\n\nQ. Nits\n\nWe accept the change in citation style and will improve the grammar in the camera-ready manuscript. We will also make changes in Limitations sections as suggested by the reviewer.\n"}, {"review_id": "1tbaoc3Fw2", "reviewer": "Reviewer_962D", "summary": "The paper aims to develop a universal object detector that is applicable to a wide range of object detection datasets. For this aim, the authors proposes Dataset-aware Mixture of Experts (DAMEX). DAMEX is an extension of the vanilla Mixture of Experts layer to the multi-dataset scenario, in which each dataset is aimed to be routed to a specific expert. The authors also suggest replacing the fully connected layer in the transformer blocks by DAMEX layers. They incorporate such transformer blocks into DINO object detector. The experiments show that the approach improves upon previous SOTA as well as training a detector specifically for each dataset.", "questions": "1. Do you think open vocabulary object detection methods are also valid baselines for this task? Specifically, why don't you use a model such as GLIP to compare your method against?\n\n2. You mention that each GPU stores the weights of one expert. Following from this, do you pay specific attention to sampling the batch? What happens if all images (16 in total across all GPUs) come from the same dataset, in which case they should be sent to the same GPU, potentially causing memory issues?\n\n3. The learning rate of the proposed method is tuned to be 1.414e-4. Did you tune the learning rate for each dataset while obtaining the single domain models (the first two lines in Table 1)? If so, how?\n\n4. Why did you prefer using COCO-style AP instead of the performance measure of the benchmark that is Pascal VOC style AP?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper adapts MoEs to obtain universal object detection, which is intuitive.\n- The approach does not increase the number of parameters significantly while obtaining a MoE for several datasets.\n- The gains in terms of AP are significant over previous baseline on UODB benchmark. Also, the improvement over training a different DINO for each dataset is notable with ~4 AP. The method also outperforms using a Vanilla MoE by 1.3 AP.\n- The analysis part is comprehensive and it shows that the method does work as expected. For example, the model learns to assign the experts to the datasets properly.\n- Generally speaking, the paper is clearly written.", "weaknesses": "1. I think this task is closely related to open vocabulary object detection, on which there are already several different baseline methods. Such models are also evaluated on a set of multiple object detection datasets, e.g. using Object Detection in the Wild benchmark [A]. As an example method, GLIP [A] can easily be prompted given the union of the label spaces of each dataset in UODB benchmark. By considering recent advances, I'd expect at least an example of these models to be considered as a baseline. \n\n2. The authors use UODB benchmark (compared to Wang et. al, 2019) to evaluate their method. This dataset normally uses Pascal VOC style AP. Here, the authors use COCO-style AP. That's why, the AP values are significantly low in this paper (compared to Wang et. al, 2019) owing to the choice of this evaluation measure. It took me a while to dig into this and see the difference in evaluation. I'd expect this to be very clear in the metrics section.\n\n3. Please note that k index to define summation is not used in p_i or e_i. I'd recommend checking that equation.\n\n[A] Grounded Language-Image Pre-training, CVPR 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time in reviewing our work. \n\nQ. “I think this …”\n\nWe note the review feedback and their suggestion for GLIP. We answer the question under common answers and report the results in Table A1 (rebuttal pdf). \n\nQ. “The authors use UODB…”\n\nWe understand reviewer's concerns regarding the metrics used in our work and Wang et al.[2]. We consistently report COCO AP scores as a metric throughout the paper (L233 and L238) as it is an average over multiple IoU thresholds between 0.5 (coarse localization) and 0.95 (near-perfect localization). Pascal Style AP score is calculated over a fixed IoU threshold of 0.5.  Nevertheless, to remove any doubts we have provided Pascal Style AP score of DAMEX against all the baselines in Table A3 in the rebuttal pdf.\n\nQ. “Please note…”\n\nWe thank the reviewer for catching this typo, we will fix it in the camera-ready version.\n\nQ. “Do you think …”\n\nWe covered this question previously.\n\nQ. “You mention that …”\n\nWe are grateful for the reviewer for bringing this insightful point. Since the datasets are extremely imbalanced with Watercolor having only 500 train images vs 23k images of DeepLesion, we found in our initial experiments that uniform sampling across training images worked best for overall performance. For e.g., sampling based on inverse dataset size hurts overall performance. However, we agree that there can be a better sampling strategy for UODB which can improve DAMEX performance further and is agnostic to our contributions. \n\nIn the scenario mentioned, the method does not suffer from memory issues but a decrease in speed as all tokens are processed through a single expert. This bottleneck exists in vanilla MoE also and is an inherent feature of Mixture-of-Experts.\n\nQ. “The learning rate …”\n\nDuly noted. We change the learning to sqrt(2)*1e-4 following Krizhevsky’s [1] recommendation of learning rate dependence on batch size as we used a batch size of 2 during training. We followed a similar strategy for all baselines for fair comparison.\n\nQ. “Why did you prefer …”\n\nAnswered previously.\n\n We hope it will resolve all the concerns!\n\n\n[1] Krizhevsky, A., 2014. One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997.\n[2] Xudong Wang, Zhaowei Cai, Dashan Gao, and Nuno Vasconcelos. Towards universal object detection by domain attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7289–7298, 2019"}, {"review_id": "LQvvr8kP9X", "reviewer": "Reviewer_xzJh", "summary": "This manuscript is motivated from training a universal object detector on the mixture of unlimited counts of datasets. Concretely, the previous works cannot scale the MoE due to budget, and the conventional MoE using balanced route suffers from knowledge sharing issues. To tackle the aforementioned issues, the authors devised a Dataset Aware Mixture of Experts (DAMEX), in which a novel MoE loss is proposed to learn the dataset index distribution. Then, extensive experiments are conducted on various settings to show the effectiveness.", "questions": "Why the authors repeatedly mention GPU? Is there any efficient design for DAMEX? How to implement MoE on multi-GPU has been mentioned in Sec. 3, is this necessary to keep mentioning GPU setting?", "limitations": "As discussed in Weakness, I don' t agree with the authors tackle the dataset shift. But this can be further discussed.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-organized and well-motivated. The authors provide detailed explaination to the conventional MoE and provides certain discussion on the motivation.\n2. The idea of this manuscript is self-consistency, where the motivation is to tackle the knowledge distribution shift among datasets for existing MoE, and the $\\mathcal{L}_{DAMEX}$ pushes router learning the dataset index distribution seems to be work in intuition.\n3. The conducted experiments are wide and adequate to demonstrate the effectiveness of the proposed method.", "weaknesses": "1. The contribution of this manuscript is limited. All of the technical contributions are summarized as loss function of DAMEX, while it seems not settle the dataset distribution shift well. Concretely, as shown in Tab. 1, DAMEX fails to balance the results among datasets. As far as I am concerned, the dataset shift can be treated as hardness. To balance it, a larger improvement should be gained at harder datasets. Comparing with baseline DINO-MoE, the improvement of DAMEX concentrates on KITTI (3.9), COCO (1.1), DOTA (1.3), Clipart (2.4), Watercolor (3.6). There goes without strong correlation with dataset hardness. Welcome further discussion.\n2. The manuscript has unfair expression. The authors claimed they outperform the state-of-the-art with almost 10% AP. I cannot agree with a paper published in 2019 could be called as sota, and the DINO looks to be sota.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for investing their time in reviewing our work and suggesting improvements. \n\n\nQ. “The contributions of …”\n\nWe want to thank the reviewer for their feedback and wish to re-highlight our contributions.\n\n-\tDAMEX does not require test-time dataset labels as DAMEX learns to route appropriate dataset-expert during training itself as mentioned in L226 under “Baselines and Metrics”. This is a more challenging setup compared to Wang et al. [1] or individual detectors which have test-time dataset source information.  Our observation is that by providing supervision over expert selection in DAMEX eliminates the need of dataset source during inference as each expert learns dataset-specific features. \n\n-\tDAMEX avoids mode collapse of vanilla MoE by learning dataset-specific experts. Furthermore, DAMEX provides better representation learning by utilizing human-prior over dataset characteristics as a part of its dataset to expert mapping. This can be seen in Table A2 (rebuttal pdf) where DAMEX mapping is compared against a random assignment and vanilla MoE. We would like to extend our gratitude to R1 for suggesting this experiment and making our manuscript better. \n\n-\tThe number of parameters of DAMEX are smaller than baselines with individual detectors having 11x more parameters and previous works [2] having 1.5x more parameters. Yet, we observe consistent improvements from DAMEX on (1) UODB set, (2) Limited data availability, (3) Disparate domains, and (4) Divergent label sets.\n\n\nWe note reviewer’s point of view regarding harder datasets and observe that DAMEX gains are distributed across multiple dataset domains, natural images, traffic images, aerial images and style images. Further, prior to this work MoE are considered to be a scalability tool only and DINO-MoE hasn’t been implemented before in this context. We believe that DINO-MoE is a strong baseline and is a part of our contribution in showcasing that MoE can handle mixture-of-datasets well. A fair comparison would be against DINO where we can observe large gains across the UODB set.\n\n\nQ. “The manuscript has …”\n\nDuly noted. We discuss this concern in the common answers and plan to update the paper accordingly.\n\nQ. “Why the authors …”\n\nThanks for bringing this to our notice. As you already pointed out, the implementation is discussed in Sec.3. In the final draft, we will reduce the repeated mentioning of GPU. \n\n[1] Xudong Wang, Zhaowei Cai, Dashan Gao, and Nuno Vasconcelos. Towards universal object detection by domain attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7289–7298, 2019\n"}]

Author rebuttal:
We thank the reviewers for their feedback on DAMEX. Through their comments, we have gained a valuable insight of the paper from a reader’s perspective, and we are thankful for their suggestions on improving our manuscript.

1. **Comparison against Vision-Language Foundation models (R1, R4)** 

Our method and baselines are based on vision-modality and that’s why we didn’t  compare it against VLFMs as has been done by recently published prior works [1,2]. However, following request of R1,R4, we ran zero-shot GLIP [3] on UODB benchmark and report the performance in Table A1 (rebuttal pdf). GLIP [3] has been pre-trained on large datasets and show very good performance on natural image datasets like VOC and COCO but fail to perform on other domains, notably DeepLesion (medical dataset). Finally, we want to highlight that DAMEX is an architectural solution for mixing datasets and can be also applied on VLFMs like GLIP that can be explored as a future work.

2. **Explanation of Fig 3 (R2, R3)**

Fig 3 indicates the expert selection with respect to all the classes in the mixture of 11 datasets in UODB. As mentioned in the caption of Fig 3, the majority of classes are from VOC and COCO dataset which are mapped to Expert 1 and hence, there is a higher expert 1 utilization as both of these datasets are majority in the set. Contrary to this in vanilla MoE, we can observe that MoE Layer 1 collapses for Expert 0 and 1 for all the datasets. However, we thank the reviewers for their suggestions, and we have realized that dividing expert utilization wrt classes has resulted in a confusing figure. Please refer to Figure A1 (rebuttal pdf), where we have updated the figure 3.  We compare datasets against expert IDs and observe high expert utilization for each dataset against its assigned expert.

3. **Reporting of improvement over prev. SOTA (R3, R5)**

We try to be transparent in our performance gains by introducing DINO based baselines for comparison of DAMEX. However, we understand reviewer’s concerns and will rephrase writing to help the readers understand our performance gains accurately. 

[1] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung-Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03605, 2022.

[2] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Simple multi-dataset detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7571–7580, 2022.

[3] Liunian Harold Li*, Pengchuan Zhang*, Haotian Zhang*, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre-training. In CVPR, 2022


Meta-review: Summary
This paper proposes a way to train object detection systems with a mixture of datasets. The authors use a mixture of experts that learns dataset specific features to tackle different datasets in detection. The method is evaluated on the Universal Object Detection benchmark.

Strengths
- The authors propose to use MoE for object detection.
- The idea of using a simple load balancing loss to route foreground tokens is elegant. This provides major improvement over a vanilla MoE.
- Proposed method is parameter efficient

Reviews & Justification
The paper received positive reviews overall. The proposed idea is simple and shows good empirical gains on UODB. The authors are encouraged to include the comments from the reviewers about positioning this work wrt open vocabulary detection, and explaining the gain in performance clearly.
Decision: Accept (poster)