Title: DAMEX: Dataset-aware Mixture-of-Experts for visual understanding of mixture-of-datasets
Venue: NeurIPS 2023
Accepted: yes
Keywords: mixture-of-experts

moe

object detection

mixture of datasets

multiple datasets
Abstract: Construction of a universal detector poses a crucial question: How can we most effectively train a model on a large mixture of datasets? 
    The answer lies in learning dataset-specific features and ensembling their knowledge but do all this in a single model.
    Previous methods achieve this by having separate detection heads on a common backbone but that results in a significant increase in parameters.
    In this work, we present Mixture-of-Experts as a solution, highlighting that MoE are much more than a scalability tool. 
    We propose Dataset-Aware Mixture-of-Experts, DAMEX where we train the experts to become an `expert' of a dataset by learning to route each dataset tokens to its mapped expert.
    Experiments on Universal Object-Detection Benchmark show that we outperform the existing state-of-the-art by average +10.2 AP score and improve over our non-MoE baseline by average +2.0 AP score. We also observe consistent gains while mixing datasets with (1) limited availability, (2) disparate domains and (3) divergent label sets.
    Further, we qualitatively show that DAMEX is robust against expert representation collapse. Code is available at https://github.com/jinga-lala/DAMEX

DAMEX: Dataset-aware Mixture-of-Experts for visual understanding of mixture-of-datasets

Abstract

Construction of a universal detector poses a crucial question: How can we most effectively train a model on a large mixture of datasets? The answer lies in learning dataset-specific features and ensembling their knowledge but do all this in a single model. Previous methods achieve this by having separate detection heads on a common backbone but that results in a significant increase in parameters. In this work, we present Mixture-of-Experts as a solution, highlighting that MoEs are much more than a scalability tool. We propose Dataset-Aware Mixture-of-Experts, DAMEX where we train the experts to become an 'expert' of a dataset by learning to route each dataset tokens to its mapped expert. Experiments on Universal Object-Detection Benchmark show that we outperform the existing state-of-the-art by average +10.2 AP score and improve over our non-MoE baseline by average +2.0 AP score. We also observe consistent gains while mixing datasets with (1) limited availability, (2) disparate domains and (3) divergent label sets. Further, we qualitatively show that DAMEX is robust against expert representation collapse. Code is available at https://github.com/jinga-lala/DAMEX .

1 Introduction

Human visual perception has naturally evolved to excel at recognizing and localizing objects in a wide range of real-world scenarios [25, 33]. However, the field of computer vision has traditionally concentrated on training and evaluating object detection models on specific datasets [42, 3, 43, 7, 20, 28, 34]. These models have shown limited adaptability to new environments [1, 32], leading to decreased accuracy and practicality in real-world settings [30, 27]. Recognizing the challenges posed by the single dataset models and the availability of diverse datasets collected over time, unified models are being developed that merge information from multiple datasets into a single framework [35, 45, 11, 41, 24, 15].

The process of combining datasets presents several challenges that need to be addressed. Firstly, these datasets have been collected over time [5] and for different purposes, resulting in variations across domains. For example, one dataset may contain images from indoor environments [9], while another may be focused on the medical domain [38]. Secondly, imbalances in training images and tail distributions may exist across the datasets. This means that certain datasets and classes might have a substantial number of training examples, while others have only a few training examples. To illustrate, the popular Universal Object-Detection Benchmark (UODB) dataset for robust object detection demonstrates this diversity: the Clipart [13] data subset comprises 500 training images, whereas the COCO dataset [22] includes 35,000 training images. Figure 1 shows such diversities in the real world data, encompassing various aspects such as domains, the number of training images, camera views, scale and label set, taken from the multiple data sources. Can a universal object detection method be designed that effectively addresses the above issues associated with mixing multiple datasets?

VOC (Natural)

COcO (Natural)

To tackle multi-datasets based object detection problem, several approaches have been proposed. [16] manually unifies taxonomy of 7 semantic segmentation datasets using Mechanical Turk whereas our objective is to learn a single universal detector on any mixture of datasets. [45] provides a way to combine multiple separate detectors by learning a unified label space for multiple datasets. Their work proposed a new loss term to unify the label space but requires training separate partitioned detectors heads which are later unified with a drop in performance. [35] trains a universal detector on 11 datasets of Universal Object Detection Benchmark (UODB) and gain robustness by joining different sources of supervision. However, their method requires knowledge of the input dataset domain during test time whereas our objective universal detector should be agnostic to the input data domain making the problem more challenging.

In this work, we propose a Mixture-of-Experts (MoE) [14] based object detection method that can handle all the desired properties of a universal detector. Traditionally, MoEs have been used as a scaling tool in Natural Language Processing [6, 18, 31, 21, 4, 2] and image classification problem [12, 29]. However, we propose that Mixture-of-Experts are more than just scalable learners, instead MoEs are an effective and efficient solution to build a universal model for mixture of datasets. Building over vanilla MoE, we introduce a novel Dataset-aware Mixture-of-Experts model, DAMEX which learns to disentangle dataset-specific features at the MoE layer while pooling information at non-MoE layers. DAMEX learns to route tokens to their corresponding expert so that during inference it automatically chooses the best pathway for the test image through the network. We use DINO [42] based detection architecture for developing our approach.

We perform our evaluation on UODB dataset which comprises of 11 diverse datasets as shown in Figure 1. We compare against previously reported state-of-the-art on UODB [35] and recent multidataset object-detection method [45]. Moreover, we also compare our method against individual dataset trained DINO detectors along with complete UODB trained DINO model. We report an average +10.2 AP score improvement over previous baseline and a +2.0 AP gain over non-MoE based method. Additionally, we mix dataset pairs that highlight that MoE based detectors are better

in handling, (1) Mixing with limited data availability, (2) Domain Adaptation, (3) Mixing divergent label sets within same domain, and show consistent gains on DAMEX. We qualitatively show that DAMEX results in better expert utilization and avoid representation collapse which is a common problem with MoE training [2].

Our contributions can be summarized as:

We introduce a novel Dataset-aware Mixture-of-Experts layer, DAMEX which disentangles dataset-specific features within the MoE layer, effectively tackling mixing of heterogeneous datasets. We experimentally show that DAMEX facilitates superior expert utilization and avoid the common issue of representation collapse in vanilla MoE.

Compared to the baselines, DAMEX does not require test-time dataset labels as it learns to route appropriate input dataset to its corresponding expert during training itself, which is a more challenging setup.

To the best of our knowledge, we are the first work to explore the potential of MoE as being more than a scalability tool but an effective learner for a mixture of datasets by serving as a knowledge-ensembling strategy within the model architecture with a marginal increase in number of parameters wrt. dense architecture.

We establish a new state-of-the-art on UODB set by beating previously reported baseline by an average of +10.2 AP score and outperforming non-MoE baseline by +2.0 AP score on average. Further, we observe consistent improvements in various dataset mixing scenarios, like (1) Limited data availability, (2) Disparate domains, and (3) Divergent label sets.

2 Related Work

2.1 Mixing Multiple Datasets in Object-detection

Training on multiple datasets has emerged as a robust way to improve task performance of a deep learning model as shown in stereo matching [39], depth estimation [17, 10] and detection performance [35, 16, 45]. Previously, [16] manually unifies taxonomy of 7 semantic segmentation datasets using Mechanical Turk whereas [44] manually merge the label mapping of datasets at the detection head. However, our objective is to build a universal object-detection model which should be able to adapt to any mixture of detection datasets. [37] trains a partitioned detector on three datasets and combine their information through an inter-dataset graph-based attention module. [45] goes a step forward and provides a way to combine multiple separate detectors by learning a unified label space for multiple datasets. Their work proposed a new loss term to unify the label space but requires training separate partitioned detectors heads which are later unified with a drop in performance. Our approach, on the other hand does not rely on unification of labels and does not need specialized partitioned detectors as it utilizes the advantages of a transformer based end-to-end detection pipeline and simply increases the number of classes in the last layer, resulting in marginal increase in parameters. Similar to [37], [35] trains a universal detector on 11 datasets of Universal Object Detection Benchmark (UODB) by having separate detection heads for each dataset with a domain attention module conditioned on datasets for supervision of backbone during training. However, like [37] their method also requires knowledge of the input dataset domain during test time whereas our objective universal detector is agnostic to the input data domain making the detection more challenging.

2.2 Mixture-of-Experts

Mixture-of-experts (MoE) [14] is a machine learning concept that employs multiple expert layers, each of which specializes in solving a specific subtask. The experts then work together to solve the entire task at hand. Recently, MoE has been widely applied to large-scale distributed Deep Learning models by using a cross-GPU layer that exchanges hidden features from different GPUs [12, 6, 29, 21, 4, 18, 31]. The MoE approach is differentiated from existing scale-up approaches for DNNs, such as increasing the depth or width of DNNs, in terms of its high cost-efficiency. Specifically, adding more model parameters (experts) in MoE layers does not increase the computational cost per token. Thus, MoE has been studied for scaling for the models of trillion-size parameters in NLP [6, 21, 4, 18].

In machine vision, [12, 29] primarily showed benefits of MoE on ImageNet classification task. In this work, we further pursue this direction by modelling MoE on object detection task and highlight the effectiveness, and robustness of MoEs in learning from a mixture of datasets. Moreover, we improve over expert utilization problem of MoEs [2] in a mixture of dataset setting by introducing a novel Dataset-Aware Mixture-of-Experts, DAMEX layer that learns to make each expert an 'expert' of its corresponding dataset while pooling shared information between the datasets in the rest of the network.

3 Preliminaries: Mixture-of-Experts (MoE)

An MoE layer consists of two parts: (1) a set of experts distributed across different GPUs, and (2) a gating function (router). The distribution across GPUs accelerate the inference speed of MoE while keeping the parameter count same. The gating function or router determines the destination GPU of each input token, then the tokens are dispatched to their target expert. These processed token representations are then combined such that the tokens are returned to their original location.

3.1 Routing of tokens

Let us denote the input tokens with x ∈ R D , a set E of experts by { e i } | E | i =1 and router variable with W r ∈ R E × D . The probability of each expert p i will then become:

After calculating the probability of each expert p i ( x ) , [31] used top-k experts for routing the token. [31] calculates the output y as a weighted combination of the processed tokens with their selection probability as:

We use k = 1 as it results in a marginal degradation in performance but a huge gain in throughput [6] and [12]. The output y is then passed forward to the remaining layers of the neural net.

3.2 Load balancing among the experts

Sparsity in the activation of experts is desirable for the MoE layer with respect to a given token. A simplistic approach would be to select the top-k experts based on the softmax probability distribution. In practice however, this approach causes majority of the tokens to be assigned to a small number of busy experts causing a load imbalance during training. This leads to a significant input buffer for a few busy experts, causing a delay in the training process, while other experts remain untrained [6]. Additionally, many experts may not receive adequate training. Therefore, a better gating function design is needed to distribute the processing burden equally across all experts.

To ensure better distribution of tokens, an auxiliary loss on experts is added along with the main task loss [18, 12]. The load balancing auxiliary loss L load-balancing can be written as an average of L importance and L load .

The importance loss L importance for expert e i is calculated by

Let us denote the normal distribution as N ( 0 , σ 2 I ) , where σ = gate noise | E | . The CDF of this normal distribution is denoted as Φ( . ) . For load L i for expert e i , we sample the CDF at probabilities p i ( x )

DAMEX:Dataset-awareMoE

Expert-1

Expert-2

Expert-3

Expert-N

FFN

FFN2

FFN3

FFNN

Expert-1

Expert-2

Expert-3

Expert-N

FFN1

FFN2

FFN3

FFNN

Router

Loadbalancingamongexperts

Inputimagesinbatch

Cocotokensrouted

Watercolortokens

to expert-1

routed toexpert-N

Router

COCO

Watercolor

Dataset-awareexpertselection

as Φ( p i ( x )) and sum it over all the tokens x in the image Im. The load L i denotes the number of assignments among the experts. The loss L load becomes

Finally, the load balancing auxiliary loss L load-balancing is calculated by

4 Method

Our method Dataset-aware Mixture-of-Experts, DAMEX is based on Mixture of Experts (MoE) applied over transformer-based object detection pipeline DINO (DETR with Improved denoising anchor boxes) [42] such that each expert is trained to align to a corresponding dataset, subtly making it an actual 'expert' of that dataset-specific features. In this section, we will explain our method in detail.

4.1 Setup

We replace alternate non-MoE transformer modules from decoder of DINO architecture with MoE transformer modules. A common problem that occurs with mixing of datasets in object-detection is to have separate detection heads for each dataset which increases the number of parameters significantly in a two-stage detection pipeline [35, 45]. We leverage the transformer architecture of DINO in handling the number of classes by increasing the dimension of the last classification layer which results in marginal increase in number of parameters ( ≈ 10 , 000 ) compared to total model parameters, 46 M . In all experiments, we use one expert per GPU which keeps our parameter count same as that of non-MoE method (with an addition of a tiny router linear layer).

In vanilla MoE unlike image classification problem where every visual token is used for loss calculation [12, 2, 29], we apply the load-balancing loss Equation 8 only on the foreground tokens in the detection task. We find that the foreground-only loss balancing results in better gradient updates and mitigate expert representation collapse on detection task.

4.2 DAMEX: Dataset-aware Mixture-of-Experts

Vanilla MoE balances the experts usage across GPUs to utilize maximum benefit from ensembling features from input tokens. However, it can lead to inefficiency in knowledge sharing among experts due to distribution shift. For e.g. , the expert domain knowledge required for a natural images dataset like COCO is very different from a surveillance-style dataset like DOTA and we would want to learn their dataset-specific characteristics. Thus, to solve this inefficiency with using MoE on a mixture-of-datasets we introduce Dataset-aware Mixture-of-Experts (DAMEX) where we train the MoE router to route tokens depending on their dataset origin.

Given a set D of datasets { d m } | D | m =1 , such that input token x ∈ d m . We define a mapping function h : { D } → { E } such that each dataset d m is assigned a specific expert e i , then auxiliary loss L DAMEX is calculated by a cross-entropy loss between logits p i (probability of selection of expert e i ) and labels h ( d m ) (target expert for token x from dataset d m ).

DAMEX trains the MoE router to dispatch all the visual tokens of the dataset to its corresponding expert as shown in Figure 2. Selecting specific expert ensure efficient usage of MoE and in-turn avoid their representation collapse.

5 Experiments

Datasets. We evaluate the presented approach on Universal Object Detection Benchmark (UODB) [35]. UODB comprises of 11 datasets: Pascal VOC [5], WiderFace [40], KITTI [8], LISA [26], DOTA [36], COCO [22], Watercolor, Clipart, Comic [13], Kitchen [9] and DeepLesions [38], shown in Figure 1. Pascal VOC and COCO have natural everyday object images while Kitchen features indoor kitchen items. WiderFace is a huge human-face web images dataset covering a large variety of regions across the globe. Both KITTI and LISA datasets are based on traffic scenes captured with cameras mounted on moving vehicles. KITTI includes categories such as vehicles, pedestrians, and cyclists, whereas LISA is primarily composed of traffic signs. DOTA is a surveillance dataset with aerial images annotating planes, ships, harbors etc. . Watercolor, Clipart and Comic are cross-domain datasets in their separate styles of images. Comic have the same label set as Pascal VOC, while Watercolor and Clipart are a subset of that label set. DeepLesions is a medical CT images dataset highlighting lesions. Altogether, the UODB offers a wide array of domains, label sets, camera views and image styles, providing a comprehensive suite for evaluating multi-dataset object detection models.

Implementation details. Following recent end-to-end object-detection pipelines [23, 19] we adapted MoE on current state-of-the-art, DINO [42]. We use pre-trained ImageNet ResNet-50 backbone with 4 scale features as input across all methods. For hyper-parameters, as in DINO, we use a 6-layer Transformer encoder and a 6-layer Transformer decoder and 256 as the hidden feature dimension. We altered DINO's decoder with alternate MoE layers following [12] using TUTEL library [12]. We use a capacity factor f of 1.25 and an auxiliary expert-balancing loss weight of 0 . 1 with top-1 selection of experts. We kept one expert per GPU and train on 8 RTX6000 GPUs with a batch-size of 2 per GPU, unless mentioned otherwise. Note that the number of parameters of model remains the same (except the addition of marginal router parameters) as each expert reside on a separate GPU replacing the existing feed-forward layer. For DAMEX, we keep the 1:1 dataset-mapping by assigning single expert to each dataset, except for full UODB run where we assign a single expert to COCO, VOC and Clipart dataset while KITTI and LISA shared a expert to fit 8 experts (on 8-GPU machine) on 11 datasets. We use a learning-rate of 1 . 4 e -4 and kept other DINO-specific hyperparameters same as [42].

Baselines and metrics. We compare our approach against [35] that is the previous state-of-the-art on UODB benchmark. [35] gains a further advantage as during inference, the object detector knows from which dataset each test image comes from and thus makes predictions in the corresponding label space. On the other hand, our setup is more challenging as during inference the method is agnostic to the source dataset. We also compare our method against Partitioned detector of [45] as their work is complimentary to our approach and establish a universal detector with a common backbone. ResNet-50 is the common backbone across all the methods in this paper, including baselines. For fair comparison on the architectural gains, we prepare competitive baselines by running DINO on each dataset individually as well as mixing all the datasets together. All the reported numbers in this work are mean Average Precision (AP) scores evaluated on the available test or val set of corresponding dataset.

5.1 Multi-dataset Object Detection on UODB

In this section, we highlight our main results where we compare our DINO baseline against MoE DINO setup along with our proposed approach DAMEX on complete UODB datasets. Moreover, we re-evaluate the AP scores of [35] by running their best released model against COCO AP evaluation. Table 1 shows the AP scores on individual dataset and average performance on UODB. We observe that the proposed method improve accuracy by an average of +10.2 AP score with previous state-ofthe-art and shows an average improvement of +2.0 AP on Mixed DINO baseline. Individually, we observe an increase in performance with DAMEX on out-of-domain datasets, like DOTA (surveillance aerial images) and DeepLesion (medical CT images) due to experts learning dataset specific features. While, KITTI suffer from a performance drop due to the reduced representation in the batch. Overall, dataset-aware MoE, DAMEX outperforms vanilla-MoE by +1.3 AP score highlighting the benefit of our proposed dataset-specific experts in multi-dataset training.

5.2 Analysis

We analyze the benefit of DAMEX in mixing datasets by quantitatively and qualitatively evaluating it for the desired properties associated with a universal object detector such as, limited data mixing, domain adaptation and same domain but different annotation set mixing. Further, we visualize the deeper routing decisions made by DAMEX and vanilla MoE across mixed dataset classes to highlight the efficiency and effectiveness of DAMEX.

Limited data mixing. In this setup, we demonstrate that the MoE based object detection model is better in handling imbalanced data. For this, we take KITTI and Kitchen dataset from the UODB set and create several few-shot scenarios by taking all the original images of Kitchen dataset and pair it against 50-shot, 100-shot, 1000-shot and complete KITTI dataset. The results have been presented in the Table 2. We demonstrate that the DAMEX is better in handling imbalanced even as we decrease the number of training images with a relative percentage gain of 15.7% in 50-shot and 24.3% in 100-shot setting over non-MoE baseline. We observe that the gap between DAMEX and vanilla MoE decreases with increase in data points. We believe that vanilla MoE is able to find a similar optimal distribution at the class-level with large amount of data.

Domain adaptation. Another common scenario of mixing dataset is when there are two datasets with the same annotation set but different domains. To replicate this setting, we conduct our analysis on a mixture of Watercolor and Comic dataset. Both Watercolor and Comic have the same label set

50

100

1000

full dataset

# Shots (Log Scale)

0

5

10

15

20

25

30

Percentage Gain

vanilla MoE

DAMEX

but have very different styles of images. Watercolor has unique characteristics of watercolor art, such as transparent, fluid color washes, and visible brush strokes while Comic have bold, outlined objects and figures, with high-contrast color fills. The performance comparison of DAMEX, vanilla MoE, and non-MoE baseline is demonstrated in Table 3. We note a significant +2.9 AP increase on Watercolor and Comic datasets, underscoring that specialized experts can effectively learn dataset-specific traits, while shared network components can aggregate shared knowledge of the datasets.

Mixing divergent label sets but within same domain. Having two datasets within the same domain but having different annotation set is a common problem with mixing datasets. A universal detector should be robust against such missing labels scenario. We conduct our analysis on the mixture of KITTI and LISA datasets. Both are traffic images dataset, however while KITTI label set contains pedestrians, cars and cyclists, LISA label set contains only traffic signs such as speed limit, stop, no turn and warning sign. Note that these classes are present in both the datasets which make this a more challenging and realistic scenario for a universal detector. Table 4 depicts the performance of DAMEX against vanilla MoE and non-MoE baseline. We observe that dataset-aware MoE outperforms the baseline with +1.7 AP score on KITTI and +2.9 AP score on LISA dataset, highlighting that universal detector can benefit from dataset-specific experts.

Deeper routing decisions. We analyze the expert utilization of MoE layer in vanilla MoE against DAMEX. Figure 3 illustrates the distribution of expert selection weights over mixture of datasets. The plots were produced by running Mixed DINO-MoE and DAMEX from Table 1 on UODB set.

The datasets belonging to same domain are mapped to same expert. Figure 3 show that experts are equally distributed across datasets with shallow MoE layer suffering from poor expert utilization. Vanilla MoE fail to specialize in discriminating among datasets. DAMEX on the other hand efficiently use experts across datasets and conclusively learn to select appropriate dataset-specific expert during inference. The router selection improves in deeper MoE layers due to dataset-specific features learnt by earlier MoE layers. Moreover, DAMEX ensures fair expert utilization across all MoE layers.

Effect of human-prior in dataset-expert mapping. Acentral part of DAMEX is the dataset-expert mapping h : { D } → { E } that the router learns during training. However, a question remains on how this mapping assignment affect the visual understanding of the model overall? We conduct an experiment where we randomly assign UODB datasets to experts and found a decrease in performance, as shown in Table 5. DAMEX allows the user to incorporate human-prior or domain knowledge in mapping datasets to experts. This in-turn helps in sharing similar features from larger datasets, e.g. COCO and VOC belong to the same domain, natural images . In our experience, assigning datasets with similar domains to same expert tend to help performance while keeping disparate domains to separate experts.

0

1

2

3

4

5

6

7

Expert ID

D0

D1

D2

D3

D4

D5

D6

D7

UODB Datasets

0

1

2

3

4

5

6

7

Expert ID

D0

D1

D2

D3

D4

D5

D6

D7

0

1

2

3

4

5

6

7

Expert ID

D0

D1

D2

D3

D4

D5

D6

D7

0.240

0.245

0.250

0.255

0.260

Vanilla MoE Expert Utilization

0

1

2

3

4

5

6

7

Expert ID

D0

D1

D2

D3

D4

D5

D6

D7

UODB Datasets

0

1

2

3

4

5

6

7

Expert ID

D0

D1

D2

D3

D4

D5

D6

D7

0

1

2

3

4

5

6

7

Expert ID

D0

D1

D2

D3

D4

D5

D6

D7

0.2

0.4

0.6

0.8

DAMEX Expert Utilization

Effect of Number of Experts on DAMEX. We study the effect of number of experts on DAMEX by experimenting on a mixture of four disparate domain datasets, namely KITTI, DOTA, VOC and Clipart to maximize the effect of dataset-aware experts. Table 6 shows the variation of mean AP score across the four datasets. We can observe that the best performance is obtained when number of experts are same as number of datasets. This observation is different

from vanilla MoEs where performance increases with increase in number of experts [12, 29]. It highlights that DAMEX is an efficient method for mixing datasets both in terms of compute and number of parameters as less number of experts require lesser compute. Note that for 8 experts, we are mapping two experts per dataset with equal probability during training while expert selection during inference is done by the trained router.

6 Conclusion

In this work, we have presented a novel Dataset-aware Mixture-of-Experts (DAMEX) layer for the object-detection task. We demonstrate the potential of MoE models for mixing datasets on objectdetection task. The introduced DAMEX model leverages the MoE architecture to learn dataset-specific features, thereby enhancing expert utilization and significantly improving the performance on a universal object-detection benchmark with marginal increase in parameters. Our experimental results establish a new state-of-the-art performance on UODB by achieving an average increase of +10.2 AP score over the previous baseline and +2.0 AP score over non-MoE method.

7 Limitations & Social Impact

One limitation of our approach is that we are currently concatenating the classes of each dataset which results in a duplicate recognition of overlapping classes. A future work in unifying these labels over our setup can be an interesting avenue.

Going forward it is important that we ensure that the universal object detectors are unbiased and maintain fairness in their predictions. Here unbiased refers to classes with very few training examples or domain shift which can be called underrepresented classes. The detectors should be able to work on both classes with large, medium and few training samples in any environment. Similarly, for fairness, a universal object detector should be able to handle objects from different geographical regions, races, genders, etc. These are issues that we believe are ameliorated by methods focused on learning under imbalance or domain shift.

As a result, we feel DAMEX could be a method that takes a step towards this where each dataset, even with very few samples can be represented by different experts. We can potentially divide a large dataset into different sets which cater to at least one aspect of fairness and bias. However, we understand the need for further analysis and research before we have a truly unbiased and fair universal object detector and implore the community to do so too.

References

Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly to small image transformations? arXiv preprint arXiv:1805.12177 , 2018.

Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, et al. On the representation collapse of sparse mixture of experts. Advances in Neural Information Processing Systems , 35:34600-34613, 2022.

Xiyang Dai, Yinpeng Chen, Bin Xiao, Dongdong Chen, Mengchen Liu, Lu Yuan, and Lei Zhang. Dynamic head: Unifying object detection heads with attentions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 7373-7382, 2021.

Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P Bosma, Zongwei Zhou, Tao Wang, Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. GLaM: Efficient scaling of language models with mixture-of-experts. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 5547-5569. PMLR, 17-23 Jul 2022.

Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision , 111:98-136, 2015.

William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research , 23(1):5232-5270, 2022.

Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv:2107.08430 , 2021.

Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition , pages 3354-3361. IEEE, 2012.

Georgios Georgakis, Md Alimoor Reza, Arsalan Mousavian, Phi-Hung Le, and Jana Košecká. Multiview rgb-d dataset for object instance detection. In 2016 Fourth International Conference on 3D Vision (3DV) , pages 426-434. IEEE, 2016.

Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A single model for many visual modalities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16102-16112, 2022.

Tanmay Gupta, Amita Kamath, Aniruddha Kembhavi, and Derek Hoiem. Towards general purpose vision systems: An end-to-end task-agnostic vision-language architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16399-16409, 2022.

Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, Joe Chau, Peng Cheng, Fan Yang, Mao Yang, and Yongqiang Xiong. Tutel: Adaptive mixture-of-experts at scale. CoRR , abs/2206.03382, June 2022.

Naoto Inoue, Ryosuke Furuta, Toshihiko Yamasaki, and Kiyoharu Aizawa. Cross-domain weakly-supervised object detection through progressive domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5001-5009, 2018.

Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation , 3(1):79-87, 1991.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643 , 2023.

John Lambert, Zhuang Liu, Ozan Sener, James Hays, and Vladlen Koltun. Mseg: A composite dataset for multi-domain semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2879-2888, 2020.

Katrin Lasinger, René Ranftl, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. arXiv preprint arXiv:1907.01341 , 2019.

Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668 , 2020.

Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Dn-detr: Accelerate detr training by introducing query denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13619-13627, 2022.

X Li, W Wang, X Hu, J Li, J Tang, and J Yang. Generalized focal loss v2: Learning reliable localization quality estimation for dense object detection. 2020 ieee. In CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle , pages 13-19, 2020.

Junyang Lin, An Yang, Jinze Bai, Chang Zhou, Le Jiang, Xianyan Jia, Ang Wang, Jie Zhang, Yong Li, Wei Lin, et al. M6-10t: A sharing-delinking paradigm for efficient multi-trillion parameter pretraining. arXiv preprint arXiv:2110.03888 , 2021.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. DAB-DETR: Dynamic anchor boxes are better queries for DETR. In International Conference on Learning Representations , 2022.

Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.08916 , 2022.

David Marr. Vision: A computational investigation into the human representation and processing of visual information . MIT press, 2010.

Andreas Mogelmose, Mohan Manubhai Trivedi, and Thomas B Moeslund. Vision-based traffic sign detection and analysis for intelligent driver assistance systems: Perspectives and survey. IEEE Transactions on Intelligent Transportation Systems , 13(4):1484-1497, 2012.

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning , pages 5389-5400. PMLR, 2019.

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems , 28, 2015.

Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems , 34:8583-8595, 2021.

Amir Rosenfeld, Richard Zemel, and John K Tsotsos. The elephant in the room. arXiv preprint arXiv:1808.03305 , 2018.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538 , 2017.

Antonio Torralba and Alexei A. Efros. Unbiased look at dataset bias. In CVPR 2011 , pages 1521-1528, 2011.

Shimon Ullman, Michel Vidal-Naquet, and Erez Sali. Visual features of intermediate complexity and their use in classification. Nature neuroscience , 5(7):682-687, 2002.

Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. Scaled-yolov4: Scaling cross stage partial network. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition , pages 13029-13038, 2021.

Xudong Wang, Zhaowei Cai, Dashan Gao, and Nuno Vasconcelos. Towards universal object detection by domain attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 7289-7298, 2019.

Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang. Dota: A large-scale dataset for object detection in aerial images. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3974-3983, 2018.

Hang Xu, Linpu Fang, Xiaodan Liang, Wenxiong Kang, and Zhenguo Li. Universal-rcnn: Universal object detector via transferable graph r-cnn. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 12492-12499, 2020.

Ke Yan, Xiaosong Wang, Le Lu, Ling Zhang, Adam P Harrison, Mohammadhadi Bagheri, and Ronald M Summers. Deep lesion graphs in the wild: relationship learning and organization of significant radiology image findings in a diverse large-scale lesion database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 9261-9270, 2018.

Gengshan Yang, Joshua Manela, Michael Happold, and Deva Ramanan. Hierarchical deep stereo matching on high-resolution images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5515-5524, 2019.

Shuo Yang, Ping Luo, Chen-Change Loy, and Xiaoou Tang. Wider face: A face detection benchmark. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5525-5533, 2016.

Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432 , 2021.

Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and HeungYeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03605 , 2022.

Haoyang Zhang, Ying Wang, Feras Dayoub, and Niko Sunderhauf. Varifocalnet: An iou-aware dense object detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8514-8523, 2021.

Xiangyun Zhao, Samuel Schulter, Gaurav Sharma, Yi-Hsuan Tsai, Manmohan Chandraker, and Ying Wu. Object detection with a unified label space from multiple datasets. In Computer VisionECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16 , pages 178-193. Springer, 2020.

Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Simple multi-dataset detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7571-7580, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript is motivated from training a universal object detector on the mixture of unlimited counts of datasets. Concretely, the previous works cannot scale the MoE due to budget, and the conventional MoE using balanced route suffers from knowledge sharing issues. To tackle the aforementioned issues, the authors devised a Dataset Aware Mixture of Experts (DAMEX), in which a novel MoE loss is proposed to learn the dataset index distribution. Then, extensive experiments are conducted on various settings to show the effectiveness.\n\nSTRENGTHS:\n1. The paper is well-organized and well-motivated. The authors provide detailed explaination to the conventional MoE and provides certain discussion on the motivation.\n2. The idea of this manuscript is self-consistency, where the motivation is to tackle the knowledge distribution shift among datasets for existing MoE, and the $\\mathcal{L}_{DAMEX}$ pushes router learning the dataset index distribution seems to be work in intuition.\n3. The conducted experiments are wide and adequate to demonstrate the effectiveness of the proposed method.\n\nWEAKNESSES:\n1. The contribution of this manuscript is limited. All of the technical contributions are summarized as loss function of DAMEX, while it seems not settle the dataset distribution shift well. Concretely, as shown in Tab. 1, DAMEX fails to balance the results among datasets. As far as I am concerned, the dataset shift can be treated as hardness. To balance it, a larger improvement should be gained at harder datasets. Comparing with baseline DINO-MoE, the improvement of DAMEX concentrates on KITTI (3.9), COCO (1.1), DOTA (1.3), Clipart (2.4), Watercolor (3.6). There goes without strong correlation with dataset hardness. Welcome further discussion.\n2. The manuscript has unfair expression. The authors claimed they outperform the state-of-the-art with almost 10% AP. I cannot agree with a paper published in 2019 could be called as sota, and the DINO looks to be sota.\n\nQUESTIONS:\nWhy the authors repeatedly mention GPU? Is there any efficient design for DAMEX? How to implement MoE on multi-GPU has been mentioned in Sec. 3, is this necessary to keep mentioning GPU setting?\n\nLIMITATIONS:\nAs discussed in Weakness, I don' t agree with the authors tackle the dataset shift. But this can be further discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to develop a universal object detector that is applicable to a wide range of object detection datasets. For this aim, the authors proposes Dataset-aware Mixture of Experts (DAMEX). DAMEX is an extension of the vanilla Mixture of Experts layer to the multi-dataset scenario, in which each dataset is aimed to be routed to a specific expert. The authors also suggest replacing the fully connected layer in the transformer blocks by DAMEX layers. They incorporate such transformer blocks into DINO object detector. The experiments show that the approach improves upon previous SOTA as well as training a detector specifically for each dataset.\n\nSTRENGTHS:\n- The paper adapts MoEs to obtain universal object detection, which is intuitive.\n- The approach does not increase the number of parameters significantly while obtaining a MoE for several datasets.\n- The gains in terms of AP are significant over previous baseline on UODB benchmark. Also, the improvement over training a different DINO for each dataset is notable with ~4 AP. The method also outperforms using a Vanilla MoE by 1.3 AP.\n- The analysis part is comprehensive and it shows that the method does work as expected. For example, the model learns to assign the experts to the datasets properly.\n- Generally speaking, the paper is clearly written.\n\nWEAKNESSES:\n1. I think this task is closely related to open vocabulary object detection, on which there are already several different baseline methods. Such models are also evaluated on a set of multiple object detection datasets, e.g. using Object Detection in the Wild benchmark [A]. As an example method, GLIP [A] can easily be prompted given the union of the label spaces of each dataset in UODB benchmark. By considering recent advances, I'd expect at least an example of these models to be considered as a baseline. \n\n2. The authors use UODB benchmark (compared to Wang et. al, 2019) to evaluate their method. This dataset normally uses Pascal VOC style AP. Here, the authors use COCO-style AP. That's why, the AP values are significantly low in this paper (compared to Wang et. al, 2019) owing to the choice of this evaluation measure. It took me a while to dig into this and see the difference in evaluation. I'd expect this to be very clear in the metrics section.\n\n3. Please note that k index to define summation is not used in p_i or e_i. I'd recommend checking that equation.\n\n[A] Grounded Language-Image Pre-training, CVPR 2022\n\nQUESTIONS:\n1. Do you think open vocabulary object detection methods are also valid baselines for this task? Specifically, why don't you use a model such as GLIP to compare your method against?\n\n2. You mention that each GPU stores the weights of one expert. Following from this, do you pay specific attention to sampling the batch? What happens if all images (16 in total across all GPUs) come from the same dataset, in which case they should be sent to the same GPU, potentially causing memory issues?\n\n3. The learning rate of the proposed method is tuned to be 1.414e-4. Did you tune the learning rate for each dataset while obtaining the single domain models (the first two lines in Table 1)? If so, how?\n\n4. Why did you prefer using COCO-style AP instead of the performance measure of the benchmark that is Pascal VOC style AP?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work, motivated by the goal of developing a “universal detector,” seeks to understand how best to train a model across a large set of existing curated, labeled datasets that might differ in collection strategy, labeling standards, categories of interest, etc. They posit that the best strategy is to train dataset-specific features within a model that can be used to jointly predict on new images via mixture-of-experts, where dataset tokens are used to route images through the network based on the “expertise” of the learned features effectively and with many fewer parameters than prior multi-headed approaches. They show nice improvement using their method on the Universal Object-Detection Benchmark, and demonstrate that their method reduces representation collapse, a common issue with MoE training.\n\nAfter reading the authors rebuttal and discussing with them, I will maintain my score of 6.\n\nSTRENGTHS:\nThis work, which extends MoE-based dataset mixing to object detection, is differentiated from prior work using MoE for dataset mixing in image classification via the inclusion of specific per-dataset-expertise-routing layers within a larger shared-across-datasets architecture. They  distribute expertise via a load balancing loss (similar to prior work), which they adapt to the detection setting by applying it only to foreground tokens (reducing the influence of background on expertise routing).  They use explicit dataset tokens to route information to specific experts during training and inference, which requires train- and test-time dataset labels. DAMEX particularly seems to do well over vanilla MoE in a few-shot setting, as seen in the figure in Table 2.\n\nWEAKNESSES:\nThe claims of performance improvements of 10% on average are slightly misleading, as there are probably lots of additional contributing factors that are not attributable to their method that led to prior reported numbers being lower. They see only a 2.3% improvement on average over vanilla DINO, and their contribution of a dataset-specific routing layer only improves performance over DINO-MoE by 1.3%. On many test sets their method underperforms by a significant margin. I would appreciate a more nuanced and representative claim, and some additional investigation and analysis as to why their dataset performs better on, e.g., DOTA, and much worse on Watercolor or LISA.\n\nSince the dataset tokens are explicit, I believe the model would need to be completely retrained to add a single additional source of data, which is (perhaps prohibitively) inefficient. Maybe an extension of this work would be to enable add-on routing/expertise generation for new mixing datasets without end-to-end retraining across all datasets.\n\nNit: \n\nI’m not sure the citation style matches the NeurIPS standard?\n\nGrammar is inconsistent and frequently in error throughout, with too many small mistakes to reasonably capture in this review. I would recommend significant text editing to increase the readability of a camera-ready manuscript for this work.\n\nQUESTIONS:\nHow would you add an additional dataset into the mix with this method? Would it require training from scratch?\n\nIt seems in figure 3 that despite the load balancing, expertise is heavily relying on “expert 1” for many of the classes. How is this related to the class distribution across datasets?\n\nLIMITATIONS:\nMore nuanced evaluation and discussion of failure modes of their method on some datasets but high performance on others would help the reader gain intuition as to when this method would be worth implementing and applying. What about the individual dataset structure contributes to when this dataset-specific routing helps vs hurts?\n\nI do appreciate the discussion of label unification as a current limitation. \n\nThe comment “Going forward it is important that we ensure that the universal object detector are unbiased and maintain fairness in their predictions” feels like a throwaway. How would this work possibly contribute to that or how could it be adapted to contribute to that goal? How is the current paradigm possibly entrenching bias?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a DAMEX layer based on the idea of assigning samples conforming to the charactersitiscs of a dataset to the corresponding expert. The underlying thought is to build dataset-relevant modules and ensemble them all together. Previous approaches leverage Mixture of Experts to scale their model while maintaining approximately the same inference speed. This work deals with the scenario of training multiple datasets together for object detection.\n\nSTRENGTHS:\nThe writing is clear. Results seem to prove the effectiveness of the proposed approach. The added number of parameters is small.\n\nWEAKNESSES:\n1. There is no introduced novelty in the proposed approach. It is good to know dataset specific experts are good for improving performances on object detection but the proposed DAMEX is in principle a linear routing layer that maps the input to different experts. The training procedure is also very straightforward: training different datasets on different GPUs. Essentially, this is equivalent to model ensemble a per-block router. Most of credits should go to GShard and Tutel. \n2. More clarification is needed for the improvement of performances. Throughout the results part, clarifications of why the proposed method is better than the others are missing. For example, I miss why DAMEX could deal with imbalanced datasets? Why it has better domain adaptation than DINO? \n3. Scalability is limited. From the experimental setting and the results, the maximum number of experts is 8, which is pretty small. It seems that it is highly correlated with the number of GPUs as well. From Tab. 5, we can see that when multiple experts are distributed to a single dataset, the performance drop is significant, which demonstrates that the approach is not scalable.\n4. Important experimental details are missing. For example, in Tab. 5, which two datasets are distributed together when # Experts = 2? The number of iterations needed for training is not given. \n5. From the visualization in Fig. 3, DAMEX has more condensed cluster such as Expert 1. This may indicate that other experts are not exploited well and it might work to delete some experts with no obvious decrease on performance. This potentially would have robustness issues. Again, it is more like doing majority voting from the ensemble perspective.\n\nQUESTIONS:\nSee weaknesses please.\n\nLIMITATIONS:\nNo. The authors mentioned the negative societal impact but does not have a solution yet.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of mixture-of-datasets training for object detection. The authors propose a mixture-of-expert-based (MoE) model that utilizes dataset-specific features to tackle mixing of heterogeneous datasets. The backbone is based on DINO transformer and one expert is assigned to one dataset. Results on the UODB dataset shows the efficacy of the proposed method.\n\nSTRENGTHS:\n1. The idea is straightforward and the intuition of using one expert per dataset makes sense. The method seems easy to implement and the authors promise to release code so the reproducibility is good.\n2. The proposed method is better than the previous SOTA by a large margin.\n\nWEAKNESSES:\n1. The design of assigning one expert for each dataset seems limited. The MoE model learns to route the tokens to a specific dataset expert, but in the real world there are many more than 11 datasets/styles. For images that are watercolor-comic etc., would the model get confused? Also, please discuss the need for a universal object detector trained on limited supervised datasets when large vision-language foundation models (VLMs) like [4*] can already get 60+ mAP (vs. 41 in this paper) on COCO. It would help to see the performance of the VLMs on these lesser-tested datasets like DOTA and Watercolor.\n2. The performance on common object detection benchmarks (MSCOCO, VOC, etc.) is lower than baselines. This may suggest that some features are difficult to learn under the MoE setting.\n\nQUESTIONS:\n1. The line of work on mixture-of-datasets for video understanding should also be discussed [1*, 2*, 3*] in related work. Also, open-vocabulary object detection based on large vision-language models (VLMs) [4*] should also be considered.\n2. A inference speed comparison is needed. What is the FPS, for example, of the proposed method vs. others?\n3. Other minor comments:\nThe citation format makes some sentences strange: “Firstly, these datasets have been collected over time Everingham et al. [2015] and …” (Line 29)\nThe text in Figure 2 is too small to read.\n\n[1*] Akbari, H., Yuan, L., Qian, R., Chuang, W. H., Chang, S. F., Cui, Y., & Gong, B. (2021). Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. Advances in Neural Information Processing Systems, 34, 24206-24221.\n[2*] Duan, H., Zhao, Y., Xiong, Y., Liu, W., & Lin, D. (2020, August). Omni-sourced webly-supervised learning for video recognition. In European Conference on Computer Vision (pp. 670-688). Cham: Springer International Publishing.\n[3*] Liang, J., Zhang, E., Zhang, J., & Shen, C. (2022). Multi-dataset Training of Transformers for Robust Action Recognition. Advances in Neural Information Processing Systems, 35, 14475-14488.\n[4*] Yuan, L., Chen, D., Chen, Y. L., Codella, N., Dai, X., Gao, J., ... & Zhang, P. (2021). Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432.\n--------------------------Post rebuttal\n\nI have read the author rebuttal and other reviewers. The authors have addressed my concerns and questions. I'm increasing my score.\n\nLIMITATIONS:\nThe authors have discussed the limitation and potential issues of the research.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}