SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Limited data mixing. In this setup, we demonstrate that the MoE based object detection model is better in handling imbalanced data. For this, we take KITTI and Kitchen dataset from the UODB set and create several few-shot scenarios by taking all the original images of Kitchen dataset and pair it against 50-shot, 100-shot, 1000-shot and complete KITTI dataset. The results have been presented in the Table 2. We demonstrate that the DAMEX is better in handling imbalanced even as we decrease the number of training images with a relative percentage gain of  $15.7\%$  in 50-shot and  $24.3\%$  in 100-shot setting over non-MoE baseline. We observe that the gap between DAMEX and vanilla MoE decreases with increase in data points. We believe that vanilla MoE is able to find a similar optimal distribution at the class-level with large amount of data.
Domain adaptation. Another common scenario of mixing dataset is when there are two datasets with the same annotation set but different domains. To replicate this setting, we conduct our analysis on a mixture of Watercolor and Comic dataset. Both Watercolor and Comic have the same label set
Table 2: Limited Data Mixing: We mix Kitchen dataset with n-shot examples of KITTI dataset. On right, we can observe the relative percentage gain of MoE and DAMEX over non-MoE method. In low-resource setting, DAMEX outperforms vanilla MoE due to efficient expert distribution whereas the gap diminishes as MoE found a similar distribution at the class-level with large amount of data.
# of examples	50-shot	100-shot	1000-shot	Full
Method	KITTI	Kitchen	KITTI	Kitchen	KITTI	Kitchen	KITTI	Kitchen
Mixed DINO	14.6	46.3	18.9	48.1	40.0	48.5	63.0	48.1
Mixed DINO-MoE	16.4	47.6	20.9	46.3	45.2	47.0	68.7	48.3
DAMEX(Ours)	16.9	47.7	23.5	46.0	46.1	47.5	68.5	48.3
but have very different styles of images. Watercolor has unique characteristics of watercolor art, such as transparent, fluid color washes, and visible brush strokes while Comic have bold, outlined objects and figures, with high-contrast color fills. The performance comparison of DAMEX, vanilla MoE, and non-MoE baseline is demonstrated in Table 3. We note a significant  $+2.9$  AP increase on Watercolor and Comic datasets, underscoring that specialized experts can effectively learn dataset-specific traits, while shared network components can aggregate shared knowledge of the datasets.
Table 3: Domain adaptation: Watercolor and Comic both have same label sets but different style image domains. By learning dataset-specific features, DAMEX efficiently adapts to both dataset compared to vanilla MoE and non-MoE baseline.
Method	Watercolor	Comic
Mixed DINO	15.3	10.7
Mixed DINO-MoE	16.6	12.9
DAMEX(Ours)	18.2	13.6
Table 4: Addressing Divergent Label Sets in Mixed Datasets: KITTI and LISA, though both consist of traffic imagery, possess entirely disparate annotation sets, leading to potentially conflicting gradient updates. Nevertheless, DAMEX effectively circumvents this issue by learning dataset-specific features.
Method	KITTI	LISA
Mixed DINO	67.7	76.6
Mixed DINO-MoE	69.2	77.9
DAMEX(Ours)	69.4	78.5
Table 5: Effect of dataset-expert mapping in DAMEX: We compare the effect of dataset-expert mapping on UODB set. We observe that random assignment is still better than vanilla MoE which is trained on load-balancing loss while DAMEX experts learns dataset-specific features. Further, adding a human prior (or domain knowledge) in mapping can lead to further performance improvements in DAMEX.
Method	KITTI	VOC	WiderFace	LISA	Kitchen	COCO	DOTA	DeepLesion	Comic	Clipart	Watercolor	Mean
Mixed DINO-MoE	57.9	56.4	35.0	74.0	49.8	40.2	45.2	43.0	24.1	27.3	19.9	43.0
DAMEX: Random mapping	60.8	56.2	35.2	73.9	48.8	41.8	46.0	43.6	24.3	28.5	21.6	43.7
DAMEX: Human-prior mapping (ours)	61.8	56.0	35.1	74.9	49.4	41.3	46.5	43.3	25.4	29.7	23.5	44.3
Mixing divergent label sets but within same domain. Having two datasets within the same domain but having different annotation set is a common problem with mixing datasets. A universal detector should be robust against such missing labels scenario. We conduct our analysis on the mixture of KITTI and LISA datasets. Both are traffic images dataset, however while KITTI label set contains pedestrians, cars and cyclists, LISA label set contains only traffic signs such as speed limit, stop, no turn and warning sign. Note that these classes are present in both the datasets which make this a more challenging and realistic scenario for a universal detector. Table 4 depicts the performance of DAMEX against vanilla MoE and non-MoE baseline. We observe that dataset-aware MoE outperforms the baseline with +1.7 AP score on KITTI and +2.9 AP score on LISA dataset, highlighting that universal detector can benefit from dataset-specific experts.
Deeper routing decisions. We analyze the expert utilization of MoE layer in vanilla MoE against DAMEX. Figure 3 illustrates the distribution of expert selection weights over mixture of datasets. The plots were produced by running Mixed DINO-MoE and DAMEX from Table 1 on UODB set.
The datasets belonging to same domain are mapped to same expert. Figure 3 show that experts are equally distributed across datasets with shallow MoE layer suffering from poor expert utilization. Vanilla MoE fail to specialize in discriminating among datasets. DAMEX on the other hand efficiently use experts across datasets and conclusively learn to select appropriate dataset-specific expert during inference. The router selection improves in deeper MoE layers due to dataset-specific features learnt by earlier MoE layers. Moreover, DAMEX ensures fair expert utilization across all MoE layers.
Effect of human-prior in dataset-expert mapping. A central part of DAMEX is the dataset-expert mapping  $h: \{D\} \to \{E\}$  that the router learns during training. However, a question remains on how this mapping assignment affects the visual understanding of the model overall? We conduct an experiment where we randomly assign UODB datasets to experts and found a decrease in performance, as shown in Table 5. DAMEX allows the user to incorporate human-prior or domain knowledge in mapping datasets to experts. This in-turn helps in sharing similar features from larger datasets, e.g. COCO and VOC belong to the same domain, natural images. In our experience, assigning datasets with similar domains to same expert tend to help performance while keeping disparate domains to separate experts.
Figure 3: Expert Utilization in DAMEX across datasets: We show distribution of expert selection weights across dataset assignments. Here,  $Dn$  on  $y$ -axis refer to datasets mapped to expert ID  $n$  while  $x$ -axis correspond to 8 experts. The (expert  $e$ , dataset  $d$ ) pair denote the average routing weights of the tokens corresponding to bounding box of examples in dataset  $d$ . We can observe that DAMEX learns to route tokens to their mapped experts and is able to replicate it during inference without the knowledge of dataset-source. On the other hand, vanilla MoE struggles to efficiently utilize experts and suffer with mode collapse at Layer 1.
Effect of Number of Experts on DAMEX. We study the effect of number of experts on DAMEX by experimenting on a mixture of four disparate domain datasets, namely KITTI, DOTA, VOC and Clipart to maximize the effect of dataset-aware experts. Table 6 shows the variation of mean AP score across the four datasets. We can observe that the best performance is obtained when number of experts are same as number of datasets. This observation is different
from vanilla MoEs where performance increases with increase in number of experts [12, 29]. It highlights that DAMEX is an efficient method for mixing datasets both in terms of compute and number of parameters as less number of experts require lesser compute. Note that for 8 experts, we are mapping two experts per dataset with equal probability during training while expert selection during inference is done by the trained router.
Table 6: Effect of number of experts on DAMEX
# Experts	Clipart	DOTA	KITTI	VOC	Mean
2	33.3	44.6	58.9	57.4	48.6
4	34.0	45.3	62.0	57.7	49.7
8	32.6	44.9	60.9	56.5	48.7
In this work, we have presented a novel Dataset-aware Mixture-of-Experts (DAMEX) layer for the object-detection task. We demonstrate the potential of MoE models for mixing datasets on object-detection task. The introduced DAMEX model leverages the MoE architecture to learn dataset-specific features, thereby enhancing expert utilization and significantly improving the performance on a universal object-detection benchmark with marginal increase in parameters. Our experimental results establish a new state-of-the-art performance on UODB by achieving an average increase of  $+10.2$  AP score over the previous baseline and  $+2.0$  AP score over non-MoE method.
7 Limitations & Social Impact
One limitation of our approach is that we are currently concatenating the classes of each dataset which results in a duplicate recognition of overlapping classes. A future work in unifying these labels over our setup can be an interesting avenue.
Going forward it is important that we ensure that the universal object detectors are unbiased and maintain fairness in their predictions. Here unbiased refers to classes with very few training examples or domain shift which can be called underrepresented classes. The detectors should be able to work on both classes with large, medium and few training samples in any environment. Similarly, for fairness, a universal object detector should be able to handle objects from different geographical regions, races, genders, etc. These are issues that we believe are ameliorated by methods focused on learning under imbalance or domain shift.
As a result, we feel DAMEX could be a method that takes a step towards this where each dataset, even with very few samples can be represented by different experts. We can potentially divide a large dataset into different sets which cater to at least one aspect of fairness and bias. However, we understand the need for further analysis and research before we have a truly unbiased and fair universal object detector and implore the community to do so too.
[1] Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly to small image transformations? arXiv preprint arXiv:1805.12177, 2018.
[2] Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, et al. On the representation collapse of sparse mixture of experts. Advances in Neural Information Processing Systems, 35:34600-34613, 2022.
[3] Xiyang Dai, Yinpeng Chen, Bin Xiao, Dongdong Chen, Mengchen Liu, Lu Yuan, and Lei Zhang. Dynamic head: Unifying object detection heads with attentions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7373-7382, 2021.
[4] Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P Bosma, Zongwei Zhou, Tao Wang, Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. GLaM: Efficient scaling of language models with mixture-of-experts. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szeptesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 5547-5569. PMLR, 17-23 Jul 2022.
[5] Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The Pascal visual object classes challenge: A retrospective. International journal of computer vision, 111:98-136, 2015.
[6] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1):5232-5270, 2022.
[7] Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv:2107.08430, 2021.
[8] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pages 3354-3361. IEEE, 2012.
[9] Georgios Georgakis, Md Alimoor Reza, Arsalan Mousavian, Phi-Hung Le, and Jana Košecka. Multiview rgb-d dataset for object instance detection. In 2016 Fourth International Conference on 3D Vision (3DV), pages 426-434. IEEE, 2016.
[10] Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A single model for many visual modalities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16102-16112, 2022.
[43] Haoyang Zhang, Ying Wang, Feras Dayoub, and Niko Sunderhauf. Varifocalnet: An iou-aware dense object detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8514-8523, 2021.
[44] Xiangyun Zhao, Samuel Schulter, Gaurav Sharma, Yi-Hsuan Tsai, Manmohan Chandraker, and Ying Wu. Object detection with a unified label space from multiple datasets. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pages 178-193. Springer, 2020.
[45] Xingyi Zhou, Vladlen Koltun, and Philipp Krahenbuhl. Simple multi-dataset detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7571-7580, 2022.