Hybrid Sharing for Multi-Label Image Classification
Authors: Zihao Yin
Chen Gan
Kelei He
Yang Gao
Junfeng Zhang
Venue: ICLR 2024 poster (2024)
Keywords: Multi-task learning
Multi-label learning
mixture-of-experts
image classification
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Existing multi-label classification methods have long suffered from label heterogeneity, where learning a label obscures another. By modeling multi-label classification as a multi-task problem, this issue can be regarded as a negative transfer, which indicates challenges to achieve simultaneously satisfied performance across multiple tasks. In this work, we propose the Hybrid Sharing Query (HSQ), a transformer-based model that introduces the mixture-of-experts architecture to image multi-label classification. HSQ is designed to leverage label correlations while mitigating heterogeneity effectively. To this end, HSQ is incorporated with a fusion expert framework that enables it to optimally combine the strengths of task-specialized experts with shared experts, ultimately enhancing multi-label classification performance across most labels. Extensive experiments are conducted on two benchmark datasets, with the results demonstrating that the proposed method achieves state-of-the-art performance and yields simultaneous improvements across most labels. The code is available at https://github.com/zihao-yin/HSQ

Published as a conference paper at ICLR 2024
HYBRID SHARING FOR MULTI-LABEL IMAGE CLASSI- FICATION
Zihao Yin Chen Gan Kelei Hel Yang Gao and Junfeng Zhang!
I Medical School of Nanjing University
2 State Key Laboratory for Novel Software Technology, Nanjing University
National Institute of Healtheare Data Science, Nanjing University
Ethao.ying chengan 3 smmall. jju. www..eduuu... edu.cr. [hkl, gaoy, jfzhang) jn edu. cn
ABSTRACT
Existing multi-label classification methods have long suffered from label het- erogeneity, where learning label obscures another. By modeling multi-label classificattin as multi-task problem, this issue can be regarded as a negative transfer, which indicates challenges to achieve simultancously satisfied perfor- mance across multiple tasks. In this work, we propose the Hybrid Sharing Query (HSQ), a transformer-based model that introduces the mifture-of-eeppets archi- tecture to image multi-label classification. HSQ is designed to leverage label correlations while mitigating heterogeneity effectively. To this end, HSQ is in- corporated with fusion expert framework that enables it to optimally combine the strengths of task- sessispeci experts with shared experts, ultimately enhancing multi-labe classification performance across most labels. Extensive experiments are conducted on two benchmark datasets, with the results demonstrating that the proposed method achieves state-of-the-art performance and yields simultaneous improvements across most labels. The code is available at this URL
INTRODUCTION
In computer vision, multi-label classification (MLC) attempts to predict multiple labels that may simultaneously appear in single sample. It is more realistic and intuitive as sample typically has multiple attributes in real scenarios. However, the semantic correlation and heterogeneity among dif- ferent labels pose significant challenge to MI.C, resulting in the labels either complement or con-
flict with each other. Previous works (Liu al. 202 Ridnik et al. 2023 Ye et al.) 2020) achieved impressive performance via transformers or graph neural networks, trying to explore the correlation among labels with shared backbone across labels. These approaches neglected the heterogeneity among labels, which becomes the key obstacle to simultaneous improvement across labels.
In contrast traditional multi-label classification approaches, MLC can be formulated as multi- task learning (MTL) problem by modeling the prediction of each label as an individual task. The correlation and heterogeneity of the labels in MILC thus correspond to the task transfer problem of MTL, where learning new task may perfect (positive transfer) or deteriorate (negative transfer) another. Under this context, the power of MTL in mitigating negative transfer may help improve the performance of MLC.
Precedent works like (Ma et al. 2018) in MTL include mixture of experts (MoE, (Jaobss et al. 199I), which utilizes a group of learned experts to handle different tasks separately. MOE has been widely adopted natural language processing, where experts are expected to process words of various lexical categories. We advocate employing MoE in MLC image classification, which shares commonality with lexical category handling. Furthermore, we notice that the conventional MoF. approach has primarily emphasized the utilization of expert groups within a specific task, with limited attention to the exchange of expertise group knowledge across different tasks. This approach may not align scamlessly with the MLC requirements, which will be scrutinized in our work.
Corresponding author

---

Published as a conference paper at ICLR 2024
In this work, we introduce Hybrid Sharing Query (HSQ), MoE- MLC method with novel proposed fusion strategy to better exploit semantic correlation and heterogeneity among labels and generate better underlying shared representation and task-specific representation. Additionally, prioritize the adaptive fusion of label-specific and shared features in the classification task of each label, suppressing negative transfer and enhancing performance the majority of labels. Specifi- cally, we employ group of shared experts to mine correlation among labels to generate multiple distinct shared features while assigning a group of task-specialized experts to each task to extract series of label-speciff features. This design can balance label-specific and shared features across labels while also emphasizing unique label-specific features for each individual label. Moreover, we employ gate networks to adaptively re- re-weight and harmonize features from task-specialized experts and shared experts, enhancing positive correlations and suppressing negatives among tasks.
Experiments show that the proposed method outperforms all tested baselines across multiple datasets on the majority of labels. The proposed method also compatible with transformer-based MILC methods, indicating potential improvement to existing works.
Our contribution is three-fold:
We present MoE to the MLC task, with gated task-specialized and shared experts to capture cor- relation and heterogeneity adaptively by formulating the ML.C as an MTI. problem.
We empirically demonstrate that the fused experts help to extract correlations between tasks, en- courage positive correlation sharing and suppress negative transfer, which benefits the overall and per-label performance and mitigates cross-label performance gap.
We verify the superiority of our proposed model on two benchmark datasets with state-of-the-art performance overall and per-label.
RELATED WORK
Multi-label classification in computer vision. Models via various approaches have been pro- posed to address MLC. Zhu et al. (2017) use convolutional networks
attention map to op- timize ResNet prediction. Rajpurkar et al. (2017) solve the medical multi-label problem by using DenseNet (Huang et al. 2017). Wang et al. (2016) attempt extract features from the image and generate the label as a sequence through learned joint embedding space. Chen et al. (019a) intro- duce graph convolutional network into this task, mapping label word embedding to inter-dependent object classifiers. Lanchantin et al. (2021a); Liu et al. (2021a) introduce transformer into MLC. These methods fail to see the negative transfer and positive correlation among labels. Some works also notice a similar problem in MLC from the MTL aspect. WW et al. (2019) try to mitigate such problem via a different architecture. Our study aims to improve overall performance in MLC while attempting to simultaneously enhance performance
many labels as possible.
Multi-task learning. MLC can be recognized as special case of MTL, treating each label separate classification task (Wu et al. 2019). Previous works on this topic include hard and soft parameter sharing, etc. Hard parameter sharing (Caruana 1997) comes with shared feature traction backbone as a bottom and task-specialized towers as a top. Soft parameter sharing does not explicitly share network components across tasks but jointly learns other information through gradient sharing or other techniques. Duong et al. (2015): Yang and Hospedales (2017) encourage knowledge sharing across experts via different constraints like L2 norm. Stisc-Stitch (Misra al.) 2016) trains two networks for two tasks and shares gradients between some layers controlled by gates. However, these architectures require more attention to the correlations among tasks, and the naive showledge-sharing strategy may hamper the performance of models. In this work, we propose HSQ to reveal these correlations in the hope generating a better representation for each task.
Mixture of experts deep learning. Efforts have been made to improve models' performance by scaling up the model size with MoE (Jacobs et al. 1991), which first attempts to combine the outputs of several experts with a gate network. MMOE (Ma et al.) 2018) with similar settings further decouples the seesaw phenomenon between several tasks by assigning exclusive gate and tower networks to each task. MOEC (Xie et al. (2023) adopts a clustering loss to impose variance- based constraints the routing stage, obtaining clusters of experts with more diverse knowledge.
2

---

Published as a conference paper at ICLR 2024
PLE Tang al. 2020) adds shared and task-specific experts to MMOE to allow better information sharing between tasks. Traditional MoE imposes a substantial computational burden since all experts tasks required. To mitigate such a cost, the sparse MoE (Shazeer et al. 2017) strategy emerges in contrast to the regular dense one. The routing strategy determines which experts contribute the task output. Zhou al. (2022); Rosenbaum et al. (2018); Nie et al. [021); Zuo et al. (2022); Roller et al. ): Dai et al. (2022) and others explore various routing strategies, including randomizing, hashing, expert-choosing, etc. Switch Transformer (Fedus al, 2022) introduces sparse MoE to the transformer layer to replace the feed-for ferw-forward neural network. Our method introduces the MoE into the multi-laber classification field by virtue of aask-specallized and shared experts exploiting correlations among tasks. Moreover, we utilize gate network to enhance positive correlation and suppress negative correlation pursuit better fusion.
activate, even when only
query
g
Backbone
Transformer Encoder
key vulue
Transformer Decoder
H ybrid Shoring Layer
x N
Kmife fc
Bottle
Pizsa
Pizza Bottle x Knifer
Shared
XB
Learnable Classification Tokens
Task Lask-Spocializat
ngi é shared experts
XRL
ne: # bask-specillized experts
CLS Head
Task- Task-Specian Specialiizzee all
L: labels
shared experts group
task-specialized experts group
Task-Speciallizet
gate network
Figure 1: The architecture of the proposed HSQ. After being extracted by the backbone, the input image's features are then processed by transformer, where learnable classification tokens are used as the query. N Hybrid Sharing Layers are employed sequentially, consisting of L groups of task- specialized experts and group of shared ones. Individual gates for every group control the weighted outputs. A final classification head is utilized to make predictions.
METHOD
The MLC task for images is to find all possible correct labels in pre-defined label set for provided image. Thus, our model takes the provided image I and gives probability scoring L R on all labels C. The proposed HSQ model comprises three main parts, namely, 1) feature extraction back- bone, 2) query transformer, and 3) mixture-of-expert hybrid sharing head. The backbone extracts image representation with a robust replaceable network, followed by transformer. based query model to explore underlying information between such extracted representation and each given la- bel. The Hybrid Sharing Layers are applied to better exploit the correlations between every possible task and suppress potential negative transfer problems.
3

---

Published as a conference paper at ICLR 2024
3.1 FEATURE EXTRACTION BACKBONE
Features in any given image will be extracted through a feature extraction backbone. Multiple preceding works have contributed to this stage. We employ various well-established models to capture global and local feature information within images more effectively. For a 3-channel input image I 3 W; H; and W, are the height and width of an image. feature extractor is applied to extract feature Re E XCxXXxWW where C₁ denotes the number of feature embedding, with a succeeding convolutional layer linearly projecting its feature space from C. into C.
3.2 QUERY TRANSFORMER
The semantic heterogeneity across labels requires the model to discern and capture unique feature representations specific to each individual task. Inspired by the remarkable performance of the query-based classifier, we employ learnable query tokens for classification to mitigate semantic conflicts between tasks. Specifically, this work employs transformer to better extract and wrap task-specific underlying features in class-wise learnable tokens.
Given an extracted image representation R. an encoder-decoder standard transformer applied to inspect features for each label. On the encoder side, the image representation from the backbone is flattened into R E C/W and proceeded by N₂ encoder layers as tokens. To decouple different labels effectively, we endorse Liu et al. (2021a); Lanchantin et al. (2021b) to use learnable tokens as the query. On the decoder side, a learnable token is fed to the transformer decoder as the query for each possible label so that the feature of each label would be learned individually. Na decoder layers are stacked to extract the features of input representations in accordance with each possible label. The decoder layer accepts T E RLXC for every L possible label, where C is the embedding dimension for each token. The cross-attention module in the transformer decoder performs on the query from the learnable label tokens (decoder) and the key and value from the extracted features (encoder), facilitating each label to mine respective representations.
3.3 HYBRID SHARING LAYER
Given the potential semantic correlations among different labels, the features extracted from corre- sponding tasks may exhibit a positive correlation, providing complementary information to enhance model performance. However, improper exploitation of these correlations through learning jointly may cause performance degeneration since parts of these labels conflict with each other semantically due to their inherent heterogeneity, making them hard to learn jointly. To better leverage the positive correlations while suppressing detrimental impact due to heterogeneity between different tasks, introduce the MoE mechanism into the multi-label classification area, inspired by the success of Progressive Layered Extraction (PLE) (Tang et al. 2020). Particularly, employ several shared and task-specialized experts to capture positively correlated features among tasks and task-specific features, respectively, with a gate network adaptively fusing these features. The design of experts and gates can be very flexible and compatible as long as the output shapes are aligned, and in this work, we employ simple but effective linear layers illustrate our approach.
Figure 2 depicts the details of Hybrid Sharing Layers, where L indicates the number of tasks, i.e. the number of labels in multi-label classification. For any task t,, i 21, L), a group of task- specialized experts j E {1, .n,,, is assigned to extract features for this task exclusively, where ny refers to the number of experts for this task. Apart from these task- sssssppecillized experts groups for every task, a group of shared experts Es E 21 ... n. } is responsible for gathering global patterns and dispatching them to those potentially positively correlated tasks. Outputs of each expert group are harmonized by gate network, respectively, so that each task would have customized control on the weight of task- specialiil and shared experts' outputs.
Algorithm I outlines the detailed mechanism of the MoE mechanism that we applied. Let X, N Lxd, be a batched input to the mixture-ofeexpert layer, where N refers to batch size and d; means input embedding dimension. And let X, be the input for shared experts group with the exact same shape as X,. The outputs of a Hybrid Sharing Layer comprise task-specialized outputs and a shared output.
In the task task-speci section, each label (task) is processed independently. For a batched input task 1, a of task-specific experts, denoted as Fi... 1Rdxxd is utilized, X E NNx xd,
4

---

Published as a conference paper at ICLR 2024
Gh
G
GiL
weight
Shared experts cuippttt to Taskl gale
Shared experts outpuis Task fy galle
Task specialiidl Haperts
Shared Esperts
Sppciiicci Experts
Taski experts ourpats to shared gate
Task experts octputs to shared gate
Σs,1
Ess...
E s
Shared inputs
Shared inputs Tisk ft expents
Task] experts
Satted experts input & ourput
Task specidlized experts for task imput & ouiput
Taseel inputs Shared experts
Task inputs Shared experts
Task-specihlized experts for task fy. imput ourput
Figure 2: The Architecture of Hybrid Sharing Layer. For L labels, the layer consists of groups of lask-specialized experts and group of shared experts. The detailed structure of the shared experts is illustrated on the right.
where j represents the j-th expert in the group. Yt,t... Xx, Et.j Nxd, represents the out- put of expert Et, on task 4. The subscript of Y, separated by refers the output task and expert subscript, respectively, meaning that it is the output of the j-th task-specialized expert in &; and takes inputs from task t.. Similarly, group of shared experts, denoted Ess IRd.xd, is used for task t,. Shared experts, which would be used in all tasks, also accept X₁, in task t, and Yesss Es, E RNx, represents the output of expert Esj on task t,. gate network, denoted as G., E (R4x(L-mctt),, is employed to produce weights for outputs from all shared and task-specialized experts on task 4. The gate network takes Xx, as input, and outputs Softmax( Xx, Gt,)
(=+-x))) as the weights for experts' outputs. Here, ne task-specialized experts for tú and all n. shared ones are employed. The task output is a weighted mean of all experts with activation o, as described in the following equations, where (k) stands for tensor indexing.
Y, = Concat (alre Yn,los) E R xxdox((menn)))
O2 = Σ (Y.)) (k) o Softmax(Xx G1,) E Nxd.
In the shared section, all shared experts E, and task-specialized experts E,, are utilized to gather potential features, with total of n, L n4 experts. These experts use shared input X, as their input. Similar to the task-specialized part, a gate fuses shared and task-specialized features. The shared gate network, denoted as G., harmonizes the outputs from both shared experts and task- specialized experts across all tasks with weights derived from the shared input X.. Algorithm described shared and task speciiaicd parts in the Hybrid Sharing Layer
Y, Concal (Yaley) Vele,,) ) R Xddx (doxx(
O, = Σ (o(Y,) (k) o Softmax(X,Gss) E Nxdo
It is worth noting that the shared and task-specialized parts receive the outputs from their respective parts in the previous layer as inputs, except the initial layer, which uses an identical input.
EXPERIMENT
We have performed extensive experiments two datasets, MS-COCO and PASCAL VOC, to verify the superiority of our model. In accordance with the preceding works, we choose mean average precision our primary metric. Some experiments also report some secondary metries. including overall Fc-sc (OF1) and pre-category Fi-score (CFI). Metrics on Top-3 are also reported. The definitions of these metries are available in the Appendix.
5

---

Published as a conference paper at ICLR 2024
Algorithm 1: Hybrid Sharing Layer Proce-
Cow Horse AP(%) Difference on VOC
10
dure
Data:
Input shared experts X,; Input task fy experts X Shared expert gate Gx; Task i, expert gate G Number shared experts Rg, Number task specialieede experts per task ny: Shared experts Ex,o): Task- specialzid experts (-)5 Number of labels I, Activation function
9
8
7
AP Defeeence
6
5
4
3
Result: Shared output Os; Task-specialized oulpul O,,
2
Y==11 for i + 1 to do | V,, =1 =1] end
1
0
CNN ANN
+ov+
HCP
RDAL
RAPL
SSGAL
MCAR
GGW
7Ros.
HSQ
for i +- to do
VGG
ADD
ozl.
for 1 do
Y Your X, Egcd
Figure 3: Absolute AP performance difference be-
apppeed ale.d)
tween cow and horse VOC dataset in (%)
Y.. Xe, Beid
Bike Motobile AP(%) Differance vOC
Y, w..append(y......
end end
for
n.
3
Ma, X, Eig.
Difforence (80)
a.appnd sts, for - 1 to do
2
Y Ve = X,, E.,) Y, appapedd
end
1
end
A. + Softmax Softmax(G,X,, (G,
Y Concat(Y, Y,)
O, < A. A.
(Ya)
for +- 1 to do
CNN RNW
VGG SVM
+er
HCP
ROAL
RARL
SSGAL
MCAR
ASL
GCN
TReat.
HSQ
Yi, Concat(Y,,
AG Softmax (1.)
ADD
On, & A, o (Y,))
Figure 4: Absolute AP performance difference be- tween bike and motobike on VOC dataset in (%)
end
4.1 ABLATION STUDY
As shown in Table 1 the number of shared experts greatly influences the performance. We choose ResNet1OT as the backbone and Q2L with the same backbone as the baseline. An ablation study is performed MS-COCO. The input image is fixed at size of 576 x 576. The proposed model, which includes shared experts (HSQ), outperforms the baseline by 1.3% on mAP, demonstrating that including shared experts facilitates the transfer of information between tasks and mitigates negative transfer. The results also reveal that removing shared experts from the model leads to a considerable drop in performance due the complete cutoff of sharing information among all tasks, underscoring the importance of sharing features in achieving substantial performance improvements. HSQ-L Linear indicates that the hybrid sharing layers of the model are replaced by fully-comnected layers with the same depth and dimensions, sharing all information across all labels without discriminating task- specialized information. in enhancing the performance of the proposed model compared with HSQ-Linear. The findings highlight the potential benefits incorporating shared experts and can inform the development of future multi-label image classification models.
demonstrated that the inclusion of shared experts crucial factor
4.2 PERFORMANCE ON THE MS-COCO DATASET
MS-COCO (Lin et al. 2014 is large dataset of 80 object classes originally for image segmentation and object detection tasks. By extracting object information annotations, is also widely used evaluate various models for multi-label image classification tasks. We test our model on MS-

---

Published as a conference paper at ICLR 2024
Table 1: Ablation Study on MS-COCO. ns. R4 stand for the number of shared experts and task- specialized experts per task, findicates that is not available in the original work and we implement in this paper.
Method
Backbone
Resolution
nt
mAP(%)
Q2L-R1OT et al. HSQ-KTOT(Our))
2021a)
ResNer10T
576 x 576
74.8
ResNet10T
576 576
0
1
71.5
HSQ-Linear HSQ-R (OT())rr)
ResNet10T
576 x 576
75.7
ResNet1OT
576 576
1
76.3
HSQ-RIOTTC (O0TOOsSS
ResNet10T 576 576
4
1
76.1
HSQ-R (Ours
ResNet1OT 576 576
16
1
76.5
COCO compare it with previous well-known works and state-of-the-an approaches. Results We use ResNet101 (He et al., 2016) and ConvNeXt (Liu et al.] (Vv2) as the backbone and input resolution to 576 576. Those backhones noted with -22k indicate that they are pre-trained on ImageNet-22k. Our HSQ model with CvN as the backbone achieves state-of-the-art performance model outperforms all its counterparts. HSQ-R101 at the resolution of 576 x 576 achieves an mAP of 87.1%. Please that for this model, we employ successive Hybrid Sharing Layers of d. = 1024, 512, MLP with one hidden layer of 128 neurons as gate and one of 64 as classifier.
shown in Table 2
mAP of 92.0%. Among all ResNet10 1aat110-bbee approaches,
Table 2: Performance (%) on MS-COCO. Bests are in bold. + indicates that it is not available in the original work and we implement in this paper.
All
Top3
Method
Backhone
Resolution
mAP
CFI
OFI
CFI
OFI
SRN 7hu
2017
ResNet101
224
224
77.1
71.2
75.8
67.4
72.9
ResNet- IUI He 2016
ResNet101
224
78.3
72.8
76.8
69.7
73.6
CADM (Chn
ResNet101
448 448
82.3
77.0
79.6
73.5
76.0
ML-GCN Chen
9019
ResNet101
448 448
83.0
78.0
80.3
74.2
76.3
KSSNet [Lill et al.
2018
ResNer101
448 448
83.7
77.2
81.5
MS-CMA You cl 2020
ResNet101
448 x 448
83.8
78.4
81.0
74.3
77.2
MCAR (Gao and Zhou 2021
10s
448 x 448
83.8
78.0
80.3
75.1
76.7
SSGRL Chen ct al. 2019c
ResNet101
576 x 576
83.8
76.8
79.7
72.7
76.2
C-Trans Lanchantin et il 2021a
ResNet101
576 x 576
85.1
79.9
81.7
76.0
77.6
ADD-GON (ee el al. 2020
ResNet
576 x 576
85.2
80.1
82.0
75.8
77.9
Q2L-R101 (Lau
202
ResNet101
448 x 448
84.9
79.3
81.5
73.3
75.4
Q2L-R101 (L.0 ct
202
ResNet101
576 x 576
86.5
81.0
82.8
76.5
78.3
SST (Chen el
2020
ResNet101
448
85.9
80.2
82.2
76.0
77.9
ResNet101+TF [Xu
2022
ResNet101
576 576
85.9
80.3
82.4
PSD+TF Xu al.
ResNet101
576 576
86.7
81.2
82.9
SCO-DCNN Zhang al. 2023
ResNet101
576 576
86.0
79.8
83.0
(SQrs)
ResNet101
576
87.1
81.8
83.4
91.8
93.4
ASL (Ridnik et al.
2021a
TResNetL
448 x 448
86.6
81.4
81.8
75.1
77.4
1ResNelL
TResNetL(22k)
448 x 448
88.4
Q2e--ResI.
(Liu
TResNetI.
448 x 448
87.3
81.6
83.1
77.0
78.5
Q2L-TResL
Liu at.
20218
TResNetL(22k))
x 448
89.2
83.8
84.9
79.0
80.2
MITr-]
Cheng el
2022
MLTr-I(22k)
384
88.5
83.3
84.9
Swin-L.
Swin-L(22kk
384
89.6
84.8
86.1
80.0
CvT-w24
Wu
CVT-w24(22))
384 384
90.5
85.4
86.6
80.3
81.3
Q2L-Swinl.
Lau
2021a
Swin-L(22k)
384 x 384
90.5
85.4
86.4
80.5
81.2
Q2L-CvT
Luu
20218
CVT-w24(22K)
384 x 384
91.3
85.9
86.8
80.8
81.6
ML-Decoder
(idmk al. 2023
TResNet-XL(Open Image)
640 x 640
91.2
76.8
76.9
90.8
92.0
HSQ--vNN(Ours)
ConvNeX:(22k)
576 576
92.0
86.6
87.5
94.0
95.2
4.3 PERFORMANCE ON THE VOC DATASET
2007 is also well-acknowledged dataset for multi-label image classification. comprises images of 20 classes and is split into train-val and test sets. We follow previous work to train train-val set and validate test set 2007 version. The results of
PASCAL- VOC (Everingham et al. 2015

---

Published as a conference paper at ICLR 2024
Table 3: Performance (%) on VOC, in terms of per-label AP and mAP. Bests are in bold.
Methods
acro
bike
bird
boat
bottle
bus
car
cat
chair
cow
mAP
UNN-RNN (Wang et al.
2016
96.7
83.1
94.2
92.8
61.2
82.1
89.1
94.2
64.2
83.6
84.0
VGG+SVM
(Simonyan and Zisserman
2015
98.9
95.0
96.8
95.4
69.7
90.4
93.5
96.0
74,2
86.6
89.7
Fev+FlV Yang al.
2016
97.9 97.0
96.6
94.6
73.6
93.9
96.5
95.5
73.7
90.3
90.6
HCP Wet al. 2015
98.6 97.1
98.0
95.6
75.3
94.7
95.8
97.3
73.1
90.2
90.9
RDAL Wang et Al. 2017
98.6 97.4
96.3
96.2
75.2
92.4
96.5
97.1
76.5
92.0
91.9
RARL Chen al 2018
98.6 97.1
97.1
95.5
75.6
92.8
96,8
97.3
78.3
92.2
92.0
SSGRL((56] (Chen ct ii
2019c
99.7 98.4
98.0
97.6
85.7
96.2
98.2
98.8
82.0
98.1
95.0
MCAR (Gao and Zhou 2021
99.7 99.0
98.5
98.2
85.4
96.9
97.4
98.9
83.7
95.5
94.8
ASL(TResNeIL)) Kidnik el al. 20
99,9 98.4
98.9
98.7
86.8
98.2
98.7
98.5
83.1
98.3
95.8
(57-) Ye ct 2020
99.8 99.0
99.0
86.7
98.1
98.5
98.3
85.8
98.3
96.0
Q2L-TResL (Liu el 2021a
99.9
98.9
99.0
98.4
87.7
98.6
98.8
99.1
84.5
98.3
96.1
HSQ-CvN(22R)
99.9
99.9
97.2
99.4
84.1
99.1
98.3
99.1
84.9
100,0
96.4
Methods
table
dog
horse
mbike
person
plant
sheep
sofa
train
mAP
CNN-RNN (Wane al.
2016
70.0
92.4
91.7
84.2
93.7
59.8
93.2
75.3
99.7
78.6
84.0
VGG+SVM (Simonyan
Ziserman
2015
87.8
96.0
96.3
93.1
97.2
70.0
92.1
80.3
98.1
87.0
89.7
Fev+Ly Yang al. 2016
82.8 95.4
97.7
95.9
98.6
77.6
88.7
78.0
98.3
89.0
90.6
IICP
Wet el al 2013
80.0 97.3
96.1
94.9
96.3
78.3
94.7
76.2
97.9
91.5
90.9
RDAL Wang ct a. 2017
87.7 96.8
97.5
93.8
98.5
81.6
93.7
82.8
98.6
89.3
91.9
RARL Chen el al 2018
87.6 96.9
96.5
93.6
98.5
81.6
93.1
83.2
98.5
89.3
92.0
SSGRL(S76J Cien al
2019cc
89.7 98.8
98.7
97.0
99.0
86.9
98.1
85.8
99.0
93.7
95.0
MCAR (Gao and Zhou
2021
88.8 99.1
95.1
99.1
84.8
97.1
87.8
98.3
94.8
94.8
ASLsTTRsNNILL) Ridmik cl
212
89.5 98.8
99.2
98.6
99.3
89.5
99.4
86.8
99.6
95.2
95.8
ADD-GCNN5700T Ye ct al. 2020)
88.9 98.8
99.0
97.4
99.2
88.3
98.7
90.7
99.5
97.0
96.0
Q2L-TResL. (Lii crai. 20218
89.2
99.2
99.2
99.2
99.3
90.2
98.8
88.3
99.5
95.5
96.1
HSQ-CVN(22RR
91.2
99.2
99.9
99.9
99.2
88.0
100.0
91.6
99.8
97.8
96.4
experiments on it are displayed in Table Our model is compared against various established meth- ods and state-of-the-ar techniques. Notably, our proposed approach surpasses all its counterparts, achieving an impressive mAP score of 96.4%. The per-class average precision is also presented, with the SOTA performance bolded. Items that improve in comparison with previous works underscored in the last two rows.
Performance among Labels and Cross-label Comparison Among the 20 available labels, our model exhibits superior performance in 15 of them. Compared to Q2L, another transformer-based model, our model improves 105 pairs of labels, 27 pairs more than Q2L achieved. To provide a visual comparison, we randomly select two labels with moderate performance (i.e.. "table" and "train") and illustrate them in Figure 7 In this graph, each dot represents a specific approach, with dots in the upper-right corner indicating better performance. We further explore the performance difference between two pairs of labels with similar semantics, as depicted Figure and Our method not only outperforms previous work but also exhibits smaller absolute cross-label performance gap.
Robust Performance across Multiple Image Scales. In addition to previous experiments on MS- COCO, we perform extra experiments on different image scales to verify the performance of our model improves as image resolution decreases in Figure 5 We perform experiments on 576 x 576, 488 x 448 and 384 x 384. Results confirm that our model provides consistent considerable performance as image scale decreases.
4.4 VISUALIZZATION RESULT
The proposed model incorporates several gates to harmonize outputs from experts. We verify that different tasks would rely on different experts on PASCAL VOC. Figure 5 depicts the weights of experts' outputs all 20 tasks and the average load across tasks on a sampled data batch. The last row represents an expert's 's average load across all tasks. Weights are gate networks' outputs, presented in log scale, along 33 different experts on the X-axis, 32 of which are shared, and the last one is respective ssssisppcialized. A lighter block color indicates an expert with more weight in the final harmonized output. It is evident to note that all tasks focus on different experts. For instance, E,,. E,,00, Es,2 have the most significant impact on chair, dog and horse respectively. The weight distributions across experts exhibit variations among tasks, indicating that
values of the
8

---

Published as a conference paper at ICLR 2024
distinct tasks rely different sets of experts, each extracting distinctive representations, while even average loads on experts show that all experts are engaged during inference.
dero
bike-
bird-
bout
bottle
100
out
chair
-00
cow-
tobie
doy
horse
10
m mbike-
per
sherp
10-1
sofe f
train
to
mean
17 18 22 23 28
Figure 5: Experts load visualization on 20 labels of VOC2007. Each block indicates a weight for an expert on one task. The first 20 rows represent 20 labels from the VOC dataset, and the last row stands for the average load of experts. The x-axis denotes different experts, where experts 1-32 are shared among all tasks, while expert is task-specific. The color represents weight log space.
92
Q2Li
SSQ
CMN RMN
91.0
ADD GON (Res.
stAP(X))
90.5
00.5
99
SOGAL
89.7
89.3
ADAL
89
VVM SVM
HCP
576
448
354
image scale
Figure 6: Performance comparison be- tween Q2L (iu et al. 2021a) and HSQ on MS-COCO with backbone Con- vNext (Liu et al. 2022
Figure 7: AP on VOC2007 (table and train). The upper-right points in the fig- ure perform better. AP is in %
CONCLUSION
In this paper, regarding MLC as MTL problem, introduce HSQ, transformer-based multi- label image classification model, which is constituted of a feature extraction backbone, query trans- former, and Hybrid Sharing Layers that provide evident information sharing among tasks with shared and task-specialized experts leveraging inter- and intra-task information, respectively. Task- specialized experts are organized by respective gate networks, allowing each task to accept corre- lated information from shared experts independently. Shared experts accept input from all tasks, fusing all potentially useful information. Our model mitigates the negative transfer problem MLC when formulating mance improvement. Our experiments demonstrate that HSQ provides a significant improvement tested datasets. Furthermore, HSQ simultaneously enhance per-label performance across multiple labels, mitigate performance gap among labels, and effectively handle semantic correlation and heterogeneity.
MTL problem, where learning several labels jointly may hinder perfor-

---

Published as a conference paper at ICLR 2024
Reproducibility Statement In this paper, make efforts to provide detailed information to sure the reproducibility and completeness of our work. Figure [ iillustrates the architecture of our model. Algorithm | and Figure A provide clear overview and procedure for crucial compo- nent, the Hybrid Sharing Layer. Section A.3 in the Appendix describes the hyper-parameters and devices use, including optimizer, learning rate, etc. Section we prepare our dataset, including version, partitioning strategy, etc. The code will be available upon acceptance.
describe details how
ACKNOWLEDGMENTS
This work is partially supported by the National Natural Science Foundation of China (grant 62106101), and the Natural Science Foundation of Jiangsu Province (grant no. BK20210180). This work is also partially supported by the AI & Al for Science Project of Nanjing University.
REFERENCES
Shilong Liu, Lei Zhang, Xiao Yang, Hang Su, and Jun Zhu. Query2label: A simple transformer way to multi-label classification, 2021a.
Tal Ridnik, Gilad Sharir, Avi Ben-Cohen, Emanuel Ben-Baruch, and Asaf Noy. MI-decoder: Scal- able and versatile classification head. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 32-41, 2023.
Jin Ye, Junjun He, Xiaojiang Peng. Wenhao Wu, and Yu Qiao. Attention-driven dynamic graph convolutional network for multi-label image recognition. In Computer Vision-ECCo 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16, pages 649- 665. Springer, 2020.
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi. Modeling Task Rela- tionships in Multi-task Learning with Multi-gate Mixture-of-Experts ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '18, pages 1930-1939, New York, NY, USA, July 2018. Association for Computing Machinery.
Proceedings of the 24th
Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79-87, 1991.
Feng Zhu, Hongsheng Li, Wanli Ouyang, Nenghai Yu, and Xiaogang Wang. Learning spatial reg- ularization with image-level supervisions for multi-label image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5513-5522, 2017.
Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, al. Chexnet: Radiologist-evel pneumo- nia detection on chest x-rays with deep learning. arXiv preprint /711v:1711.0000 2017.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
Jiang Wang, Yi Yang. Junhua Mao, Zhiheng Huang, Chang Huang, and Wei Xu. Cnn-rnn: A unified framework for multi-label image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2295-2294, 2016.
Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo. Multi-label image recognition with graph convolutional networks. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 5177-5186, 2019a.
Jack Lanchantin, Tianlu Wang, Vicente Ordonez, and Yanjun Qi. General multi-label image classi- fication with transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16478-16488, 2021a.
Sen Wu, Hongyang R Zhang, and Christopher Ré. Understanding and improving information trans- fer in multi-task learning. In International Conference on Learning Representations, 2019.
10

---

Published as a conference paper at ICLR 2024
Rich Caruana. Multitask Learning. Machine Learning, 28(1):41-75, July 1997. ISSN 1573-0565.
Long Duong, Trevor Cohn, Steven Bird, and Paul Cook. Low resource dependency parsing: Cross- lingual parameter sharing in neural network parser. In Proceedings of the 53rd annual meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (volume short papers), pages 845-850, 2015.
Yongxin Yang and Timothy Hospedales. Trace norm regularised deep multi-task learning. In 5th International Conference on Learning Representations, 2017.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hehert. Cross-stitch networks for multi-task learning. In Proceedings the IEEE conference on computer vision and pattern recog- nition, pages 3994-4003, 2016.
Yuan Xie, Shaohan Huang, Tianyu Chen, and Furu Wei. Moec: Mixture of expert clusters. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13807-13815, 2023.
Hongyan Tang, Junning Liu, Ming Zhao, and Xudong Gong. Progressive layered extraction (ple): A novel multi-task learning (mtl) model for personalized recommendations. In Proceedings of the 14th ACM Conference on Recommender Systems, pages 269-278, 2020.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously Large Neural Networks: The Sparsely-Gated cetture-of-Expertt Layer. In International Conference on Learning Representations, 2017.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103-7114, 2022.
Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. In International Conference on Learning Represen- tations, 2018
Xiaonan Nie, Xupeng Miao, Shijie Cao, Lingxiao Ma, Qibin Liu, Jilong Xue, Youshan Miao, Yi Liu, Zhi Yang, and Bin Cui. Evomoe: An evolutional mixture-of-experts training framework via dense-to-sparse gate. arXiv preprint arXiv:2112 14397, 2021.
Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Tuo Zhao, and Jianfeng Gao. Taming sparsely activated transformer with stochastic experts. International Conference on Learning Representutions, 2022.
Stephen Roller, Sainbayar Sukhbaatar, Jason Weston, et al. Hash layers for large sparse models. Advances in Neural Information Processing Systems, 1455555--7756, 2021.
Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, and Furu Wei. Stable- moe: Stable routing strategy for mixture of experts. In Proceedings of the 60th Annual Meeting of the Association for Computeational Linguistics (Volume I: Long Papers), pages 7085-7095, 2022.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1): 5232-524 5270, 2022.
Jack Lanchantin, Tianlu Wang, Vicente Ordonez, and Yanjun Qi. General multi-label image classi- fication with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16478- 16488, 2021b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Ziticck. Microsoft coco: Common objects in context. In Computer Vision Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.

---

Published as a conference paper at ICLR 2024
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
Zhuang Liu, Hanzi Mao, Chao- Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11976-1 1986, 2022.
Zhao-Min Chen, Xiu-Shen Wei, Xin Jin, and Yanwen Guo. Multi-label image recognition with joint class-aware map disentangling and label correlation embedding. In 2019 IEEE International Conference on Multimedia and Expo (ICME), pages 622-627. IEEE, 2019b.
Yongcheng Liu, Lu Sheng, Jing Shao, Junjie Yan, Shiming Xiang, and Chunhong Pan. Multi-label image classification via knowledge distillation from weakly-superrised detection. In Proceedings of the 26th ACM International Conference on Multimedia, pages 700-708, 2018.
Renchun You, Zhiyao Guo, Lei Cui, Xiang Long, Yingze Bao, and Shilei Wen. Cross-modality attention with semantic graph embedding for multi-label classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12709-12716, 2020.
Bin-Bin Gao and Hong- Yu Zhou. Learning discover multi-class attentional regions for multi-label image recognition. IEEE Transactions on Image Processing, 30:5920-5932, 2021.
Tianshui Chen, Muxin Xu, Xiaolu Hui, Hefeng Wu, and Liang Lin. Learning semantic-specific graph representation for multi-label image recognition. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 522-531, 2019c.
Zhao-Min Chen, Quan Cui, Borui Zhao, Renjie Song, Xiaoqin Zhang, and Osamu Yoshie. Sst: Spatial and semantic transformers for multi-label image recognition. IEEE Transactions on Image Processing, 31 31:2570-2583, 2022.
Jaazhi Xu, Sheng Huang, Fengtao Zhou, Luwen Huangfu, Daniel Zeng, and Bo Liu. Boosting multi- label image classification with complementary parallel delf-distillation. In Lud De Raedt, editor, Proceedings of the T'hittyFFrrst International Joint Conference on Artificial Intelligence, IJCAI- 22, pages 1495-1501. International Joint Conferences on Artificial Intelligence Organization, 2022.
Jialu Zhang, Jianfeng Ren, Qian Zhang, Jiang Liu, and Xudong Jiang. Spatial context-aware object- attentional network for multi-label image classification. IEEE Transactions on Image Processing, 2023.
Tal Ridnik, Emanuel Ben-Baruch, Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, and Lihi Mannik-Manor. Asymmetric loss for multi-label classification. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 82-91, 2021a
Xing Cheng, Hezheng Lin, Xiangyu Wu, Dong Shen, Fan Yang, Honglin Liu, and Nian Shi. Mltr: Multi-label classification with transformer. 2022 IEEE International Conference on Multime- dia and Expo (ICME), pages 1-6. IEEE, 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 10012-10022, 2021b.
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 22 31, 2021.
Mark Everingham, SM Ali Eslami, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision 111 11:98-11 136, 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale im- age recognition. In International Conference on Learning Representations. Computational and Biological Learning Society, 2015.
12

---

Published as a conference paper at ICLR 2024
Hao Yang, Joey Tianyi Zhou, Yu Zhang, Bin-Bin Gao, Jianxin Wu, and Jianfei Cai. Exploit bounding box annotations for multi-label object recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 280-288. 2016.
Yunchao Wei, Wei Xia, Min Lin, Junshi Huang, Bingbing Ni, Jian Dong, Yao Zhao, and Shuicheng Yan. Hep: flexible enn framework for multi-label image classification. IEEE Transactions on Patlern Analysis and Machine Intelligence, 38(9): 1901-1907, 2015.
Zhouxia Wang, Tianshui Chen, Guanbin Li, Ruijia Xu, and Liang Lin. Multi-label image recog- nition by recurrently discovering attentional regions. In Proceedings of the IEEE International Conference on Computer Vision, pages 464-472, 2017.
Tianshui Chen, Zhouxia Wang, Guanbin Li, and Liang Lin. Recurrent attentional reinforcement learning for multi-label image recognition. In Proceedings of the AAAI conference on Artificial Intelligence, volume 32, 2018
Tal Ridnik, Emanuel Ben nen-Baruch, Asaf Noy, and Lihi Zelnik-Mand Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2 10972, 2021b.
A APPENDIX
A.1 METRICS DEFINITION
We here provide the definitions of metrics that are mentioned in our paper.
Pre-Category Precision: CP
1 N
Σ
C
N
1
N
Pre-Category Recall: CR =
Σ
C
N
x CP CR
Pre-Category FI score: CF1
CP CR
Σ, N
Overall Precision: OP
E, N
E, No
Overall Recall: OR
N
2 x OP x OR
Overall Fl score: OF1
OP + OR
where O' stands for the number of labels, N refers to the number of samples that are correctly predicted for the i-th label, N₁" denotes the number of predicted samples for the i-th label, and N means the number of ground truth samples for the i-th label.
A.2 FLOPS AND #PARAMETERS DETAILS
Model
Backbone
Resolution
FLOPs
#Parameters
Q2L-RI0TY (Liu et al. 202 HSQ (n, = =1, R4 I)
10sNet10T
576
8.4G
14.8M
10T
576
8.6G
14.3M
HSQ (n, 16, nt
ResNet10T
576
8.7G
15.5M
A.3 IMPLEMENTATION DETAILS
Unless otherwise stated, the following setting is valid for all experiments. We resize all input images from any dataset to H, x W; 576 x 576. After pre-trained backbone with timm, a convolutional layer projection would keep the embedding size C Ca 2048 as default.
13

---

Published as a conference paper at ICLR 2024
Two layers of mixture-of-expert layers with output embedding dimension d, 64, 16 are imple- mented after a transformer with one encoder layer and two decoder layers, followed by linear layer to make final prediction.
We train the model for 100 epochs using the Adam optimizer, with weight decay of 1e-2, (31, 32) (0.9, 0.9999), and learning rate of le-4. All experiments The pre-training details of the experiments are provided in parentheses. We, by default, do not train our model extra data, except if it is otherwise stated. Note that models with -22k indicate that they use a pre-trained backbone on ImageNe122k (Ridnik et al. (021b). Additionally, experiments marked with f indicate that they have been replicated in this work.
4 Tesla 1100-SXM2-32GH
14


Reviews:
[{"review_id": "zhRrC1RCpR", "reviewer": "Reviewer_2H28", "summary": "The paper addresses the issue of heterogeneity in the multi-label learning process. It transforms multi-label tasks into multi-task learning and innovatively incorporates the MoE model into visual classification tasks. Drawing inspiration from models like MMoE and PLE, the authors employed multiple experts and gating theories to learn various labels in their experiments. To prevent negative transfer during the model learning process, they also employed a shared expert approach to learn label correlations. Finally, the authors conducted experiments on the VOC and MS-COCO datasets and achieved outstanding performance.", "questions": "1. What is the significance of Figure 6? The reduction in image resolution implies a decrease in information, which may lead to a decline in the model's performance. How does this relate to robustness?\n2. What is the difference between hybrid experts and attention mechanisms? Is the model's good performance due to the introduction of a large number of parameters? Does MoE in the paper improve training speed? It is recommended to conduct an efficiency experiment to verify this question.\n3. MoE models generally encounter the issue of load balancing. Is there a possibility that certain experts consistently dominate during the model training process? It is recommended to conduct an experiment to verify this question.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Innovatively introducing the MoE model from the NLP field into the domain of computer vision, the paper achieved outstanding results. The use of gating and expert effectively enhanced the model's capacity for relationship modeling in multi-label tasks.\n2. The paper's algorithmic description is concise and clear, with pseudo-code illustrating the core model workflow.\n3. The experiments are comprehensive, including model comparisons, ablation studies, and visualizations.", "weaknesses": "1. In the ablation experiments section, there is a lack of explanation for the decrease in experimental performance, particularly why the performance declines when n_t is 1 and n_s is 0, and why is there not a more in-depth exploration of the impact of the quantities of n_s and n_t on the results?\n2. There is a lack of visualization of how different experts in the model process images.\n3. Contribution 1 and Contribution 2 appear quite similar. The experiments on heterogeneity are not sufficiently intuitive, why is it solely demonstrated through experiments rather than being theoretically proven? \n4. The paper uses extensive textual descriptions in the methodology section; using formulas would provide a more concise representation.\n5. Although the paper incorporates the MoE model, it has relatively few innovative aspects of its own.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9jHxIItmvZ", "reviewer": "Reviewer_rQYR", "summary": "In this work, the authors regard the multi-label image recognition task as a multi-task problem. To address the \"label heterogeneity\", the authors propose a transformer-based model, Hybrid Sharing Query (HSQ), that introduces the mixture-of-experts architecture to multilabel classification, which leverages label correlations while mitigating heterogeneity effectively. As presented in their experiment results, the proposed framework achieves state-of-the-art performance, which demonstrates the effectiveness of the proposed method.", "questions": "Please see the above weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. It is novel that presenting a \"mixture of experts\" to the multi-label image recognition task, which is ignored in previous MLR works.\n2. The design of the proposed framework is technically clear, and the experiment results demonstrate its effectiveness.", "weaknesses": "1. As the core motivation, the authors should provide a detailed and comprehensive discussion about \"label heterogeneity\" in MLR.\n2. In the hybrid sharing layer, what is the difference between the task-specialized experts group and the semantic features in other MLR works? And what is the shared experts group? These are crucial for understanding this work.\n3. Could you provide some visualizations of the impact of the shared experts group? Considering the claim \"shared experts help to extract correlations between tasks, encourage positive correlation sharing and suppress negative transfer\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gc9P5F8n1v", "reviewer": "Reviewer_7ihD", "summary": "This paper presents a transformer-based approach with mixture-of-experts design for multi-label classification task.  The approach is design for better leveraging semantic correlation and heterogeneity among labels. The experiments on Pascal Voc and MSCOCO datasets show the effectiveness of the propose approach and get SoTA performance.", "questions": "Please answer the questions in [weaknesses].", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "+ The paper offers valuable insights on multi-label classification, the solution based on MoE is a pretty novel view.\n\n+ The approach achieves good results.\n\n+ The paper is well-organized and have clear figures and demonstrations.", "weaknesses": "The main concern I have is in the experiment section.\n\n1). The author missed some important experiments on MS-COCO.  The latest paper chosen for comparison is the ML-Decoder. However, the author didn't choose the same model backbone and the same image resolution to make a fair comparison. Besides, the model backbone pretraining details can also be an important factor in the model performance, so what's the difference between these methods?.\n\n2). The ML-Decoder's code is available for years, the author should also fill the blanks in (CF1 OF1 CF1 OF1).\n\n3). The ML-Decoder's paper and code is publicly available on arxiv at 2021, they didn't make any change to their best results even when they published to WACV 2023. This means all the results you compared with are before or in 2021, so why not choose some latest methods in 2022 and 2023 for comparison?\n\n4). Please indicate the Flops and Parameters in the experiments and compared with the mentioned methods.\n\n5). MSCOCO is a relatively small dataset with well annotated labels in the multi-label classification task. Why not using NUS-WIDE or OpenImages Dataset which are stronger benchmark and can also show the robustness of the model under the case with more labels and more data?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)