Disambiguated Attention Embedding for Multi-Instance Partial-Label Learning
Authors: Wei Tang
Weijia Zhang
Min-Ling Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: Machine Learning
Multi-Instance Partial-Label Learning
Multi-Instance Learning
Partial-Label Learning
TL;DR: We propose the first multi-instance partial-label learning algorithm based on the embedded-space paradigm.

Abstract: In many real-world tasks, the concerned objects can be represented as a multi-instance bag associated with a candidate label set, which consists of one ground-truth label and several false positive labels. Multi-instance partial-label learning (MIPL) is a learning paradigm to deal with such tasks and has achieved favorable performances. Existing MIPL approach follows the instance-space paradigm by assigning augmented candidate label sets of bags to each instance and aggregating bag-level labels from instance-level labels. However, this scheme may be suboptimal as global bag-level information is ignored and the predicted labels of bags are sensitive to predictions of negative instances. In this paper, we study an alternative scheme where a multi-instance bag is embedded into a single vector representation. Accordingly, an intuitive algorithm named DEMIPL, i.e., Disambiguated attention Embedding for Multi-Instance Partial-Label learning, is proposed. DEMIPL employs a disambiguation attention mechanism to aggregate a multi-instance bag into a single vector representation, followed by a momentum-based disambiguation strategy to identify the ground-truth label from the candidate label set. Furthermore, we introduce a real-world MIPL dataset for colorectal cancer classification. Experimental results on benchmark and real-world datasets validate the superiority of DEMIPL against the compared MIPL and partial-label learning approaches.

Disambiguated Attention Embedding for Multi-Instance Partial-Label Learning
Wei Tang .2, Weijia Zhang?, Min-Ling Zhang
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China 2 Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China
School of Information and Physical Sciences, The University of Newcastle, Callaghan, NSW 2308, Australia
tangwSseu. edwaeseeeeedd..oo cn, weijia. zhang@newasstlee edu. au, zhangml (aengml@eee..mm @seu. edu.
Abstract
In many real-world tasks, the concerned objects can be represented multi- instance bag associated with a candidate label set, which consists of one ground- truth label and several false positive labels. Multi-instance partial-label learning (MIPL) is learning paradigm to deal with such tasks and has achieved favorable performances. Existing MIPL approach follows the instance-space paradigm by assigning augmented candidate label sets of bags each instance and aggregating bag-level labels from instance-level labels. However, this scheme may subopti- mal as global bag-level information is ignored and the predicted labels of bags are sensitive to predictions negative instances. this paper, we study an alternative scheme where mulli-instanee bag is embedded into single vector representation. Accordingly, an intuitive algorithm named DEMIPL, i.c., Disambiguated attention Embedding for Multi-Instane Partial-Labe learning, is proposed. DEMIPL em- ploys a disambiguation attention mechanism to aggregate a multi mustii-nstannce bag into single vector representation, followed by momentum-baee based disambiguation strategy to identify the ground-trutt label from the candidate label set. Further- more, we introduce real-world MIPL dataset for colorectal cancer classification. Experimental results on benchmark and real-world datasets validate the superiority of DEMIPL against the compared MIPL. and partial-label learning approaches.
1 Introduction
Significant advancements in supervised machine learning algorithms have been achieved by utilizing large amounts of labeled training data. However, in numerous tasks, training data is weakly-supervised due to the substantial costs associated with data labeling [1-] Weak supervision can be broadly categorized into three types: incomplete, inexact, and inaccurate supervision [7]. Multi-instance learning (MIL.) and partial-label learning (PLL) are typical weakly-supervised learning frameworks based on inexact supervision. In MIL, samples are represented by collections of features called bags, where each bag contains multiple instances [8H1 Only bag-level labels are accessible during training, while instance-level labels are unknown. Consequently, the instance space in MIL contains inexact supervision, signifying that the number and location of positive instances within positive bag remain undetermined. PLL associates each instance with a candidate label set that contains one ground-truth label and several false positive labels 22 As result, the label space in PLL embodies inexact supervision, indicating that the ground-truth label of an instance is uncertain.
*Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

S. (0, 1.4)
Sc (1,34)
%
multi-instance bags
negative instances
positive instances
features of multi-
S, {0. 2,3)
8, =0
instance bags
(b) Aggregated bag-level features and predicted labels by DEMIPL Figure 1: A brief illustration of DEMIPL, where S and Y are candidate label sets and predicted labels, respectively. The ground-truth labels are shown in red.
(a) Instance-level distribution of three MIPL samples
However, inexact supervision exist in instance and label space simultaneously, i.e., dual inexact supervision [36.. This phenomenon can be observed in histopathological image classification, where an image is typically partitioned into multi-instance bag [37,40), and labeling ground-trute labels incurs high costs due to the need for specialized expertise. Consequently, utilizing crowd-sourced candidate label sets will significantly reduce the labeling cost [1]. For this purpose, a learning paradigm called multi-instance partial-label learning (MIPL) has been proposed to work with dual inexact supervision. In MIPL,, training sample is represented a bag-level candidate label set, which comprises a ground-trutt label along with several false positive labels. is noteworthy that the multi-instance bag includes at least one instance that is affiliated with the ground-truit label, while none of the instances belong to any of the false positive labels.
multi-instance bag associated with
Due to the difficulty in handling dual inexact supervision, to the best of our knowledge, MIPLGP 36 is the only viable MIPL approach. MIPLGP learns from MIPL data at the instance-level by utilizing a label augmentation strategy assign an augmented candidate label set each instance, and integrating a Dirichlet disambiguation strategy with the Gaussian processes regression model [42). Consequently, the learned features of MIPLGP primarily capture local instance-level information, neglecting global bag-level information. This characteristic renders MIPLGP susceptible to negative instance predictions when aggregating bag-level labels from instance-level labels. As illustrated in Figure I a). identical or similar negative instances can simultancously occur in multiple multi-instang bags with diverse candidate label sets, thereby intensifying the challenge of disambiguation.
In this paper, we overcome the limitations of MIPLGP by introducing a novel algorithm, named DEMIPL, i..., Disambiguated attention Embedding for Instti-Instance Partial- learning, based on the embedded-space paradigm. Figure [ illustrates that DEMIPL aggregates each multi-instance bag into single vector representation, encompassing all instance-level features within the bag. Furthermore, DEMIPL effectively identifies the ground-truth label from the candidate label set.
Our contributions can be summarized follows: First, propose disambiguation attention mechanism for learning attention scores in multi-instance bags. This is in contrast to existing attention-basee MIL approaches that are limited to handling classifications with exact bag-level labels Second, we propose an attention loss function that encourages the attention scores of positive instances approach one, and those of negative instances approach zero, ensuring consistency between attention scores and unknown instance- labels. Third, we leverage the multi-class attention scores to map the multi-instanm momentum-based disambiguation strategy to identify the ground-truth labels of the multi-instance bags from the candidate label sets. In addition, we introduce real-world MIPI. dataset for colorectal cancer classification comprising 7000 images distributed across seven categories. The candidate labels of this dataset are provided by trained crowdsourcing workers.
B [4 4
bags into embedded space, and propose
Experiments are conducted on the benchmark as well as real-world datasets. The experimental results demonstrate that: (a) DEMIPL achieves higher classification accuracy on both benchmark and real- world datasets. (b) The attention loss effectively enhances the disambiguation attention mechanism, accurately discerning the significance of positive and negative instances. (c) The momentum baased based disambiguation strategy successfully identifies the ground-tree -ruth labels from candidate label sets, especially in scenarios with increasing number of false positive labels.
The remainder of the paper is structured as follows. First, we introduce DEMIPI. Section and present the experimental results in Section Finally, conclude the paper in Section 4

---

multi-instance bag X;
instance-level feature H;
attention mechanism
attention
scores
Cm
candidate label set S;
bag- leg-level feature
probability P;
classifier
Figure 2: The framework of DEMIPL, where Co and fm the attention loss and momentummbassee disambiguation loss, respectively.
2 Methodology
2.1 Notations and Framework of DEMIPL.
Let X =Rd represent the instance space, and let y (1,,2, ... (.. represent the label space containing & class labels. The objective of MIPL is to derive classifier f 2 - y D {(X,, (.))) S.) < i < m) is a training dataset that consists of bags and their associated candidate label sets. Particularly, St) is the i-th multi-instance partial-label sample, where X; { 2s. Ii.n, constitutes bag with n, instances, and each instance æi E X' for Vj E f1,2. ... ). S: y is the candidate label set that conceals the ground-truth label Y, Le.. E is worth noting that the ground- truth label is unknown during the training process. Assume the latent instance-level labels within X; is y, 1 U1... V..,. h. then 3y,.j = Y, and Vys.j & } hold. the context MIPL, an instance considered a positive instance its label is identical to the ground-truth label of the bag: otherwise, it is deemed negative instance. Moreover, the class labels of negative instances do belong to the label space.
The framework of the proposed DEMIPL is illustrated in Figure 2 It consists of three main steps. First, we extract instances in the multi-instance bag X; and obtain instance-level feature H, ;. Next, we employ the disambiguation attention mechanism to integrate the multi-instance bag into a single feature vector z;. Finally, we use classifier predict the classification confidences P, of the multi-instance bag. To enhance classification performance, we introduce two loss functions for model training: the attention loss La and the momentum- based disambiguation loss C. During the training process, the attention mechanism and the classifier work collaboratively.
2.2 Disambiguation Attention Mechanism
Based on the embedded-space paradigm, a key component of DEMIPL is the disambiguation attention mechanism. The attention mechanisms common models [44 46), which calculate attention scores to determine the contribution of each instance to the multi-instance bag [13 [4]. The attention scores are then utilized to aggregate the instance-level features into single vector representation.
For multi-instance bag X; = (.r,11, 2i,2, ... ). ). we employ a neural network-based function parameterized by h to extract its feature information:
H. h x:) (h.,,,,,,."" h....... hin))
where h;,; = h(xx,)) ) E Rd is the feature of the j-th instance within i-th bag. For the MIPL problems, propose multi-class attention mechanism. First, calculate the relevance of each instance to all classes, and then transform the relevance into the contribution of each instance to the bag-level feature by learnable linear model. The attention score at,j of æi is calculated follows:
aij
W [tanh [ h + by o sigm [ his + b., bb))
+ exp
3

---

where W E R W w. E R and by, bu E R are parameters of the attention mecha- nism. tanh(-) and (-gm(-) are the hyperbolic tangent and sigmoid functions to generate non-linear outputs for the models, respectively. @ represents an glement-wise multiplication. Consequently, the bag-level feature is aggregated by weighted sums of instance-level features:
0i.j hiih
2==1 Oi
j=1
where z; is the bag-level feature of x, ;. To ensure that the aggregated features accurately represent the multi-instance bag, it is necessary to maintain the consistency between attention scores and instance-level labels, that is, the attention scores of positive instances should be significantly higher than those of negative instances. To achieve this, the proposed attention loss is shown below:
m
n2g
C.
Oi log di.j
Different from existing attention-based MIL approaches where most of them can only handle binary classification, DEMIPL produces multi-class attention scores using Equation (2). Furthermore, unlike loss-based attention [4] that extends binary attention score to multi-class using a naive softmax function, DEMIPL utilizes learnable model with the attention loss encourage attention scores of negative and positive instances to approach O and 1, respectively. The ambiguity in attention scores is reduced since the differences in attention scores between positive and negative instances amplified. As result, the disambiguated attention scores can make bag-level vector representations discriminative, thereby enabling the classifier to accurately identify ground-truth labels.
2.3 Momentum- based Disambiguation Strategy
After obtaining the bag-level feature, the goal is to accurately identify the ground-trute label from the candidate label set. Therefore, we propose a novel disambiguation strategy, namely using the momentum- based disambiguation loss compute the weighted sum of losses for each category. Specifically, the proposed momentum-based disambiguation loss is defined as follows:
(1) wie ( (z.),,
Lm
(5)
where (4) refers to the 1-th epoch. z(() is the bag-level feature of multi-instance bag X; and s!²()) the model output on the e-th class at the 1-th epoch. E(-) is the cross-entropy loss, and w(()) weights
the loss value on the c-th class the t-th epoch.
Following the principle of the identification-bel based disambiguation strategy [47]. the label with the minimal loss value on the candidate label set can be considered the ground-truth label. We aim to assign weight of 1 to the single ground-truth label and a weight of to the rest of the candidate labels. However, the ground-truth label is unknown during the training process. To overcome this issue, we allocate weights based on the magnitude of class probabilities, ensuring that larger class probabilities are associated with higher weights. Specifically, we initialize the weights by:
1
if Yf S;,
w s,e
|5.1 O otherwise,
(6)
where 1 is the cardinality of the candidate label set S,. The weights are updated follows: [
(x( xw (t-1) + (1 - 111) Ejes, 17
(t)
if Yi,e E S,,, otherwise,
w
0
where the momentum parameter (1) I a trade-off between the weights the last epoch and the outputs at the current epoch. T is the maximum training epoch.
It is worth noting that the momentum-baan based disambiguation strategy is general form of the pro- gressive disambiguation strategy. Specifically, when (()) 0, the momentum-based disambigua- tion strategy degenerates into the progressive disambiguation strategy [4). When ( 1, the
momentum-based disambiguation strategy degenerates into the averaging-based disambiguation strategy (11) which equally treats every candidate label.

---

2.4 Synergy between Attention Mechanism and Disambiguation Strategy
Combining the attention loss and disambiguation loss, the full loss function derived as follows:
L = Lm + Xaha:
(8)
where Åa serves as a constant weight for the attention loss. In each iteration, the disambiguation attention mechanism aggregates discriminative vector representation for each multi-instance bag. Subsequently, the momentum-basse disambiguation strategy takes that feature as input and yields the disambiguated candidate label set, i.e., class probabilities. Meanwhile, the attention mechanism relies on the disambiguated candidate label set to derive attention scores. Thus, the disambiguation attention mechanism and the momentum- -baedd disambiguation strategy work collaboratively.
3 Experiments
3.1 Experimental Setup
We utilize four benchmark MIPL datasets stemming from MIPLGP literature [361, i.c., MNIST-MILL, FMNIST-MIRE Birdsong- MIPL, and SIVAL-MIPL from domains of image and biology 48 51. Table summarizes the characteristics of both the benchmark and real-world datasets. We use #bags, #ins, #dim, avg. #ins, #class, and avg. #CLs to denote the number of bags, number of instances, dimension of each instance, average number of instances in all bags, number of class labels, and the average size of candidate label set in each dataset.
Benchmark Datasets
Dataset We introduce CRC-MIPL, the first real-world MIPL dataset for colorectal cancer classification (CRC). comprises 7000 hematoxylin and eosin (H&E) staining images taken from colorectal cancer and normal tissues. Each image has dimensions of 224 x 224 pixels and is categorized into one of the seven classes based on the tissue cell types. CRC-MIPL derived from larger dataset used for colorectal cancer classification, which originally contains 100000 images with nine classes [52] The adipose and background classes exhibit significant dissimilarities compared to the other categories. Therefore, we choose the remaining seven classes to sample 1000 images per class. These classes include debris, lymphocytes, mucus, smooth muscle, normal colon mucosa, cancer-associated stroma, and colorectal adenocareinoma epithelium.
We employ four image bag generators [53] Row [54] single blob with neighbors (SBN) 54 k-means segmentation (KMeansSeg) 55 and scale-invariant feature transform (SIFT) [56 to obtain a bag of instances from each image, respectively. The candidate label sets of CRC-MIPL are provided by three crowdsourcing workers without expert pathologists. Each of the workers annotates all 7000 images, and each worker assigned candidate labels with non-zero probabilities to form label set per image. A higher probability indicates higher likelihood of being the ground-truth label, while probability zero implies the label a non-candidate label. After obtaining three label sets for each image, we distill a final candidate label set as follows. A label present in two or three label sets is selected as a member of the final candidate label set. the final candidate label set consists of only one no label, we pick the labels corresponding the highest probability in each label set. The average length of the final candidate label set per image is 2.08. More detailed information on the MIPL datasets can be found in the Appendix.
Compared Algorithms For comparative studies, we consider one MIPL. algorithm MIPI.GP 36 and four PLL algorithms, containing one feature-aware disambiguation algorithm PL-AGGD 32 and three deep learning-based algorithms, namely PRODEN [47, Rc [57]. and Lws (58)
Table 1: Characteristics of the Benchmark and Real-World MIPL Datasets.
Dataset
#bags
#ins
#dim
avg. fins
#class
avg. #CLs
domain
MIPP
784
41.33
image
PMNIST-MIPL
500
20810
784
41.62
5
2, 3,4
image
Birdsong-MIPL
1300
48425
38
37.25
13
2, 5. 6,7
biology
AVAL-MIPL
1500
47414
30
31.61
25
3.4
image
CRC MIPL Row
7000
56000
9
8
2,08
image
CRC MIPL.SBN
image
ERC-MIPL-KMenne
7000
30178
6
4,311
7
2.08
image
CRC-MIPL-SIFT
7000
175000
128
25
2.08
image
5

---

Implementation DEMIPL is implemented using PyTorch [59] on single Nvidia Tesla V100 GPU. We employ the stochastic gradient descent (SGD) optimizer with a momentum of 0.9 and weight decay of 0.0001. The initial learning rate is chosen from set of (0.01, 0.05) and is decayed using a cosine annealing method [6)] The number of epochs is set to 200 for the SIVAL-MIPL. and CRC-MIPI datasets, and 100 for the remaining three datasets. The value of X is selected from set of {0.0001, 0.001}. For the MNIST-MIPL and FMNIST-MIPL datasets, we utilize a two-layer CNN in work of Ilse et al. [I3 as a feature extraction network, whereas for the remaining datasets, no feature extraction network is employed. To ensure the reliability of the results, we conduct ten runs of random train/test splits with ratio of 7 3 for all datasets. The mean accuracies and standard deviations are recorded for each algorithm. Subsequently, we perform pairwise t-test at a significance level of 0.05.
To map data into PI.L. data, we employ the Mean strategy and the MaxMin strategy as described in the MIPI.GP literature [36]. The Mean strategy calculates the average values of each feature dimension for all instances within a multi-instance bag to yield a bag-level vector representation. The Max strategy involves computing the maximum and minimum values of each feature dimension for all instances within multi-instance bag and concatenating them to construct bag-level features. In the subsequent section, we report the results of PRODEN, Rc, and Lws using linear models, while the results with multi-layer perceptrons are provided in the Appendix.
3.2 Experimental Results on the Benchmark Datasets
Table presents the classification results achieved by DEMIPL. and the compared algorithms on the benchmark datasets with varying numbers of false positive labels r. Compared to MIPL.G, DEMIPL. demonstrates superior performance in out of 12 cases, with no significant difference observed in the remaining 1 out of 12 cases. MIPLGP performs better than DEMIPL on the SIVAL-MIPL dataset, primarily due to the unique characteristics of the SIVAL-MILL dataset. The dataset encompasses 25 highly diverse categories, such as apples, medals, books, and shoes, resulting in distinctive and discriminative features within multi-instance bags. Each instance's feature includes color and texture information derived from the instance itself as well as its four cardinal neighbors, which enhances the distinctiveness of instance-level features. This suggests that instances with similar features are rarely found across different multi-instance bags. Therefore, by following the instance-space paradigm, MIPLGP effectively leverages these distinctive attributes of the dataset.
Compared to PLL algorithms, DEMIPL outperforms them on all benchmark datasets. This superiority can be attributed to two main factors. First, PLL algorithms cannot directly handle bags, whereas the original multi-instance features possess better discriminative power than the degenerated features obtained through the Mean and MaxMin strategies. Second, the proposed momentum-based disambiguation strategy is more robust than the disambiguation strategies of the compared algorithms. It should be noted that although these PLL algorithms achieve satisfactory results in PLL tasks, their performance in addressing MIPL problems is inferior to dedicated MIPL algorithms, namely DEMIPL and MIPLGP. This observation emphasizes the greater challenges posed by MIPL problems, which involve increased ambiguity in supervision compared to PLL problems, and highlights the necessity of developing specialized algorithms for MIPL.
Additionally, we experiment with another extension of applying PLL algorithms to MIPL data by directly assigning a bag-level candidate label set as the candidate label set for each instance within the bag. However, all of them perform worse than MIPL. Moreover, the majority of the compared PLL algorithms fail to produce satisfactory results. This is likely caused by the fact that the ground-truth labels for most instances are absent from their respective candidate label sets. Consequently, the absences of ground-trutte labels impede the disambiguation ability of MIPL algorithms.
3.3 Experimental Results on the Weal-World Dataset
The classification accuracy of DEMIPL and the compared algorithms on the CRC-MIPL dataset is presented in Table 3 where the symbol indicates that MIPLGP encounters memory overflow issues on our V100 GPUs DEMIPL demonstrates superior performance compared to MIPL.G on the CRC- MIPL-SRN and CRC-MIPI-KMannee dataset. When compared to the PI.L algorithms, DEMIPL achieves better results in 28 out of 32 cases, and only underperforms against PL-AGGD in 2 cases CRC-MIPL-Row and CRC-MIPL-SNN,
datasets, while only falling behind MIPL.C .P on the CRC-MIPL

---

Table 2: Classification accuracy (meantistd) of each comparing algorithm on the benchmark datasets
in terms of the different number of false positive labels [r 2, 3) /ac indicates whether the
performance of DEMIPL is statistically superior/inferior to the compared algorithm on each dataset.
Algorithm
r
MIIL
FMNIST-MIPL
Birdsong MIPL
SIVAL-MIPL
0.97610.008
0.,01
0.74410.016
0.635.10.041
DEMIPL
2
0.94310.027
0.823110.025
0.701+0.024
0.554::0.051
0.709110088
0.657110.025
0.696110.024
0.50310.018
1
0.049=000016*
0.847=#0.030
0.716+00.026
0.669:00.019
MIPLGP
2
0.817110030
0.79110.027
0.672110.015
0.613110.26c
0.62110.064 3
0.670110.052
0.625110.015
0.56910.032.
Mean
0.605_00023
0.69710.042
0.21910.014 0,29610,014
PRODEN
2
0.481=00033
0.573++0.226
0.272+0.009
0.184+0.014e
3
0.283++0002
0.345=0.027
0.211+0.013
0.166+0.0177
1
0..08
0.753.10.042
0.362.10.015
0.27910.011
RC
2
0.598=00.033
0.649+0.028
0.335+00011
0.258
3
0.392::0.033+
0.401#0.063+
0.298+0.000
0.237=0.020+
1
0.463
0.726.00031
0.2651000010
0.240-10.014
LWS
2
0.209::0.02
0.720=0.025
0.254:0.000
0.223+0.008
3
0.205
0.549
0.237+00.005
0.194+0.026
I
0.601
0.743
0.353110.019
0.
PL-AGGD
2
0.095
0.677+00.028
0.31410.018
0.315
0.380=0.032 3
0.474
0.296:10.015
0.286+0.018
MaxMin
0.508
0.424
0.387100014
0.316:0.019
PRODEN
2
0.400
0.377++0.040
0.357+-0.012
0.287+0.024
3
0.044
0.309++0.58.
0.336+0..12.
0.250+0.018
I
0.519:10.028
0.001
0.39010.014
0.306:10.023
RC
0.469:100335
0.66610.027
0.37110.013
0.288+0.02
3
0.380++0.048
0.524
0.363++0.010
0.267+0.000
1
0.242:000022
0,405.
0.225
0.28910.017
LWS
2
0.239:10.048
0.406
0.2074
0.27110.014
3
0.218=0.011
0.318+0.064
0.216+000029
0.244+00.023
1
0.52710.035
0,39110.040
0.38310.014
0.39710.0228
PL-AGGD
2
0.439110.020
0.37110.037
0.37210.020
0.36010.029e
0.304
0.32710.028
0.34410.011
0.32810.0233
Table Classification accuracy (td) of each comparing algorithm on the CRC-MIPL dataset.
Algorithm
CRC-MIPL-ROw
CRC-MIPL-SBN
CRC-MuPL-KMeunse
CRC-MIPL-SIT
DEMIPI.
0.408=0.010
0.486
0.521=0.012
0.532=0.013
MIPLGP
0.432:0.005o
0.335110.00w
0.329=0.012
Mean
PRODEN
0.365:10.009
0.392-10.008
0.233:10:18$+
0.33410.029
RC
0.21410.011
0.242-10.012
0,22610.009
0.209:10.007
LWS
0.201
0.310++0.006
0.237=00000
0.270=t0.007
PL-AGGD
0.412+00008
0.480=t0.005
0.358+00008
0.363+0.0122
MaxMin
PRODEN
0.401
0.447=0.0011
0.205
0..01
RC
0.22710.012
0.338.10.110
0.20810.007
0,246.10.008
LWS
0.299100.008 0.382-10.009
0.24710.005e
0.230.10.007
PL-AGGD
0.460-10.008
0.524-10.008
0.434.10.009
0..00
The above results indicate that DEMIPL achieves the best performance when combined with stronger bag generators such as CRC-MILL Keeussyy and CRC-MIPL This combination enables the dis- ambiguation attention mechanism learn meaningful embeddings. This aligns with the fact that CRC-MIPL and CRC-MIPL-SBN use classic culti-insstanee bag generators that only consider pixel colors, and lack the ability to extract any of the content information. In contrast, CRC-MIPL-KMenne and CRC-MPI-SIIT are content-aware generators that are capable of producing semantically meaning- ful features. Both CRC-MIPL-RMN and CRC-MIPI-SH segment images using fixed grids, and represents instances based on their pixel-level colors and the colors of their adjacent rows grids. Consequently,
7

---

Table 4: Classification accuracy (mean of DEMIPL- MD and DEMIPL.
-MPIST-MPPL.
SIVAL
Algorithm
T=
r r=
==3
r ==1
7=2
DEMIPL-MD
0.744110.273 0.881+0.021
0.78410.018
0.58611.101 0.823
0.530=1.021 0.607-1.0024 0.635=00.041
0.499_100035
DEMIPL
0.657=0.025
0.554+0.051 0.503+0.018
13500
13500
trequency
DEMINL-MD
1000
13000
13000
0,474
0.46
0.47
0.0
0.2
0.0
frequency
DEMIPL
13500
135000
500-
13000
13000
250
0.500
0.8
1.0
attention score (epoch 10)
attention score (epoch 50)
attention score (epoch 100)
Figure 3: The frequency distribution of attention scores MNIST-MIPL. dataset (r 1).
instances in CRC-MIPI-Row and CRC-MPPL-SRE exhibit similar feature representations, and possess lim- ited discriminative power when distinguishing positive and negative instances. With more powerful bag generators such as CRC-MPLL KMeamsseg and CRC-MIPL-SITT, which generate content-aware features that are more informative and discriminative, the disambiguation power of DEMIPL can be fully utilized as demonstrated by the significant performance advantages against all compared baselines.
Furthermore, the CRC-MIPL dataset exhibits distinct differences between tissue cells and the back- ground in each image. The Mean strategy diminishes the disparities and discriminations, leading to superior outcomes for the Maxmin strategy in most cases when compared to the Mean strategy.
3.4 Further Analysis
Effectiveness of the Attention Loss To validate the effectiveness of the attention loss, we introduce degenerated variant named DEMIPL-MD, which excludes the attention loss from DEMIPL. Table verifies that DEMIPL achieves superior accuracy compared to DEMIPL-MD on both the FMNIST-MIL and SIVAL-MIPL datasets. Notably, the difference is more pronounced on the FMNIST-MIPL. dataset than that on the SIVAL-MIPL dataset. This can be attributed to the fact that the feature representation of each instance in the FMNIST-MIPL. dataset solely comprises self-contained information, enabling clear differentiation between positive and negative instances. Conversely, the feature representation of each instance in the SIVAL-MIPL dataset encompasses both self and neighboring information, leading to couplings between the feature information of positive instances and negative instances.
To further investigate the scores learned by the attention loss, visualize the frequency distribution of attention scores throughout the training process. As illuminated in Figure 3 the top row corresponds to DEMIPL-MD. while the bottom row corresponds to DEMIPL. At epoch=10, attention scores generated by DEMIPL show higher dispersion, suggesting that DEMIPL trains faster than DEMIPL- MD. At epoch=50 and 100, attention scores computed by DEMIPL. tend converge towards two extremes: attention scores for negative instances gravitate towards zero, while attention scores for positive instances approach one. In conclusion, the attention loss is conducive to calculating appropriate attention scores for positive and negative instances, thereby improving accuracy.
Effectiveness of the Momentum-basee Disambiguation Strategy To further investigate the momentum-basee disambiguation strategy, the performance of DEMIPL is compared with its two degenerated versions denoted as -EMIPL-PR and DEMIPL-AV. DEMIPL-PR is obtained by setting the momentum parameter X (+) 0 in Equation (7). which corresponds progressively updating the weights based on the current output of the classifier. In contrast, DEMIPL-AV is obtained by setting the momentum parameter (t) 1, resulting in uniform weights throughout the training process.
Figure 4 illlustrates the performance comparison among DEMIPL, DEMIPI.PRR, and DEMIPL-AAV on the MNIST-MPPL, FMNIST-MIr,, and Birdsong-MIPL datasets. When the number of false positive labels is small, DEMIPI.-PE and DEMIPI-AAV demonstrate similar performance to DEMIPL.. However,
8

---

MNIST-MIPL
FMNIST-MIPE
Birdsong-Mor
1.0
0.9
0.7
0.9
0.8
accuracy
0.6
0.8
0.7
-0-
DEMIPL
DEMIPL-PP
0.6
AEM
0.1
0.6
r (# the false positive lahels)
7. (# the false positive labels)
r. (# the false positive labels)
Figure 4: Classication accuracy of DEMIPL, DEMIPL. PR, and DEMIPL-AV with varying r.
MNIST-MIPL
CRC-MIPL-SIF
0.6
aecuracy
0.5
0.5
0.4
0.0 0.000s
0.001
0.005
0.31
0.35
0.1
0.8
0,3001
0.0005
0.901
0.005
0.01
0.05
0.1
0.5
Xa
Figure 5: Performance of DEMIPL with varying weight Aa
as the number of false positive labels increases, DEMIPL consistently outperforms DEMIPL-PR and AEMIPL-AV by a significant margin. This observation suggests that the momentum based disambiguation strategy is more robust in handling higher levels of disambiguation complexity. Furthermore, it can be observed that DEMIPL- PR generally outperforms DEMIPL-AV across various scenarios. However, when r = 3 in the MNIST-MIL. and FMNIST-MIP datasets, DEMIPL-AV surpasses DE.. We believe this can be attributed to the following reason: having three false positive labels within the context of five classifications represents an extreme case. DEMIPL. PR likely assigns higher weights to false positive labels, whereas DEMIPL--AV uniformly assigns weights to each candidate label, adopting more conservative approach to avoid assigning excessive weights to false positive labels. In a nutshell, the proposed momentum-basee disambiguation strategy demonstrates superior robustness compared to existing methods for disambiguation.
The weight Aa in Equation (8) is the primary hyperparameter in DEMIPL. FigureBjillustrates the sensitivity analysis of the weight Aa on the MNIST-MILL and CRC- MIPI datasets. The learning rates on the MNIST-MPLL dataset are set to 0.01, 0.01, 0.05 for r 1, 2, 3. respectively, while on the CRC-MIPI dataset, the learning rate set to 0.01. As illuminated in Figure 5]] DEMIPL demonstrates insensitivity to changes in the weight Xa In the experiments involving DEMIPL and its variants, the weight A is chosen from a set of (0.0001, 0.0013.
Parameter Sensitivity Analysis
4 Conclusion
In this paper, we propose DEMIPL, the first deep learning-based algorithm for multi-instance partial- label learning, accompanied by real-world dataset. Specifically, DEMIPL utilizes the disambiguation attention mechanism to aggregate each multi-instance bag into a single vector representation, which is further used in conjunction with the nomentuumbassed disambiguation strategy to determine the ground-truth label from the candidate label set. The disambiguation attention mechanism and momentum-based strategy synergistically facilitate disambiguation in both the instance space and label space. Extensive experimental results indicate that DEMIPL outperforms the compared algorithms in 96.3% of cases on benchmark datasets and 85.7% of cases on the real-world dataset.
Despite DEMIPL'S superior performance compared to the well-established MIPL and PLL approaches, it exhibits certain limitations and there are several unexplored research avenues. For example, DEMIPL assumes independence among instances within each bag. A promising avenue for future research involves considering dependencies between instances. Moreover, akin to MIL. algorithms grounded in the embedded-space paradigm [33, accurately predicting instance-level labels poses a challenging endeavor. One possible approach entails the introduction of an instance-level classifier.

---

Acknowledgements
The authors wish to thank the anonymous reviewers for their helpful comments and suggestions. This work was supported by the National Science Foundation of China (62225602, 62206047), the Postgraduate Research & Practice Innovation Program of Jiangsu Province (KYCX23_00) (0112,, and the Big Data Computing Center of Southeast University.
References
[1] Bo Han, Quamming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in Neural Information Processing Systems 31, Montréal, Canada, pages 6536-8546, 2018.
[2] David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: holistic approach to semi-supervised learning. In Advances in Neural Information Processing Systems 32, Vancouver, BC, Canada, pages 5050-5060, 2019.
[3] Takashi Ishida, Gang Niu, Weihua Hu, and Masashi Sugiyama. Learning from complementary labels. In Advances in Neural Information Processing Systems 30, Long Beach, CA, pages 5639-5649, 2017.
[4] Yi Wei, Mei Xue, Xin Liu, and Pengxiang Xu. Data fusing and joint training for learning with noisy labels. Frontiers of Computer Science, 16(6):166338:1 2022.
[5] Ming-Kun Xie and Sheng-Jun Huang. Partial multi-label learning with noisy label identification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7):3676-3677, 2022.
[6] Gengyu Lyu, Songhe Feng. Yi Jin, Tao Wang, Congyan Lang, and Yidong Li. Prior knowledge regularized self-reprrsenttation model for partial multi-label learning. IEEE Transactions on Cybermetics, 53(3):1618-1628 2023.
[7] Zhi-Hua Zhou. A brief introduction to weakly supervised learning, National Science Review, 5 (1):44 53, 2018.
[8] Jaume Amores. Multiple instance classification: Review, taxonomy and comparative study. Artificial Intelligence, 201:81-105, 2013.
[9] Marc-André Carbonneau, Veronika Cheplygina, Eric Granger, and Ghyslain Gagnon. Multiple instance learning: A survey of problem characteristics and applications. Pattern Recognition, -533, 2018.
[10] Zhi-Hua Zhou, Yin Sun, and Yu-Feng Li. Multi-instance learning by treating instances as non-i.idd samples. In Proceedings of the 26th International Conference on Machine Learning, Montreal, Quebec, Canada, pages 1249-1256, 2009.
[11] Xiu-Shen Wei, Han-Jia Ye, Xin Mu, Jianxin Wu, Chunhua Shen, and Zhi-Hua Zhou. Multi- instance learning with emerging novel class. IEEE Transactions on Knowledge and Data Engineering, 33(5):2109 2120.. 2019.
[12] Xinggang Wang, Yongluan Yan, Peng Tang, Xiang Bai, and Wenyu Liu. Revisiting multiple instance neural networks. Pattern Recognition, 74:15 14:15-24, 2018
[13] Maximilian Ilse, Jakub M. Tomezak, and Max Welling. Attention- based deep multiple in- stance learning. In Proceedings of the 35th International Conference on Machine Learning, Stockhollsmüsssun, Stockholm, Sweden, pages 2132-2141, 2018
[14] Xiaoshuang Shi, Fuyong Xing, Yuanpu Xie, Zizhao Zhang, Lei Cui, and Lin Yang. Loss-based attention for deep multiple instance learning. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, New York, NY. USA, pages 5742-5749, 2020.
10

---

[15] Bin Li, Yin Li, and Kevin W. Eliceiri. Dtal-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proceedings of the 34th IEEE/CVE Conference on Computer Vision and Pattern Recognition, Virtual Event, pages 14318-14328, 2021.
[16] Weijia Zhang. Non-iid. multi-instance learning for predicting instance and bag labels with variational auto-encoder. In Proceedings of the 30th International Joint Conference on Artificial Intelligence, Virtual Event / Montreal, Canada, pages 3377-338, 2021.
[17] Weijia Zhang, Xuanhui Zhang, Han- Wen Deng, and Min-Ling Zhang. Multi-instance causal representation learning for instance label prediction and oft-o--distribution generalization. In Advances in Neural Information Processing Systems 35, New Orleans, LA, USA, pages 34940 -4900,3343,, 2022.
[18] Syed Ashar Javed, Dinkar Juyal, Harshith Padigela, Amaro Taylor-Weiner, Weiner, Limin Yu, and Aaditya Prakash. Additive MIL: Intrinsically interpretable multiple instance learning for pathology. In Advances in Neural Information Processing Systems 35, New Orleans, LA, USA, pages 20689-20702, 2022.
[19] Jinxi Xiang, Xiyue Wang. Jun Zhang, Sen Yang, Xiao Han, and Wei Yang. Exploring low-rank property in multiple instance learning for whole slide image classification. In Proceedings of the 1 11th International Conference on Learning Representations, Kigali, Rwanda, pages 1-18, 2023.
[20] Rong Jin and Zoubin Ghahramani. Learning with multiple labels. In Advances in Neural Information Processing Systems 15, Vancouver, British Columbia, Canada, pages 897-904, 2002.
[21] Timothee Cour, Ben Sapp, and Ben Taskar. Learning from partial labels. The Journal of Machine Learning Research, 12:1501-1536, 2011.
[22] Liping Liu and Thomas Dietterich. conditional multinomial mixture model for superset label learning. In Advances in Neural Information Processing Systems 25, Cambridge, MA, USA, pages 548-556, 2012.
[23] Fei Yu and Min-Ling Zhang. Maximum margin partial label learning. Machine Learning, (106))573-593, 2016.
[24] Min-Ling Zhang. Fei Yu, and Cai-Zhi Tang. Disambiguation free partial label learning. IEEE Transactions on Knowledge and Data Engineering, 29(10): 2017.
[25] Chen Gong, Tongliang Liu, Yuanyan Tang, Jian Yang, Jie Yang. and Dacheng Tao. A regular- ization approach for instance-based superset label learning, IEEE Transactions Cybernetics, 48(3):967-978, 2018
[26] Yao Yao, Jichui Deng, Xiuhua Chen, Chen Gong, Jianxin Wu, and Jian Yang. Deep dis- criminative CNN with temporal ensembling for ambiguously-labeled image classification. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, New York, NY, USA, pages 18669-12676 2020.
[27] Yan Yan and Yuhong Guo. Partial label learning with batch label correction. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, New York, NY, USA, pages 6575-6582, 2020.
[28] Gengyu Lyu, Songhe Feng, Tao Wang, Congyan Lang, and Yidong Li. GM-PLL: Graph matching based partial label learning. IEEE Transactions on Knowledge and Data Engineering, 33(2):521-535. 2019.
[29] Ning Xu, Congyu Qiao, Xin Geng, and Min-Ling Zhang. Instance-dependent partial label learning. Advances in Neural Information Processing Systems 34, Virtual Event, pages 27119. 19- 27130, 2021.
11

---

[30] Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao. PiCO: Contrastive label disambiguation for partial label learning. In Proceedings of the 10th International Conference on Learning Representations, Virtual Event, pages 1-18, 2022.
[31] Dong-Dong Wu, Deng-Bar Bao Wang, and Min-Ling Zhang. Revisiting consistency regularization for deep partial label learning. In Proceedings of the 39th International Conference on Machine 2022.
Learning, Baltimore, Maryland, USA, pages
[32] Deng-Bar Bao Wang, Min-Ling Zhang, and Li Li. Adaptive graph guided disambiguation for partial label learning. IEEE Transactions on Pattern Analysis and Machine Intelligence. 44(12); 8796-88 2022.
[33] Xiuwen Gong, Dong Yuan, and Wei Bao. Partial label learning via label influence function. In Proceedings of the 39th International Conference on Machine Learning, Baltimore, Maryland, USA, pages 7665-7678, 2022.
[34] Haobo Wang, Mingxuan Xia, Yixuan Li, Yuren Mao, Lei Feng, Gang Chen, and Junbo Zhao. SoLar: Sinkhorn label refinery for imbalanced partial-label learning. In Advances in Neural Information Processing Systems 35, New Orleans, LA, USA, pages 8104-8117 2022.
[35] Yingjie Tian, Xiaotong Yu, and Saiji Fu. Partial label learning: Taxonomy, analysis and outlook. Neural Networks, 161:708-734, 2023.
[36] Wei Tang, Weijia Zhang, and Min-Ling Zhang. Multi-instance partial-label learning: Towards exploiting dual inexact supervision. Science China Information Sciences, pages 1-16, 2023.
[37] Gabriele Campanella, Matthew G Hanna, Luke Geneslaw, Allen Miraflor, Vitor Werneck Krauss Silva, Klaus J Busam, Edi Brogi, Victor E Reuter, David S Klimstra, and Thomas Fuchs. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature Medicine, 25(8):1301-130 2019.
[38] Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and Faisal Mahmood. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature Biomedical Engineering, 5(6):555-5700 2021.
[39] Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang. Jian Zhang, Xiangyang Ji, and Yongbing Zhang. TransMIL: Transformer based correlated multiple instance learning for whole slide image classification. In Advances in Neural Information Processing Systems 34, Virtual Event, pages 2136 2147, 2021.
[40] Hongrun Zhang, Yanda Meng, Yitian Zhao, Yihong Qiao, Xiaoyun Yang. Sarah Coupland, and Yalin Zheng. DTFD-MIL: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification. In Proceedings of the 35th IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, USA, pages 18802- 18812, 2022.
[41] Zehui Liao, Yutong Xie, Shishuai Hu, and Yong Xia. Learning from ambiguous labels for lung nodule malignancy prediction. IEEE Transactions on Medical Imaging, 41(7):18741188, 2022.
[42] Ke Alexander Wang, Geoff Pleiss, Jacob R. Gardner, Stephen Tyree, Kilian Q. Weinberger, and Andrew Gordon Wilson. Exact gaussian processes on a million data points. In Advances in Neural Information Processing Systems 32, Vancouver, BC, Canada, pages 14622-14632, 2019.
[43] Yufei Cui, Ziquan Liu, Xiangyu Liu, Xue Liu, Cong Wang, Tei-Wei Kuo, Chun Jason Xue, and Antoni B. Chan. Mayes-MIL:: A new probabilistic perspective on attention- multiple instance learning for whole slide images. In Proceedings of the 1 11th International Conference on Learning Representations, Kigali, Rwanda, pages 1-17, 2023.
[44] Meng-Hao Guo, Tian-Xing Xu, Jiang-Jiang Liu, Zheng-Ning Liu, Peng- Tao Jiang, Tai-Jiang Mu, Song-Hai Zhang, Ralph R. Martin, Ming-Ming Cheng, and Shi-Min Hu. Attention mechanisms in computer vision: A survey. Computational Visual Media, 8(3):331-366 2022.
12

---

[45] Yang Wang, Jinjia Peng, Huibing Wang, and Meng Wang. Progressive learning with multi- scale attention network for cross-domain vehicle re-identification. Science China Information Sciences, 65(6):160103:1-15 2022.
[46] Wei-Chen Chen, Xin- Yi Yu, and Linlin Ou. Pedestrian attribute recognition in video surveillance scenarios based on view-attribute attention localization. Machine Intelligence Research, 19(2): 153-168, 2022.
[47] Jiaqi Lv, Miao Xu, Lei Feng, Gang Niu, Xin Geng, and Masashi Sugiyama. Progressive identification of true labels for partial-label learning. In Proceedings of the 37th International Conference Machine Learning, Virtual Event, pages 6500-6510, 2020.
[48] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applicd to document recognition. Proceedings of the IEEE, 86(1 (6:111))22782333 2324, 1998.
[49] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms. CoRR, abs/1708.07747, 2017. URL http://
arxiv wexxivoor//ae http///bb//1/..../. 1708. 07747
[50] Forrest Briggs, Xiaoli Z. Fern, and Raviv Raich. Rank-loss support instance machines for MIML instance annotation. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Beijing, China, pages $34-542, 2012.
[51] Burr Settles, Mark Craven, and Soumya Ray. Multiple-instance active learning. In Advances in Neural Information Processing Systems 20, Vancouver, British Columbia, Canada, pages 1289-1296, 2007.
[52] Jakob Nikolas Kather, Johannes Krisam, Pornpimol Charoentong, Tom Luedde, Esther Herpel, Cleo-Aron Weis, Timo Gaiser, Alexander Marx, Nektarios A Valous, Dyke Ferber, al. Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study. PLoS Medicine, 16(1):e1002230::1 1-2222 2019.
[53] Xiu-Shen Wei and Zui-Hua Zhou. An empirical study image bag generators for multi-instance learning. Machine Learning, 105:155-198, 2016.
[54] Oded Maron and Aparna Lakshmi Ratan. Multiple-instance learning for natural scene classifi- cation. In Proceedings of the 15th International Conference Machine Learning, Madison, Wisconsin, USA, pages 341-349, 1998.
[55] Qi Zhang, Sally A. Goldman, Wei Yu, and Jason E. Fritts. Content-based image retrieval using multiple-instance learning. In Proceedings of the 19th International Conference on Machine Learning, Sydney, Australia, pages 682-689, 2002.
[56] David G. Lowe. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2):91-110, 2004.
[57] Lei Feng, Jiaqi Lv, Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. Provably consistent partial-label learning. In Advances in Neural Information Processing Systems 33, Virtual Event, pages 10948-10960, 2020.
[58] Hongwei Wen, Jingyi Cui, Hanyuan Hang, Jiabin Liu, Yisen Wang, and Zhouchen Lin. Lever- aged weighted loss for partial label learning. In Proceedings of the 38th International Conference on Machine Learning, Virtual Event, pages 11091-11100 2021.
[59] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf. Edward Z. Yang, Zachary De Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances Neural Information Processing Systems 32, Vancouver, BC, Canada, pages 0024-8035.0 2019.
[60] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In Proceedings of the 5th International Conference on Learning Representations, Toulon, France, pages 1-16, 2017.
13

---

A Appendix
A.1 Pseudo-Code of DEMIPL.
Given an unseen multi-instance bag X. [ 2.,,,, ..,n. with n, instances, DEMIPL.
initially utilizes the feature extractor obtain the instance-level representation as follows:
H. h(x,, [h+,1, 2, ).,n...
(9)
Subsequently, DEMIPL maps the instance-level representation H, into vector representation using the disambiguation attention mechanism, which is described as follows:
On 1 expp exp
(10)
TW (tanh W a h + b.) @ sigm W hess + bu )
(11)
Σラ=1 a+d
Finally, we employ the trained classifier to predict the label of X. by:
Y, arg max fe(z.),
(12)
cEy
where sa(.) is the e-th element of f(-), and the normalized function se(-) represents the probability of class label being the ground-trun -truth label.
Algorithm summarizes the complete procedure of DEMIPL. First, the algorithm uniformly initializes the weights the nomentuum-aseed disambiguation loss (Step 1). Next, the model training can be divided into two sub-steps (Steps 2-13). The initial sub-step involves extracting features for each mini-bare mani-batch and aggregating them into bag-level vector representations (Steps 5-7). The subsequent sub-step encompasses calculating the loss function and updating the model (Steps 8-11). Finally, for an unseen multi-instance bag, instance-level features are extracted and aggregated into a bag-level vector representation, which is used predict the label (Steps 14-16).
Algorithm Y, DEMIPL (D, Ao, T. X.)
Inputs:
D : the multi-instance partial-label training set (,) < S m). where X; {æ,,1, [0,,1, Zi... Li,n, Li X, X' Rd S; V.y- = (1,12,-- 14
Ag the weight for the attention loss
T: the number of iterations
X.: the unseen multi-instane bag with n, instances
Outputs:
Y. the predicted label for X.
Process:
1: Initialize uniform weights w stated by Equation (6)
2: to to T do
3: Shuffle training set 'D into B mini-batches
4: for to B do
5: Extract the instance-level features of X according to Equation !
6:
Calculate the attention scores as stated by Equation (2)
7:
Map the instance-level features into single vector representation according Equation
8:
Update weights (t( according Equation
7
9:
Calculate according to Equation (
10:
Set gradient V L.
11: Update & by the optimizer
end for
13:
end for
Extract the instance-level features of X, according to Equation 9
15: Calculate the attention scores and map the instance-level features into single vector representa-
tion according Equations (10) and
16: Return Y. according to Equation 2
14

---

A.2 Additional Experiment Results
Table 5: Classification accuracy (mean=istd) of each comparing algorithm the benchmark datasets in terms of the different number of false positive labels [r E (1, 3 e/c indicates whether the performance of DEMIPI. is statistically superior/inferior to the compared algorithm each dataset
(pairwise t-test at a significance level of 0.05).
Algorithm
MNIST-MIPL
FMNIST MIPL
Birdsong- MIPL
SIVAL-MIPL
0.076
0.881+0.021
0.744
0.635+00004
DEMIPL
0.94310.027
0.823:10.028
0.701=100244 0.696:10.024
0.55410.055
3
0.709:10088
0.657:10.025
0.503:10.018
Mean
0.535:10.033
0.632:10.033e
0.303:10.016
0.303:10.20
PRODEN
2
0.372:+0.038.
0.463++0.067
0.287+00.01
0.274+0.022
3
0.235++0.032+
0.288+0.039
0.278+0.006
0,242++0.009
1
0.660110.031
0.69
0.3291000044
0.34410.014
RC
2
0.577+:0.03
0.694
0,301=0.014
0.299
3
0.362++0.29
0.414+0.050
0.288+0.019
0.256+0.013
1
0.605
0.702.0.033
0.34410.018
0.34610.014
LWS
2
0.431=:0.02
0.547
0.310#0.014.
0.312+0.015
0.335+000229 3
0.411=0.033
0.289+0.021
0.286+0.011
MaxMin
0.465=:0.023
0.358#00019
0.339:00010
0.322+0.018
PRODEN
2
0.338:10.01
0.315:10.023
0.329110.016
0.095
0.260=10.037+
0.203
0.305+10.015
0.24440.018e
1
0.518
00001
0.379:00.114
0.304#0.015
RC
2
0.462::0.028
0.363+00018
0.359=0.015
0.268+000023
3
0.336+
0.204
0.332++0.024
0.244+-0.014
I
0.457:0.028
0,346:00:033
0.349H0.013
0.005:
LWS
2
0.351
0.32310.031
0.3361000033
0.314:0.019e
3
0.274++0.037
0.267+0.034
0.307++0.016
0.268++0.019
Table 6: Classification accuracy (mean of each comparing algorithm on the real-world dataset.
Algorithm CRC-MIPL.Re Row
CRC-MIPL SBN
CRC-MIPL KMaansSey
CRC-MIPL SIFT
DEMIPL
0.408=0.010
0.486=00011
0.521=0.012
0.532=0.013
Mean
PRODEN
0.405:10.012
0.515:10.010
0.512:10.14.
0.352:10.015
RC
0.290=10.010 0.394-10.010
0.304:1.011e
0.248:10.008
LWS
0.360-10.008
0.440-10.009
0.422+
0.009:
MaxMin
PRODEN
0,059
0.529=0.010s
0.563=0.0110
0,294:10,00$
RC
0.347+0.013+ 0.432-10.008%
0.36610.010
0.204=10.00%
LWS
0.301+0.011
0.442+t0.009
0.335+0.049
0.287+00000
report the results of PRODEN, Rc, and Lws using linear models. In this section, we supplement the results of the compared PLL algorithms with multi-layer perceptrons (MLPs) described in the respective literature. Table and Table presee the experimental results of DEMIPL compared to PLL algorithms on the benchmark and real-world datasets, respectively. It is noteworthy that DEMIPL. employs two-layer CNN network for feature extraction the MNIST-MIPL. and FMNÍST-WIP1. datasets, while on the remaining datasets, DRMIPI. only utilizes linear models.
In Section and
and Section 3.3
On the benchmark datasets, DEMIPL consistently outperforms the compared algorithms in almost all cases. Moreover, the compared algorithms using MLPs do not consistently yield superior results compared those using linear models, especially when the benchmark datasets exhibit relatively simple features. This suggests that linear models are sufficient to achieve satisfactory results given the benchmark datasets, while MLPs might introduce unnecessary complexity.
On the real-world dataset, DEMIPL outperforms the compared algorithms 19 out of cases. When combined with complex image bag generators such as CRC-MMPI-KMeInSS. and CRC-MIPL.SITT, DEMIPL outperforms the compared algorithms 11 out of 12 cases. the majority of cases, the compared algorithms using MLPs demonstrate better performance than those using linear models.
15

---

However, on the MIC-MIPL-SIPT dataset, the improvement provided by MLPs is not particularly evident and sometimes even leads to decline in performance. Therefore, when dealing with complex multi-instance features, the bag features obtained through the Mean or MaxMin strategies do not accurately reflect the characteristics of multi-instance bags. This highlights the need for specialized MIPL algorithms to accurately capture the features of multi-instance bags.
A.3 Theoretical Analysis
Theorem 1. In a
bag X when the normalized attention score of an instance æi, ",.3' 1, the probability of the multi-instance bag X, being classified as approaches e.g., Een 4s,3
the e-th class is approximately equal to that of the instance Zijj' belonging to the c-th class.
Proof. Equation 22 demonstrates that the attention score for each instance ranges between 0 and After normalizing by ش the sum of attention scores for all instances within a multi- instance bag becomes equal approach 1, the normalized attention scores of the remaining instances { Li,2 Zi,n, ( } approach 0. Based on Equation (3), the aggregated bag-level vector representation 2; Σee di,j Σj= Or.hhi, = ha."' h( Therefore, is confirmed that instances with higher attention scores contribute significantly the bag-level predictions, underlining the significance of attention mechanisms in multi-instance partial-labe learning.
When the normalized attention score of an instance Zi,j' is
Theorem 1 suggests that high attention scores of individual instances can play crucial role in determining the bag-level class prediction, emphasizing the significance of accurately capturing and interpreting attention scores multi-instance partial-label learning scenarios.
A.4 Image Bag Generator
We utilize four image bag generators to extract multi-instance features from the CRC-MIPL dataset. The detailed descriptions of these image bag generators are provided below:
It treats each row of the image as an individual instance. To extract the feature for each instance, the Row generator computes the average RGB color value of the row and the color differences in the rows above and below it.
Row Generator:
SBN Generator: It considers each 2 x 2 blob within the image and includes the RGB color values of the blob itself and its four neighboring blobs features for each instance. It generates instances by iteratively moving one pixel at a time. However, should be noted that the SBN generator ignores the feature information at the four corners of the image. KMeansSeg Generator: It divides the image into K segments or partition blocks. For each segment, it generates a 6-dimensional feature. The first three dimensions represent color values in the YCbCr color space, while the last three dimensions represent values obtained by applying the wavelet transform to the luminance (Y) component of the image.
SIFT Generator: applies the scale-invariant feature transform (SIFT) algorithm to extract features, which partitions each instance into multiple 4 x 4 subregions and assigns the gradients of the pixels within these subregions to bins. Consequently, the SIFT generator produces 128 di-dimensional feature vector for each instance.
The implementations of the four image bag generators are available at http: /ww. lamda. nju edu. wt/ccc//ceeeeee MIL-BG. ashx
A.5 Data and Code Availability
The implementations of the compared algorithms are publicly available. MIPLGP and PL- AGGD are implemented at http:://palm....m. seu, edu. cn/ cn/2hangml// PRODEN is implemented at /https://gitaiii cow ttchub.io/coeediee LW- moss-foor-partill.
RC is implemented at lattps///ffeg-ntte LWS implemented
Additionally, the code of DEMIPL, the benchmark datasets, and the weal-world dataset are publicly available wtttt::p/ppplla seu. edu. www.eed......aaaaee cn/zhangml/
16


Reviews:
[{"review_id": "4BzYzc4lmS", "reviewer": "Reviewer_pq8x", "summary": "In this paper the authors study multi-instance partial label learning, where a bag of instance is associated with several candidate labels of which it is assumed one is correct. The propose a deep learning approach for such a problem. The key contribution is a \"disambiguation attention mechanism\" where they find the relevance of each instance to each class. Each bag is then represented by a weighted average of instance features with the attention being the weights. They modify the loss function to improve consistency between attention scores and instance labels. Further they also propose a momentum based loss for the labels that gives higher weight to higher class probabilities. The provide experimental results on several synthetic datasets and one \"real-world\" dataset.", "questions": "How was label quality controlled in the histo dataset to ensure there was one correct label?", "limitations": "not adequately discussed", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ the MIPL setting could have some applications in the real world.\n+ Using attention to construct bag features seems like an interesting idea.\n+ a variety of experimental results have been reported against several baselines.\n+ additional experiments study the effect of the contributions in isolation.", "weaknesses": "- The assumption of a single correct label seems very strong and unrealistic in practice.\n- The histopathology task seems really contrived. It is hard to imagine non-experts labeling such images, and if they did, how could it be assumed any labels were correct?\n- Without a compelling application, it is not clear what is the significance of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback and insightful comments, which have greatly contributed to the enhancement of our paper. We also appreciate your acknowledgment of the positive attributes of our work. We will now address the mentioned negative aspects. Below, we provide a summary of your comments along with our corresponding responses.\n\n> The assumption of a single correct label seems very strong and unrealistic in practice.\n\nThe assumption that the candidate label set contains a single correct label has been extensively investigated in partial-label learning (PLL). Unlike PLL, the task of multi-instance partial-label learning (MIPL) is more intricate and is still in its initial stages of development. To the best of our knowledge, the only existing work on MIPL is [30], which shares this assumption. It is noteworthy that this assumption holds true for many scenarios. For instance, in the CRC-MIPL dataset, over $98.5\\\\%$ of ground-truth labels for multi-instance bags are indeed encompassed within their respective candidate label sets. The primary objectives of this paper are twofold: the proposal of an algorithm based on the embedded-space paradigm and the introduction of a real-world MIPL dataset. In future research, we intend to relax the assumption that candidate label sets invariably contain the true label and aim to devise effective algorithms.\n\n\n> It is hard to imagine non-experts labeling such images, and if they did, how could it be assumed any labels were correct? How was label quality controlled in the histo dataset to ensure there was one correct label? Without a compelling application, it is not clear what is the significance of this work.\n\nBased on our investigation, the utilization of non-expert annotations for histopathological images has been established as a viable approach, and crowdsourcing amateur labelers can achieve a good level of accuracy [a, b]. CRC-MIPL is derived from a larger dataset NCT-CRC-HE-100K used for colorectal cancer classification [44]. We select $1000$ images from each of the seven classes. Before annotating the CRC-MIPL dataset, we invite an expert to provide training to three crowdsourcing workers. During the labeling process, we shuffle the order of the images, and the ground-truth labels are hidden from the crowdsourcing workers. Each worker annotates a candidate label set for each image, and the final candidate label set is obtained by aggregating the candidates from all three workers. The detailed methodology is outlined as follows: First, each of the workers assigns candidate labels with non-zero probabilities to form a label set per image. A higher probability indicates a higher likelihood of being the ground-truth label, while a probability of zero implies the label is a non-candidate label. Second, after obtaining three label sets for each image, we derive the final candidate label set through the following procedure. A label present in two or three label sets is included in the final candidate label set. If the final candidate label set comprises only one or no label, we select the labels corresponding to the highest probability in each label set. \n\nUtilizing this procedure, the candidate label sets for more than $98.5\\\\%$ of the samples contain the ground truth labels in the CRC-MIPL dataset. Compared to involving experts to annotate a ground-truth label for each image, our annotation approach significantly alleviates the burden on experts while yielding commendable results. Thus, we argue that this application is indeed compelling. MIPL holds considerable promise across diverse computer vision scenarios, including histopathological image classification, fine-grained image classification, video classification, and video anomaly detection. At present, MIPL remains in its nascent stage, and CRC-MIPL is one real-world dataset that will be made publicly available. Moving forward, we intend to curate additional datasets tailored to various scenarios to provide further support to the research community.\n\nReferences:\n\n[a] Humayun Irshad, Eun-Yeong Oh, Daniel Schmolze, Liza M Quintana, Laura Collins, Rulla M Tamimi, and Andrew H Beck. Crowdsourcing scoring of immunohistochemistry images: Evaluating performance of the crowd and an automated computational method. Scientific Reports, 7(1):43286:1-10, 2017.\n\n[b] Anne Grote, Nadine S. Schaadt, Germain Forestier, Cédric Wemmert, and Friedrich Feuerhake. Crowdsourcing of histological image labeling and object delineation by medical students. IEEE Transactions Medical Imaging, 38(5):1284–1294, 2019.\n\n[30] Wei Tang, Weijia Zhang, and Min-Ling Zhang. Multi-instance partial-label learning: Towards exploiting dual inexact supervision. Science China Information Sciences, pages 1–16, 2023.\n\n[44] Jakob Nikolas Kather, Johannes Krisam, Pornpimol Charoentong, Tom Luedde, Esther Herpel, Cleo-Aron Weis, Timo Gaiser, Alexander Marx, Nektarios A Valous, Dyke Ferber, et al. Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study. PLoS Medicine, 16(1):e1002730:1–22, 2019."}, {"review_id": "sbwAQhC8cA", "reviewer": "Reviewer_EhJn", "summary": "The paper introduces a novel algorithm named Disambiguated attention Embedding for Multi-Instance Partial-Label learning (DEMIPL) to address the problem of multi-instance partial-label learning (MIPL). The proposed algorithm embeds multi-instance bags into a single vector representation using a disambiguation attention mechanism and a momentum-based disambiguation strategy. The paper also presents a real-world MIPL dataset for colorectal cancer classification. Experimental results demonstrate the superiority of DEMIPL over existing MIPL and partial-label learning approaches.", "questions": "See the above.", "limitations": "See the above.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. DEMIPL introduces a unique scheme of embedding multi-instance bags into a single vector representation, addressing the limitations of existing MIPL approaches.\n2. Disambiguation Attention: The use of a disambiguation attention mechanism allows for better aggregation of bag-level information, considering global context, and reducing sensitivity to negative instance predictions.\n3. Momentum-Based Disambiguation: DEMIPL employs a momentum-based strategy to identify the ground-truth label, improving the accuracy of label prediction.\n4. The paper introduces a real-world MIPL dataset for colorectal cancer classification, enabling evaluation in a practical setting. Experimental results demonstrate that DEMIPL outperforms other established MIPL and partial-label learning approaches, showcasing its effectiveness in solving the MIPL problem.", "weaknesses": "1. When discussing the experimental results, provide more information about the benchmark datasets used for evaluation. Explain the characteristics and sizes of the datasets, as well as any preprocessing steps applied. Additionally, describe the performance metrics used to assess the classification accuracy and provide a clear comparison with existing MIPL and partial-label learning approaches.\n2. While the proposed DEMIPL algorithm demonstrates superior performance compared to existing approaches, it is important to acknowledge any limitations of the proposed method. Discuss potential scenarios or datasets where DEMIPL may not perform optimally and propose directions for future research to address these limitations.\n3. Proofread the paper for grammatical errors and improve the overall clarity of the writing.\n4. Consider providing more details about the real-world MIPL dataset for colorectal cancer classification. Describe the acquisition process of the dataset, the number of images per class, and any challenges or biases associated with the dataset.\n5. Consider providing a high-level visual representation or diagram of the DEMIPL algorithm to aid in understanding the proposed approach.\n6. Ensure consistency in notation and symbols throughout the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and insightful suggestions to enhance the quality of our manuscript. We have diligently integrated your input into the revised version. In the following sections, we present your comments, followed by our detailed responses.\n\n> When discussing the experimental results, provide more information about the benchmark datasets used for evaluation. Explain the characteristics and sizes of the datasets, as well as any preprocessing steps applied. Additionally, describe the performance metrics used to assess the classification accuracy and provide a clear comparison with existing MIPL and partial-label learning approaches.\n\nDue to space constraints, we provided a concise overview of the benchmark datasets in Section 3.1. For a more comprehensive exposition, encompassing processing steps, we have elaborated on these aspects in the Appendix.\n\nThe performance metric employed in multi-instance partial-label learning (MIPL) is the conventional classification accuracy for multi-classification, defined as $\\text{accuracy}=\\frac{1}{m}\\sum_{i=1}^{m}\\mathbb{I}(f(\\boldsymbol{z}_i), \\boldsymbol{Y}_i)$, where $m$ denotes the number of multi-instance bags in the test dataset, and $\\boldsymbol{z}_i$ and $\\boldsymbol{Y}_i$ signify the aggregated feature representation and the ground-truth label of the $i$-th multi-instance bag.\n\nRegarding the MIPLGP algorithm, we utilize the same dataset as DEMIPL and adopt the parameter settings recommended in [30]. As partial-label learning (PLL) algorithms are not inherently suited for MIPL data, we apply Mean and Maxmin strategies to transform MIPL datasets into PLL-compatible ones (lines 205-209). Moreover, we have augmented the revised version with further detailed comparisons and in-depth analyses of divergent outcomes.\n\n\n> It is important to acknowledge any limitations of the proposed method. Discuss potential scenarios or datasets where DEMIPL may not perform optimally and propose directions for future research to address these limitations.\n\n\nIn the Appendix, we have attached limitations as follows:\n\nDEMIPL exhibits certain limitations, and there are several unexplored directions in the realm of MIPL. For example, DEMIPL assumes independence among instances within each bag. A promising avenue for future research involves considering dependencies between instances during aggregation. Moreover, akin to multiple-instance learning (MIL) algorithms grounded in the embedded-space paradigm, accurately predicting instance-level labels poses a challenging endeavor. One possible approach entails the introduction of an instance-level classifier. Furthermore, DEMIPL operates under the assumption that the ground-truth label of each multi-instance bag resides within the candidate label set. In the future, relaxing this assumption could encompass scenarios where some ground-truth labels of multi-instance bags lie outside the candidate label set.\n\n\n> Proofread the paper for grammatical errors and improve the overall clarity of the writing.\n\nWe have thoroughly and persistently proofread the paper to rectify any grammatical concerns and ensure that the content is presented with utmost clarity.\n\n\n> Consider providing more details about the real-world MIPL dataset for colorectal cancer classification. Describe the acquisition process of the dataset, the number of images per class, and any challenges or biases associated with the dataset.\n\nDue to limited space, we only offered a concise overview of the real-world CRC-MIPL dataset in Section 3.1 and Appendix A.5. We now provide additional comprehensive details about the CRC-MIPL dataset, which will be added to the revised manuscript.\n\nWe select $7000$ images from the NCT-CRC-HE-100K dataset [44] for colorectal cancer classification, with $1000$ images per class. Instance-level features for each image are obtained using four image bag generators, while the candidate label set for each image is derived from the annotations of three crowdsourcing workers. The sample distribution is balanced across classes. The dataset poses significant challenges due to the lack of exact label information, and approximately $1.5\\\\%$ of ground-truth labels are not included in the candidate label set.\n\n> Consider providing a high-level visual representation or diagram of the DEMIPL algorithm to aid in understanding the proposed approach.\n\nWe will incorporate your suggestion by including a high-level visual representation or diagram of the DEMIPL algorithm in the revised version. The high-level procedure of DEMIPL can be described as follows: given a histopathological image along with its corresponding candidate label set, we aim to provide a clear visual overview. The following outlines the succinct workflow. Initially, we employ an image bag generator to extract instance-level features. Subsequently, an attention mechanism, implemented through a fully connected network, calculates attention scores. These attention scores, in conjunction with the instance-level features, are used to aggregate a bag-level feature representation. Finally, a classifier is employed to predict the class probabilities of the bag-level features.\n\n> Ensure consistency in notation and symbols throughout the paper.\n\nWe have conducted thorough and meticulous reviews of the entire paper, implementing essential revisions to ensure consistency in notation and symbols. For instance, we have replaced $\\boldsymbol{S}$ with $\\mathcal{S}$ to denote the candidate label set and modified Equation (4) to $\\mathcal{L}\\_a = - \\sum\\_{i=1}^{m}\\sum_{j=1}^{n_i} a_{i,j} \\log a_{i,j}$.\n\nReferences:\n\n[30] Wei Tang, Weijia Zhang, and Min-Ling Zhang. Multi-instance partial-label learning: Towards exploiting dual inexact supervision. Science China Information Sciences, pages 1–16, 2023.\n\n[44] Jakob Nikolas Kather, et al. Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study. PLoS Medicine, 16(1):e1002730:1–22, 2019."}, {"review_id": "oFrhuWw2Oc", "reviewer": "Reviewer_suFM", "summary": "The article proposes a novel algorithm called DEMIPL for addressing the challenges in multi-instance partial-label learning (MIPL) tasks. The algorithm uses a disambiguation attention mechanism to embed multi-instance bags into a single vector representation and a momentum-based disambiguation strategy to identify the ground-truth label from candidate label sets. Experimental results on benchmark and real-world datasets demonstrate the superiority of DEMIPL over existing MIPL approaches.", "questions": "1. In addition to the MNIST-MIPL and FMNIST-MIPL data sets, what is the input of other data sets to attention, are the features alreadly extracted?\n2. Comparing the attention-based network with the comparison method, is there any unfairness?", "limitations": "See above questions.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is easy to read and the core idea is clearly presented.\n2. DEMIPL proposes a new algorithm that addresses the limitations of existing MIPL approaches and introduces a disambiguation attention mechanism and momentum based disambiguation strategy.\n3. The article presents experimental results on benchmark and real-world datasets, demonstrating the superiority of DEMIPL over other approaches.", "weaknesses": "1. Recently,  using distribution of labels to generate partial labels have been considered as more difficult data scenario,  which  increases the difficulty for the model to identify similar samples. The authors are  encoraged to  incorporate such type of data for experiments.\n2. The reason why Eq. 7 uses model prediction confidence and weights in previous training rounds needs to be further clearly clarified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing constructive feedback and insightful comments to enhance our paper. We greatly value your recognition of the positive aspects of our work. We will now address the specific concerns you raised.\n\n> Recently, using distribution of labels to generate partial labels have been considered as more difficult data scenario, which increases the difficulty for the model to identify similar samples. The authors are encoraged to incorporate such type of data for experiments.\n\nThe approach you mention for generating partial labels bears a resemblance to instance-dependent partial-label learning [23], which generates partial labels for benchmark datasets based on predicted label distributions. Our proposed CRC-MIPL dataset comprises $7$ categories, each containing $1000$ images. Noteworthy similarities exist both within individual images and in their candidate label sets. Consequently, experimental results on the CRC-MIPL dataset can effectively showcase DEMIPL's proficiency in handling more intricate scenarios. We appreciate your astute suggestion and have plans to create dedicated datasets in the future.\n\n\n> The reason why Eq. 7 uses model prediction confidence and weights in previous training rounds needs to be further clearly clarified.\n\nFirst, we expound upon the necessity for weight updates. Subsequently, we elucidate the motivation behind employing Equation (7) for weight updates.\n\nIn the realm of partial-label learning (PLL), the empirical superiority of PRODEN over PRODEN-naive validates the practice of updating weights, which typically yields improved outcomes in contrast to scenarios where weights remain unchanged [39]. This observation extends to multi-instance partial-label learning (MIPL) as well. As illustrated in Figure 4, the classification accuracy of DEMIPL and DEMIPL-PR surpasses that of DEMIPL-AV.\n\nFurthermore, MIPL data involves dual inexact supervision. The presence of inexact supervision in multi-instance bags affects the disambiguation, and inexact supervision within the candidate label set influences the aggregation of bag-level features. As the count of false positive labels increases, the accuracy of prediction confidence diminishes. Therefore, directly updating weights using low-confidence predictions can pose challenges. Instead, we choose to gradually calibrate the model's output by combining the prediction confidence of the current round with that of the previous round. As training rounds progress, the model's predictions gradually become more accurate, leading to a decrease in the weight of the previous round's prediction confidence. This weight update strategy effectively enhances classification accuracy, particularly when dealing with a substantial number of false positive labels.\n\nFor these reasons, we adopt the weighted sum of prediction confidence from the current round and the previous round as the new weight.\n\n\n\n> In addition to the MNIST-MIPL and FMNIST-MIPL data sets, what is the input of other data sets to attention, are the features alreadly extracted?\n\nYes. The benchmark datasets are curated and made openly available in [30]. For the MNIST-MIPL and FMNIST-MIPL datasets, we utilize a convolutional neural network with two convolutional layers to extract primitive features of size $28 \\times 28 \\times 1$. The Birdsong and SIVAL datasets have their features extracted and released by [42] and [43], respectively. These datasets are transformed into MIPL datasets in [30], namely Birdsong-MIPL and SIVAL-MIPL. To leverage the pre-extracted features, we employ a fully connected network.\n\n\n\n> Comparing the attention-based network with the comparison method, is there any unfairness?\n\nWe believe that there is no inherent unfairness in our approach. To the best of our knowledge, DEMIPL represents the second algorithm in this field. Currently, the only existing MIPL algorithm is MIPLGP [30]. Given that PLL algorithms are incapable of directly handling MIPL data, we are compelled to transform the MIPL data into PLL data using the Mean and Maxmin degradation strategies. Initially, we attempted to employ a multi-instance bag's candidate label set for each individual instance within the bag. Unfortunately, due to the lack of numerous ground-truth labels for negative instances within the candidate label set, the outcomes of the PLL algorithm proved unsatisfactory. In the future, as more MIPL algorithms become available, our primary focus will be on comparative evaluations against these algorithms.\n\nReferences:\n\n[23] Ning Xu, Congyu Qiao, Xin Geng, and Min-Ling Zhang. Instance-dependent partial label learning. Advances in Neural Information Processing Systems 34, Virtual Event, pages 27119–393 27130, 2021\n\n[30] Wei Tang, Weijia Zhang, and Min-Ling Zhang. Multi-instance partial-label learning: Towards exploiting dual inexact supervision. Science China Information Sciences, pages 1–16, 2023.\n\n[39] Jiaqi Lv, Miao Xu, Lei Feng, Gang Niu, Xin Geng, and Masashi Sugiyama. Progressive identification of true labels for partial-label learning. In Proceedings of the 37th International Conference on Machine Learning, Virtual Event, pages 6500–6510, 2020.\n\n[42] Forrest Briggs, Xiaoli Z. Fern, and Raviv Raich. Rank-loss support instance machines for MIML instance annotation. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Beijing, China, pages 534–542, 2012.\n\n[43] Burr Settles, Mark Craven, and Soumya Ray. Multiple-instance active learning. In Advances in Neural Information Processing Systems 20, Vancouver, British Columbia, Canada, pages 1289–1296, 2007.\n\n"}, {"review_id": "QRUAeZ9m0P", "reviewer": "Reviewer_7xX7", "summary": "The paper proposes the first deep learning-based method for multi-instance partial-label learning (MIPL) problem. The proposed method employs a disambiguation attention mechanism to approximate the contribution of each instance to the bag-level feature and then aggregates a multi-instance bag into a single vector representation, with an attention loss introduced to enhance the discriminativeness between positive and negative instances. The method then uses the extracted feature to identify the ground-truth label from the candidate label set using a momentum-based disambiguation strategy and simultaneously learns the classifier. Furthermore, the paper introduces a real-world MIPL dataset for colorectal cancer classification. Empirical results confirm the effectiveness of the proposed method and the contribution of the key designs included.", "questions": "1.\tI have questions about why does the method work. The momentum-based disambiguation strategy simply minimizes the cross entropy loss between the predicted label and all candidate labels, with importance weights added which are obtained by momentum updates from probability prediction. I wonder why this method ensure that the positive label in the candidate set to be correctly selected (disambiguated) ? If this does not give a correct direction, the representation of the instance bag will also be misled, and how can the model reach satisfactory state when converge as the author claims in line 158 under this situation?\n2.\tThe method proposes to minimize the “entropy” of the attention weights calculated to make the attention scores between positive and false positive labels discriminative enough. However, if multiple instances in the instance bag share this positive label, I worry that simply minimizing the entropy would result in a single instance with high attention score against all other low-score instances, neglecting the other positive instances, so the consistency between attention scores and instance-level labels is broken.\n", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper is reasonably clear and comprehensive.\n2.\tThe paper’s claims are supported by experimental and ablative results.\n", "weaknesses": "1.\tA notation in Section 2.1 at line 91 is incorrect. Since none of the instances in the bag belong to any of the false positive labels as the author claims in line 42, the first \\mathcal{Y} at line 91 should be replaced to \\boldsymbol{S}_{i}.\n2.\tI don’t think the weights calculated by equation (2) is “attention” scores, since the computation process only involves a single instance without interaction between different instances, and there does not exist query, key and value, or they are all the same instance. These scores are only weights that approximate the contribution of different instances to the bag.\n3.\tI have some doubts about the proposed method. See the question part below.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for dedicating your time to reading and reviewing our paper. Your comments have provided valuable insights that greatly contribute to the enhancement of our work. In the following sections, we present your comments and provide detailed responses to each of them.\n\n> A notation in Section 2.1 at line 91 is incorrect. Since none of the instances in the bag belong to any of the false positive labels as the author claims in line 42, the first \\mathcal{Y} at line 91 should be replaced to \\boldsymbol{S}_{i}.\n\nWe have diligently addressed the issue highlighted in your comment and conducted a comprehensive review of the entire paper.\n\n\n\n> I don’t think the weights calculated by equation (2) is “attention” scores, since the computation process only involves a single instance without interaction between different instances, and there does not exist query, key and value, or they are all the same instance. These scores are only weights that approximate the contribution of different instances to the bag.\n\nIn the field of multi-instance learning (MIL), pioneering work by [7] introduced attention mechanisms to consolidate multi-instance bags into single feature representations, where the query, key, and value are all identical. We extend this concept by introducing the disambiguation attention mechanism and conventionally refer to the result of Equation (2) as attention scores. These scores offer insights into the contributions of each instance to the bag's overall representation. We have incorporated these modifications in the revised version.\n\n>  I have questions about why does the method work. The momentum-based disambiguation strategy simply minimizes the cross entropy loss between the predicted label and all candidate labels, with importance weights added which are obtained by momentum updates from probability prediction. I wonder why this method ensure that the positive label in the candidate set to be correctly selected (disambiguated) ? If this does not give a correct direction, the representation of the instance bag will also be misled, and how can the model reach satisfactory state when converge as the author claims in line 158 under this situation?\n\nThe empirical risk of MIPL can be expressed as $\\mathcal{R}(f)=\\mathbb{E}[\\mathcal{L}\\_{MIPL}(\\boldsymbol{X}\\_i, \\boldsymbol{S}\\_i)]$. To estimate the empirical risk, an intuitive approach is to define $\\mathcal{L}\\_{MIPL}(\\boldsymbol{X}\\_i, \\boldsymbol{S}\\_i)= \\text{min}\\_{Y\\_i \\in \\boldsymbol{S}\\_i}\\mathcal{L}(\\boldsymbol{X}\\_i, Y\\_i)$. However, straightforwardly optimizing the above empirical risk has two drawbacks: firstly, optimizing the $\\min$ operator is intricate, and secondly, if an incorrect label is chosen initially, subsequent iterations will also be erroneous. To address these issues, we propose to assign higher weights to labels with lower loss values. \n\nDue to the existence of dual inexact supervision in multi-instance partial-label learning (MIPL), the learning challenge surpasses that of partial-label learning (PLL). If only the current round's model output is used as weights, the disambiguation performance would deteriorate when confronted with an increased number of false positive labels. Therefore, in order to enhance the disambiguation effectiveness of our algorithm under conditions of a substantial number of false positive labels, we propose using a weighted sum of previous and current round prediction probabilities as weights. The strategy facilitates a measure of correction within the iterative process, even in cases where exact labels are not initially discerned. Experimental results in Figure 4 demonstrate the superior effectiveness of the momentum-based disambiguation strategy compared to directly using the current round's output as weights or disregarding weight updates.\n\nSince the attention mechanism and the classifier exert reciprocal influence on each other throughout the training process, model convergence occurs exclusively when both of their respective loss values are minimized. Thus, we posit that a state of satisfaction is attained when the individual loss values of both components independently converge.\n\n\n> The method proposes to minimize the “entropy” of the attention weights calculated to make the attention scores between positive and false positive labels discriminative enough. However, if multiple instances in the instance bag share this positive label, I worry that simply minimizing the entropy would result in a single instance with high attention score against all other low-score instances, neglecting the other positive instances, so the consistency between attention scores and instance-level labels is broken.\n\nThe remarks are insightful, and we have indeed incorporated this consideration during the design of our algorithm. While the attention scores calculated using Equation (2) are bounded within the interval $[0, 1]$, our implementation does not enforce the sum of attention scores across all instances within a bag to equate to $1$. Therefore, we can ensure that, while minimizing the entropy of the attention scores, large attention weights can be assigned to multiple positive instances within a multi-instance bag.\n\nReferences:\n\n[7] Maximilian Ilse, Jakub M. Tomczak, and Max Welling. Attention-based deep multiple instance learning. In Proceedings of the 35th International Conference on Machine Learning, Stockholmsmässan, Stockholm, Sweden, pages 2132–2141, 2018."}, {"review_id": "i53aoohCfm", "reviewer": "Reviewer_hzaM", "summary": "This paper proposes a new algorithm for a weakly supervised learning paradigm called multi-instance partial label learning, where each sample is represented by a multi-instance bag and its label is provided as a set of candidate labels. This is a particularly challenging learning paradigm since learners do not have access to which label is the ground truth label from the candidate label set, and do not know which instance corresponds to the ground truth label.  The authors proposed a deep learning algorithm to solve the problem integrating the attention mechanism with a disambiguation strategy. The attention mechanism map the multi-instance representation into a single-vector and the disambiguation strategy identifies the true label from the candidate label set. The synergy of these two components is validated with a wide range of benchmark datasets. Furthermore, this manuscript also proposed a new crowd-labelled medical dataset that can be used as future benchmarks for this learning task.", "questions": "* DEMIPL performs classification by aggregates the instances into a single-vector bag representation. Would it be better if instance representation and bag representation are used simultaneously for classification and disambiguation?\n* Since the momentum-based disambiguation strategy degenerates to progressive disambiguation strategy proposed in [39], how would the performance be if momentum-based disambiguation is used directly in the algorithm proposed in [39]? This may provide empirical evidence on the synergy of momentum-based disambiguation and disambiguation attention.\n* Is there feasibility to learn which instance corresponds to the true label in the candidate label set, i.e., instance-level classification from MIPL datasets?\n* Is there any plan for releasing the CRC-MIPL dataset?\n", "limitations": "This paper does not explicitly discuss its limitations.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "* The problem setting that motivates this study is novel and potentially impactful. The multi-instance partial label learning paradigm simultaneously addresses the challenges two of the most widely studied weakly supervised learning setting, multi-instance learning and partial label learning.\n* The paper introduces a new real-world dataset for histopathology image classification from weak supervision in the form of multi-instance bags supervised with candidate label sets. This indicates that the proposed learning paradigm and algorithm can potentially alleviate the labelling cost of medical image dataset. \n* The proposed disambiguation attention mechanism applies to both binary and multi-class classification scenarios, whereas existing attention mechanisms designed for multi-instance bags can only handle binary classification.\n* The proposed algorithm demonstrates good performance improvement compared to a wide range of existing arts.\n", "weaknesses": "* The synergy of disambiguation attention mechanism and momentum-based disambiguation strategy can be further explored/evaluated. See questions for more on this. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your constructive feedback and comments, which have proven instrumental in enhancing the quality of our paper. Your recognition of the positive aspects of our work is sincerely valued. In the subsequent sections, we present your comments and our responses in a point-by-point manner.\n\n> The synergy of disambiguation attention mechanism and momentum-based disambiguation strategy can be further explored/evaluated. \n\nBelow, we elaborate further on key points:\n\nThe MIPL dataset inherently features dual inexact supervision. The presence of inexact supervision within multi-instance bags directly impacts the disambiguation process, while the inexact supervision associated with candidate label sets influences the aggregation of features at the bag level. First, the disambiguation attention mechanism plays a pivotal role in aggregating multi-instance bags into a coherent bag-level feature representation. Precise predictive probabilities contribute significantly to refining the accuracy of the aggregated bag-level features. Second, the classifier employs this feature representation as input, generating predictive probabilities for the given multi-instance bag. Accurate bag-level features markedly enhance the precision of the predictive probabilities. Last, the comprehensive loss function entails a weighted summation of losses from both the disambiguation attention mechanism and the momentum-based disambiguation strategy. Any failure of either of the losses to converge could potentially hamper the performance of the other component. The model attains convergence solely when both these losses converge. Consequently, we posit that the disambiguation attention mechanism and the momentum-based disambiguation strategy exhibit a synergistic interplay.\n\n> DEMIPL performs classification by aggregates the instances into a single-vector bag representation. Would it be better if instance representation and bag representation are used simultaneously for classification and disambiguation?\n\nThis comment presents an intriguing perspective. We hypothesize that a potential avenue lies in effectively leveraging both instance and bag representations concurrently, leading to a valuable integrated representation for classification and disambiguation purposes. Nonetheless, the challenge lies in identifying such an effective approach; otherwise, the risk of overfitting remains a concern.\n\n\n> Since the momentum-based disambiguation strategy degenerates to progressive disambiguation strategy proposed in [39], how would the performance be if momentum-based disambiguation is used directly in the algorithm proposed in [39]? This may provide empirical evidence on the synergy of momentum-based disambiguation and disambiguation attention.\n\nWe replace the progressive disambiguation strategy in PRODEN with the momentum-based disambiguation strategy, which we refer to as PRODEN-MD. Both PRODEN and PRODEN-MD utilize the parameter recommendations outlined in [39]. Aside from the disambiguation strategy, no other distinctions exist between the two approaches. For the MNIST, FMNIST, and KMNIST datasets, we set the flipping probabilities $q$ to $\\\\{0.1, 0.3, 0.5, 0.7, 0.9\\\\}$ and perform $5$ runs for each configuration. The resulting mean accuracy and standard deviation are presented in the table below.\n\n| Dataset | Algorithm |      $q=0.1$      |      $q=0.3$      |      $q=0.5$      |      $q=0.7$      |      $q=0.9$      |\n| ----- | ------- | :---------------: | :---------------: | :---------------: | :---------------: | :---------------: |\n|  MNIST  |  PRODEN   | $0.926 \\pm 0.000$ | $0.925 \\pm 0.001$ | $0.923 \\pm 0.001$ | $0.916 \\pm 0.001$ | $0.900 \\pm 0.002$ |\n|         | PRODEN-MD | $0.927 \\pm 0.001$ | $0.925 \\pm 0.001$ | $0.923 \\pm 0.001$ | $0.918 \\pm 0.001$ | $0.902 \\pm 0.002$ |\n| FMNIST  |  PRODEN   | $0.843 \\pm 0.001$ | $0.843 \\pm 0.001$ | $0.840 \\pm 0.002$ | $0.834 \\pm 0.002$ | $0.818 \\pm 0.003$ |\n|         | PRODEN-MD | $0.844 \\pm 0.000$ | $0.842 \\pm 0.001$ | $0.840 \\pm 0.002$ | $0.835 \\pm 0.001$ | $0.820 \\pm 0.002$ |\n| KMNIST  |  PRODEN   | $0.695 \\pm 0.002$ | $0.691 \\pm 0.002$ | $0.681 \\pm 0.003$ | $0.668 \\pm 0.005$ | $0.632 \\pm 0.003$ |\n|         | PRODEN-MD | $0.697 \\pm 0.001$ | $0.691 \\pm 0.002$ | $0.680 \\pm 0.003$ | $0.666 \\pm 0.004$ | $0.639 \\pm 0.003$ |\n\nThe results indicate that, in most instances, PRODEN-MD achieves a slightly higher average accuracy compared to PRODEN, albeit with a negligible difference. We attribute this phenomenon to the presence of dual inexact supervision in MIPL data, in contrast to PLL data which only contains inexact supervision in the output space. Given the lower learning complexity of PLL compared to MIPL, the progressive disambiguation strategy suffices to achieve satisfactory performance in disambiguation. Additionally, this underscores the significant performance improvement achieved through the synergistic utilization of the momentum-based disambiguation strategy and the attention mechanism. Your insights are greatly appreciated.\n\n\n\n> Is there feasibility to learn which instance corresponds to the true label in the candidate label set, i.e., instance-level classification from MIPL datasets?\n\nWe posit the feasibility of acquiring an instance-level classifier from MIPL datasets. Our forthcoming endeavors will encompass the design of a dedicated instance-level classifier aimed at predicting instance-level labels, facilitating the identification of instances aligned with ground-truth labels.\n\n\n\n> Is there any plan for releasing the CRC-MIPL dataset?\n\nUpon acceptance of the paper, we will make the CRC-MIPL dataset and the DEMIPL code publicly available as open-source.\n\nReferences:\n\n[39] Jiaqi Lv, Miao Xu, Lei Feng, Gang Niu, Xin Geng, and Masashi Sugiyama. Progressive identification of true labels for partial-label learning. In Proceedings of the 37th International Conference on Machine Learning, Virtual Event, pages 6500–6510, 2020.\n"}]

Author rebuttal:

Meta-review: This paper explores multi-instance partial-label learning (MIPL), a domain where bags of instances are associated with multiple candidate labels, including one true label. The proposed DEMIPL algorithm tackles this challenge by embedding multi-instance bags into single vector representations. It leverages a disambiguation attention mechanism and a momentum-based disambiguation strategy to identify the ground-truth label. The paper introduces a real-world MIPL dataset for colorectal cancer classification, highlighting DEMIPL's superior performance compared to other MIPL and partial-label learning methods.

The reviewers recognize the significance of this innovative problem setting and the potential impact of building the real-world dataset. The design of the attention mechanism is deemed reasonable, and the experimental results show promise. Most major concerns raised during the rebuttal have been addressed. However, some reviewers still have reservations about the practicality of the problem and suggest further discussion of its limitations.
Decision: Accept (poster)