Rethinking Multiple-Instance Learning From Feature Space to Probability Space
Authors: Zhaolong Du
Shasha Mao
Xuequan Lu
Mengnan Qi
Yimeng Zhang
Jing Gu
Licheng Jiao
Venue: ICLR 2025 Poster (2025)
Keywords: Multiple-Instance Learning；Representation Learning；Semi-supervised Learning; Weakly-supervised Learning;
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Transition MIL model to align and aggregate instance features in probability space to improve the MIL model

Abstract: Multiple-instance learning (MIL) was initially proposed to identify key instances within a set (bag) of instances when only one bag-level label is provided. Current deep MIL models mostly solve multi-instance problem in feature space. Nevertheless, with the increasing complexity of data, we found this paradigm faces significant risks in representation learning stage, which could lead to algorithm degradation in deep MIL models. We speculate that the degradation issue stems from the persistent drift of instances in feature space during learning. In this paper, we propose a novel Probability-Space MIL network (PSMIL) as a countermeasure. In PSMIL, a self-training alignment strategy is introduced in probability space to cope with the drift problem in feature space, and the alignment target objective is proven mathematically optimal. Furthermore, we reveal that the widely-used attention-based pooling mechanism in current deep MIL models is easily affected by the perturbation in feature space and further introduce an alternative called probability-space attention pooling. It effectively captures the key instance in each bag from feature space to probability space, and further eliminates the impact of selection drift in the pooling stage. To summarize, PSMIL seeks to solve a MIL problem in probability space rather than feature space. Experimental results illustrate that PSMIL could potentially achieve performance close to supervised learning level in complex tasks (gap within 5\%), with the incremental alignment in propability space bring more than 19\% accuracy improvements for current existing mainstream models in simulated CIFAR datasets. For  existing publicly available MIL benchmarks/datasets, attention in probability space also achieves competitive performance to the state-of-the-art deep MIL models. Codes are available at \url{https://github.com/LMBDA-design/PSAMIL}.

Published as a conference paper at ICLR 2025
RETHINKING MULTIPLE-INSTANGE LEARNING FROM FEATURE SPACE TO PROBABILITY SPACE
Zhaolong Du', Shasha Mao'; Xuequan Lu², Mengnan Qi',
Yimeng Zhang', Jing Gu', Licheng Jiao' Xidian University, The University of Western Australia
ABSTRACT
Multiple-instance learning (MIL) was initially proposed to identify key instances within a set (bag) of instances when only one bag-level label is provided. Current deep MIL models mostly solve multi-instance problem in feature space. Nev- ertheless, with the increasing complexity of data, we found this paradigm faces significant risks in representation learning stage, which could lead to algorithm degradation in deep MIL models. We speculate that the degradation issue stems from the persistent drift of instances in feature space during learning. In this paper, we propose a novel Probability-Space MII. network (PSMIL.) as a coun- termeasure. In PSMIL. self-training alignment strategy is introduced in prob- ability space to cope with the drift problem in feature space, and the alignment target objective is proven mathematically optimal. Furthermore, we reveal that the widely-used attention based pooling mechanism in current deep MIL models is easily affected by the perturbation in feature space and further introduce an al- ternative called probability-span space attention pooling. It effectively captures the key instance in each bag from feature space to probability space, and further eliminates the impact of selection drift in the pooling stage. To summarize, PSMIL seeks solve a MIL problem in probability space rather than feature space. Experimental results illustrate that PSMIL could potentially achieve performance close pervised learning level in complex tasks (gap within 5%), with the incremental alignment in propability space bring more than 19% accuracy improvements for current existing mainstream models in simulated CIFAR datasets. For existing publicly available MIL benchmarks/datasses, attention in probability space also achieves competitive performance to the of deep MIL models. Codes are available at httptt::////g tttp::///guuuuuuuu.. www..ttt.....a......ooooooa deseeaDA--eeiioee
desi -ww.//DDdddiiiiiin/SaMi.a
INTRODUCTION
Multiple-Instance Learning (MIL) (Dietterich et al. 1997: Maron & Lézeno-Pérez, 1997) was intro- duced to identify the key instances within a set of instances when only a bag-level label (indicating whether there is any key instance in the set) is available. Originated in the machine learning era, MIL was initially studied as a pure classification algorithm, thus it heavily relies on simple initial inputs. MIL has found wide upplications in coarse label learning for images, videos, and texts. With the rise of the deep learning, the deep MIL. models are typically formulated as representation learning, followed by classification. However, existing works in MIL have primarily focused relatively simple data inputs such as raw properties of molecular, text statistical representations (Andrews al. 2002 Zhou al. 2009). Even in recent complex applications such as cancer analysis based on W hole Slide Images (WST) (Bejnordi et al. 2017) and video-based anomaly detection (Sultani al. 2018), is still necessary to simplify the instance representations as much as possible before taking them as inputs for the MIL model. To satisfy the conventional requirements of MIL, such complex data instances need to be pre-processed in a certain way (for example, pre-extracting the instances by a heavyweight pre-trained model) to ensure the initial separability as the model input, to guarantee low classification difficulties. Under such condition, the representation learning stage
*Corresponding author. Email: ssmao @ wwdiixiiddiuu. edu.cn. This work was supported in part by the State Key Program of National Natural Science of China under Grant 62234010.

---

Published as a conference paper at ICLR 2025
could then be simplified to a few (or no) fully-connected layers to achieve satisfactory performance, and in this way deep MIL models could be able to focus on classification stage.
In contrast to the simplistic practice commonly adopted in current MIL, many deep-learning based classification tasks pose significant challenges due to the inherent complexity of the data. It is un- likely that simplified representation extraction process can effectively handle all complex cases. Rather than taking simplified inputs, representation learning of complex data often requires se- rious consideration in many weakly-supervised tasks (Rolnick al. 2017 Zhang al. 2016
However, introducing a sophisticated feature ex- tractor for complex data instances in an end-to-
end MIL model is not a straightforward process:
Test Accuracy (5)
Entropy of Precicted Probability
the increased complexity of the feature extrac- tor brings greater capability, but more instances or more accurate label guidance are meanwhile required to ensure the quality of representation learning. While in cases where data is limited or the label guidance is insufficient, such feature tractors are prone to under-fitting thus failing to learn useful representations. In the absence of in- stance labels to guide the representation learning for instances, the complex data/feature extractor may easily lead the MIL model to learn irregular or non-discriminative representations, eventually resulting in degradation for the MIL model (an il- lustration shown in Figure I To address this key problem, we argue that the learning process of ev- ery instance in a bag deserves more attention.
2.8
1.95
0.85
Epochs
Figure 1: A mainstream MIL model (ABMIL) with complex inputs. For a 100-class classification task, the performance variation over 20 epochs on the CIFAR-100 100 dataset from ABMIL is reported. The MIL model is trained in multi-instance mode with complex feature extractor (ResNet) being trained si- multaneously.
In this paper, we first point out rather common issue in current deep MIL models, which is the difficulty in conducting general representation learning on data. As the complexity of data increases, MIL. models tend to learn drifted instance representations, ultimately leading to model degradation. To address this, we design and constrain the MIL algorithm in probability space, and propose the SobbabliitySSpace MIL network (PSMIL). PSMIL comprises two key strategies implemented in probability space, including probability-space attention pooling and probabílity-space alignment objective. To demonstrate the potential degradation issues that current MIL. models may face in feature space, we introduce comprehensive simlulated datasets to evaluate the ability of MIL mod- els in learning instance representations. On complex tasks, experiments show that the designed probability- -aaabiiitt-spcceeeeeee alignment objective effectively constrains instance representations to a more sta- ble space during the representation learning stage, meanwhile bringing non-trivial performance im- provements and stability across current MIL methods. In addition, we analyze the mechanism of the method and demonstrate that it is also easily affected by the contin-
widely used attention-poolie uous drift of instance features. Then, the probability- eliminate the impact of drift problem in the pooling stage of MIL paradigm. With these two new strategics in probability space, the model can circumvent the drift issues that traditional MIL models may face in feature space, achieving performance close that of fully supervised models more complex tasks (gap within 5%). We also validated our model on various existing MII. datasets with SOTA-Ievel performance.
attention pooling is proposed to further
RELATED WORK
Multiple-instance learning was initially defined as solving binary classification problems given fixed instance features and bag-level labels, evaluated at the bag-level accuracy during the test. In deep learning era, the mainstream MIL models typically consist of a feature extractor, an instance pool- ing component, and a classifier concatenated together. The deep MIL model first extracts instance features through the feature extractor, aggregates features into a legelevel feature via the pooling component, and then feeds them into the classifier for classification.
2

---

Published as a conference paper at ICLR 2025
Instance Distribution
Probability spase
Gass Pretohyee Q
Aug pig)
(H
(a splar Pal Poose Morti
quor
Peckchiity space (ckysst)
Adamiio
Gussfier
Tiigg
Execder
Aunzaciatio
[xn(X)
Ingut: muttiinstance bag
Feature extraction
Probability space Algomment
Figure 2: Overview of our Probability-span Calttple-Inssance Learning Network (PSMIL).
Research on pooling/classfficationn strategies in deep MIL model is very progressive. Initially, deep MIL models take instances or instance features input, simply selecting the instance with the high- est score (thought as key instance) as the bag-level feature (max pooling). Recently, Attention-bass MIL. (ABMIL) (IIse et al. (0188) has been one of the prominent works where the attention module is introduced into deep models as pooling component in the MIL paradigm. These methods have been validated on simple benchmarks such as MNIST images (LeCun et al.) 1998) and other manually crafted features (Andrews et al. 2002). In real applications, such WSI fiagnosis/video-bee anomaly detection, models generally require well-extracted separable features for input. Besides, there are efforts extend MII. multi-class classification problems. Regressor-guideds MII. (Du eet al. 2024) decomposes k-class problems into k binary classification branches, providing solutions on the UNBC pain classification dataset.
Existing works in MIL are usually based on the primitive separability of initial inputs. When the initial inputs are easily ensured separable, the MIL algorithm can mainly focus on pooling com- ponents/instance analysis or loss function design. For example, the MNIST dataset, composed of simple gray images, easily ensures separability after the simple feature extraction. The UNBC fa- cial expression pain dataset also shows small intra-class variations where expressions with the same pain-intensity level exhibit remarkable similarity (Lucey et al. 2012). In VAD problem, features pre- extracted by the I3D network as input exhibit better initial separability than C3D, and as result, more than 5% performance improvement under the same conditions compared to the C3D features was achieved (Kamoona al. 2023). Similar to the VAD, recent MIL models used in WSI diag- nosis also adopt pre-extracted convolutional features as input, including the latest work (Fourkioti et al. 2023).
The premise of high-quality input in the MIL paradigm is reasonable but also oversimplified to some extent. Introducing heavyweight feature extractors required by complex data rather than ensuring high quality separable inputs can easily lead to dilemmas when training both the feature extractor and pooling strategy in two stages. As shown in Figure Attention-based MIL (ABMIL) uses R as the feature extractor for multiple-instance learning on CIFAR-100 image bags. Dur- ing the inference stage, when evaluating the accuracy using single CIFAR image, the classification accuracy at random- guessing level indicates that the model did not learn actual class-relevant in- stance representation. With the significant model capability brought by a complex feature extractor, the model potentially extracts features in the way that we do not expect. This is what we call the problem of feature drift.
3 MATERIAL AND METHODS
3.1 PRELIMINARIES: DEEP MIL MODELS REPRESENTED BY ATTENTION-BASED MIL
MIL problem formulated generally: Given a bag composed of n instances, denoted as X [x1, 22,....... ,,,, 1,,, X E R and a bag label Y E {0, 1} which indicates whether an instance with non-zero label (key instance) exists in the bag, we need to learn a model : X + The perfor- mance of the mapping g is typically demonstrated through bag-level binary classification accuracy and instance localization results, where localization reflects the ability to learn and capture key in- stances from a bag, forming the inference basis for MIL model. MII considers individual instances
3

---

Published as a conference paper at ICLR 2025
to be independent, without any sequential or other correlated relationships. Recently, the concept of MIL has been extended to multi-class problems, and Y is denoted as to i: - 1 in k-class scenario.
Due to the effecctivnness of the attention mechanism, Attention- MIL (ABMIL) has been widely applied and further developed. ABMIL and its family are the current mainstream deep MIL models. Deep models in this family often simplify the representation learning stage as much as possible and use attention- pooling strategy to select key instances to form an overall bag-level feature, which is then passed to a classifier. Here we denote the feature matrix obtained by the feature extractor Enc as H E R where each row of H corresponds to the feature of each instance, The pooling strategy then aggregates the feature matrix into a bag-level feature Hbag E R to be sent into the linear classifier to get the bag-level prediction Poay. The parameters of the final linear classification
layer of the MIL model are (C' E R for weights, D' E I for biases). When the raw data is relatively simple, it can often be directly used as model input X. When the raw data is complex, ABMIL typically requires pre-extracted separable features as input X to reduce the difficulty of representation learning. Through this strategy, ABMIL can perform form of limited representation learning using several (or no) simplified layers. When setting the feature extractor to 0 layers, H Enc(X) = X. The overall forward process of ABMIL can be stated follows:
+ = softmax HCagg + D')
Hbag snreettion-pooinng (on(ooliingEn(());;;; Pbay
Recently, there are some advances in iiremion-based MIL. -urea-t MIL (DSMIL) (Li et al., 2021] adjusts intention-baseen pooling by introducing an additional max pooling branch. Double-tier feature distillation MIL (DTFDMIL) (Zhang et al., 2022) applies attention- pooling twice se- quentially by dividing the raw bag into several pseudo-bags. Disambiguated
MIL (DEMIL) (Tang et al. 2024) introduces a value mapping operation to transform H into a hidden space V and applies attention -baded pooling in hidden feature space to address multi-classification problems. Additionally, there are works that directly use transformer block as the pooling compo- nent (TransMIL)) (Shao al., 202T). However, these models all learn with relatively simple feature extractors internally, and thus they are still restricted by the complexity of the data inputs. For gen- eral representation learning, there should not be restriction on oversimplifying the feature extractor or data input. We observed that once the restriction is not strictly satisfied, ABMILS tends to degrade to be random guessing. Specifically, the involved major issues are as follows.
Feature Drift in representation learning. Considering to solve a wuakly-supervised classification problem, ABMILS needs obtain discriminative features relevant to the classification task through the feature extractor. Given commonly-used feature extractor with greater capability, when insuf- ficient guidance is provided it highly likely to learn features irrelevant to classification. As shown in Figure [ we set the feature extractor part of ABMIL to the relatively complex ResNet18, which directly takes bags composed of raw CIFAR- 100 images as inputs for training. During test, we use the classification accuracy of single image to evaluate instance quality directly. Here ResNet18 (He) jet al. (0166) was pre-trained on ImageNet (Deng et al.] 2200)), so the model demonstrated perfor- mance above random guess level in the early training epochs. However, as training progressed, the model eventually converged to a completely random guessing algorithm. Additionally, ABMIL maintains high information entropy for the predicted probability distribution of each instance, in- dicating the low discriminability and confidence of instance features all over the training process. This suggests that the model has always been learning unintended drifted features. The drifted low-quality features could further impact the pooling strategy during training.
Selection Drift in pooling stage. The goal of the instance pooling stage in ABMILS is to aggregate a representative bag-level feature, thereby climinating the influence of non-key instances within the bag. ABMIL utilizes attention based pooling to select key instances and aggregate them into an overall legel feature. A general form of diention-basee pooling is as follows:
ezp(H, H T
Hbag Σ H,H, where oj
(1)
ΣE= eapl H m))
The pooling stage transforms H into weight vector consistent with the bag length n using weights- transformation parameter ÿ E Rd, and aggregates the feature matrix H into a bag-level feature Hong based on the weights d E R².
Proposition 1. Attention-based pooling updates the parameter के based on whether the real- time inferred class of instance is consistent with the bag-level ground truth.
4

---

Published as a conference paper at ICLR 2025
-VaL = =H Pbbgg
(layer-wise differentiation)
=E== 00 (2) Ebag) H;
(decompasition & recombination)
s.t. Σ ク 0: Ør > 0: o7 <0: o E
In k--clas ABMIL model, the update formula is shown above, where L represents the negative log- likelihood loss, Y denotes its ground-truth bag label (scalar), Y denotes any other label in [0, k - 1] except for Y. is the bag ground-truth one-hot label, 7/ TRt, Pbag expresses the model output prediction, Pbag R expresses the Jacobian matrix with shape R=*"" and 2, 2bag E R$ are the logits produced by the model's 's linear classification layer of the instance feature H; and the bag-level feature Hang respectively.
In the detailed derivation presented in Appendix A.4 we have already verified the properties of o Next, we first define 7/ 2 Zbag. Imagine for an instance 2,, the corresponding logits output ž higher than the overall bag level Zbag at position Y and lower at position Y. Therefore, compared to the overall bag output Zbag. the logits output of instance 2 after the softmax function is closer to the one-hot ground truth vector 3/ This is what the model considers as key instance in this bag. We say that when the instance I; has such high level of consistency with the ground truth, the vector 7 satisfies the following constraints. In this sense, the fact and the following constraint are equivalent:
uy
0;
O; d R
T We define con(i) 0² 1. and then we have
-V -
con(i) Hy.
(4)
In the final form above, the sign of the consistency parameter con(i) directly indicates the update direction of w. It is easy verify that when 7/ satisfies the above constraints, implying that the instance has a high level of consistency with ground truth (key instance), the consistency parameter con(1) is positive definite. In this case, the positive direction of H,, which will further lead to an increase in the corresponding weight of the key instance 24. Conversely, the opposite outcome occurs. Thus far, we could summarize the attention mechanism as a complete statement in Proposition 1 above.
transformation parameter Φ is updated in the
As a selection process, the intention of attention-basee pooling to gradually assign higher/lower weights to key/non-key instances during the learning process. Based on the analysis above, the aitention-based mechanism involves examining the real-time probabilistic inference results of in- stances and updating weights T in direction decided by the instances' consistency level with the bag-level ground truth. The transformation parameter // is finally updated based on the instance features H. Considering when simple data inputs, where the given H is either fixed varies reg- ularly, the attention mechanism can continuously update regular instance features to the parameter dt based consistency, leading to increased/deerresed weights of corresponding key/non-ke key in- stances. However, when complex data and the feature extractor with great capability are introduced, the continuous drastic variation of H can significantly impact the learning of weight parameters w. For key instances with inferences more consistent with the ground truth, features are often irregular and constantly changing. Thus, an update from key instance w does not mean an increase in the weight of any key instance. The drift of features further leads to selection drift in this case.
3.2 PROBABILITY-SPACE ALIGNMENT: TOWARDS FEATURE DRIFT
In the representation learning stage, one major issue causing feature drift is the lack of effective guidance for instances. Representation learning is typically guided by labels for classification tasks. In MIL, due to the absence of guidance from instance-level labels, the models often lack confident inference for instances, as reflected in the high entropy of instance probability inference during train- ing shown in Figure I Data augmentation technique is often used in weakly supervised learning to enhance the robustness of the model(Sheng et al. (024) Wu et al. 2022). To provide effective guid- ance, we propose an objective called probability-spee space alignment based on pseudo-label inference, with C.E representing cross entropy function:
5

---

Published as a conference paper at ICLR 2025
Lins (X) m())(a)) Ei=1 EweA(.)) CE(1O, (())),
(5)
where p.0) ( (x))
p(o) RR and 10 onchot(p(o))
Σ, (TAe)) Pi (a()")
Proposition For an augmented set {x', "e","",, ..... A(x), the optimal target pseudo-label is the one-hot encoding of the normalized mutliplicated probabilities within the set.
Proof: We denote the position where the elements in optimal one-hot label equals as and
for any non-optimal one-hot label I, the position where it equals as r. Since / obtained through Proposition 2, the product of probabilities in the augmented set at position c greater than at position We have
Lins (X;1) Lins (X;)
Σ n'EA(z))
(CE (CE(),,(<')) -(x))) 77())) (6)
n. [A(x)
1
Σ
(- log Po((') (- log (x')))
[ax)]
i= ('EA(()))
1
IL (x(2')
log
0
(8)
n. AA(x)
ILLEA((..)) Pe(x')
That is to say, the target label that we adopt Equation enables an optimal loss for Lins
As shown in Equation 5 for each instance in the bag, we generate augmented set A(x) and aligns the augmented instances in A(x) to the inferred label which is obtained from Proposition 2.
A A X * o...ohheiise.
L = (bbgg()) + Ay Lins(X) where Ay
The overall loss function is presented Equation 9. Here the term Lbag () represents the conven- tional loss of deep MIL models, Lbag (X) CF Pbag). The coefficient Xr is controlled by the base parameter X and the current training epoch I,, increasing gradually as the training progresses until it reaches a threshold epoch т. By default, the base parameter is 0.1.
3.3 PROBARILITY-SPAG ATTENTION: TOWARDS SELECTION DRIFT
Probability- space attention applies attention mechanism on the probability space of instances, which the largest difference compared traditional sueenoo-bbased pooling. Specifically, we introduce the class prototypes Q (E Rdx*) to achieve the transformation of the representations from the feature space to the probability space, Through real-time estimation based on the class prototypes Q (E Rdx&), PSMIL turns to select instances in the probability space, achieved by the following formula:
exp(P.T Ø
Hoog Ed.ti, where a;
En_ exp(P,T P= (=H
(10)
Here P (E R expresses probability estimation matrix, which is obtained by computing the similarity between the instance feature matrix H and the class prototype matrix Q. where Vi ex- presses the function of similarity estimation. In implementation, we set P, softmazl (H," TQ)- The estimated probability of each instance further participates intention-hased pooling, as shown in Equation 10 In addition, the probability estimation of each instance in P is processed by the argmaz function, resulting in label estimation.
Based on the estimated label for each instance, we calculate the corresponding mean value to obtain new class prototype matrix Quea- Specifically, the j-th column of Qnew is obtained by averaging

---

Published as a conference paper at ICLR 2025
the features of the corresponding instances in H with the label estimated the j-th class. The class prototype Q is then updated by a momentum- based strategy at each training step (t), formulated as
Q g++ Normalize((1 - 7) Q' + Qnew),
(11)
where 7 is the hyper-parameter defaulting to 0.001 in our experiments.
Under the same condition in Equation 2 for the weighhs-transformation parameter wt E R the update rule in probability-space attention would be redefined as follows
Zhog) P..
(12)
Compared with Equation 2 Equation| 12 shows that the trighttttrassformttion parameter is updated by the pattern of real-time probability estimation P, instead of the real-time feature matrix H. From the perspective of the key instance 24, when the inferred result 2 is more consistent with the ground truth 3/ than the bag level result Zbag. the corresponding estimated probability P. will also be similar to the ground truth y. According to Equation 12 the transformation parameter w is updated in the positive direction close to the ground truth V which ensures increasing the weight of corresponding key instance æ;z As an alternative, the probability -pace pooling strategy is less susceptible to drastic changes in feature space than the previous tbbsed rule.
EXPERIMENTS AND ANALYSES
4.1 SIMULATED EXPERIMENTS TO EVALUATE INSTANCE REPRESENTATION QUALITY
In the context of multi-class multiple-instance learning, current benchmarks only consist of a few simple bag-level evaluation datasets (Briggs et al. 2012 Setles et 2000).. To evaluate the model's ability in representation learning, we first introduce more simulated datasets for comprehensive evaluation.
Table Statistics of synthesized datasets.
Dataset
color
#cls
#ins tr
#ins-te
#dims
#bag-len
#bags-per-cls
#key- peeeyppeccentge(55)
FMNIST
gray
10
60,000
10,000
28*28
64
12,00
7.8
SVHN
RGB
10
73,257
26.032
3*28+28
14.59
CIFAR-10
RGB
10
50,000
10,000
3*32#32
10.00
7.8
CIFAR-100
RGB
100
50,000
10,000
3*32*32
100
7.8
We synthesize the multi-instance bag version based on four corresponding datasets, and the statistics are shown in Table [ These four datasets all belong to the domain of images, with the difficulty of representation learning increasing sequentially from Rashion-MNIST (denoted FMNIST) to CIFAR- (Xiao et al.) 2017 ( Netzer et al. 2011 Krizhevkky et al.) 2009). and we use color, #cls, #ins-tr, #ins-te, #dims, #bag-len, #bag-per-cls, #l-r, and Key-percentage to denote the color mode, number of classes, number of training pustances,numbee of testing instances, dimension of instance, bag length. bag counts per class, key instance percentage in a bag in each dataset, respectively. For more detail about synthesized datasets, see Appendix A.3
4.2 INSTANCE REPRESENTATION QUALITY EVALUATION
Implementation Details. All the algorithms including PSMIL are implemented single Nvidia RTX 4090 GPU. Training in multi-instance mode on complex data generally time-consuming. with single epoch CIFAR-100 possibly taking over 1.5 hours. We apply the stochastic gradient descent (SGD) optimizer with a momentum of 0.9 and a weight decay 0,0001. The initial learning rate is chosen from set of (0.01, 0.001} and is decayed by steps. On first epoch freeze the backbone as warming up to improve stability. The value of is selected from a set of (0.1, 0.01), with the threshold epoch being 10.
Performance And Analyses. In Table 2 the difficulty representation learning increases grad- ually across four datasets, where a two-layer convolutional neural network(2-CNN) is used as the
7

---

Published as a conference paper at ICLR 2025
Table 2: Accuracies of instance-level image classification reported during 20 epochs.
Methods
FMNIST
SVHN
CI
CIFAR-I0U 100
CIFAR-IU 1O
CIFAR-IOU 100
ABMIL
Tise al. 2018
80.81
70.58
Deg
Deg
*69.98
#46.13
DEMIL
lang et al.
2024
80.48
64.39
Deg
Deg
#69.89
#44.82
TiL
Shao
2021
74.14
67.37
Deg
Deg
*55.76
*Deg
DTFDMIL
Zhang et
81.27
78.04
Deg
Deg
*69.82
*44,92
PSMIL
77.62
84.38
88.59
70.4
*83.66
*57.31
(a) FMNIST Eval
(b) SVHN Eval
(c) CIFAR-I0 Eval
(d) CIFAR-106 100 Eval
Figure 3: Evaluation on four simulated datasets during 20 training epochs.
feature extractor for FMNIST while ResNer18 8 is applied for the remaining datasets, and the best per- formance in bold. "Deg" denotes that the model degraded to random guessing when the feature extractor is involved in training.
From Table 2 it seen that PSMIL outperforms the compared methods on three more complex datasets (SVHIN, CIFAR-10 and CIFAR-100), which indicates that PSMIL is more effective for complex datasets than ABMILs. Mcanwhile, the results also demonstrate that most existing models effectively learn discriminative instance features relatively simple data inputs (FMNIST, SVHN). Particularly on the FMNIST dataset, due to the regularity of the data styles and the simplicity of the feature extractor, applying attention mechanisms in the feature space can also lead to excellent performance. But ABMILS could totally fail for complex datasets, like CIFAR-10 10 and CIFAR- 100. The performance at the level of random guessing and the high entropy of instance prediction probabilities during training (an illustration shown in Figure I indicate that these models are not fully capable of handling a general representation learning task, leading to frequent degradation issue during the training process with these complex data.
Moreover, we implement extra experiments which insteads train models in traditional manner using pre-extracted features for two complex datasets (CIFAR- 10 and CIFAR-100), Specifically, we fix the parameters of the pre-trained feature extractor and train the pooling and classification parts of the model. We present the performance marked with "*" in the last two columns of Table [ In this compromised learning approach, ABMILS demonstrate the ability to learn from fixed features without any degradation issue but still exhibit over a 11 % accuracy gap compared with PSMIL on CIFAR-10 10 and CIFAR-100 100.
4.3 ABLATION STUDY& & VISUALIZATION ON REPRESENTATION LEARNING MIL
To enhance the representation learning ability for complex data, our idea fully transfer ABMILS to probability space. And we propose two strategies; a pooling strategy and an alignment objective strategy.
The detailed ablation of two separate strategies on the existing models is shown in Table 3 where the highest accuracy of each model is reported in 20 training epochs. In Table PSAtt and PSAli respectively denote the probability-space attention pooling and the incremental probability-space alignment objective, and RL is an abbreviation for "representation learning". indicating whether the feature extractor is involved in training. The crossmark (x) on the option "RL" indicates that the model fixes the feature extractor with pre-trained weights on ImageNet, leading to no involvement of representation learning, and the results of ABMIL and DEMIL illustrate that such traditional approach guarantees effective performance. However, when we unfreeze the feature extractor, AB- MILS degrade and their results (ABMIL+ and DEMIL+) may "Deg". Noticeably, by applying the PSAli objective, the learning ability of ABMILS is reinvigorated, and an accuracy improvement of between 7% and 17% is obtained compared the original models, shown ABMIL ++ and DE-
8

---

Published as a conference paper at ICLR 2025
MIL++. Finally, by transferrigg the pooling strategy probability space (PSAtt), PSMIL further enhances the quality of instance representations and totally eliminated the degradation issue. We demonstrate the impact of the value of A, which controls the power of alignment objective. Accord- ing to the ablation experiments, the complex data cases when the model may face a significant risk of degradation, the impact of X significant. An appropriate parameter À can enhance the stability and performance of the model, preventing the model from failure. We recommend carefully tuning the alignment parameter À complex datasets. Additionally, the results of supervised learning manner (ResNet18 (Sup.)) are given in the last row, which shows that our solution PSMIL achieves close supervised learning, with accuracy less than 5%.
Table 3: Ablation results on two challenging datasets CIFAR-10 and CIFAR-100.
Datasets
CIFAR-TO
CIFAR- 100
Strategies
RL
PSAB PSAtt
RL
PSAR
PSÄtf
ABMIL
69.98
46.13
DEMIL
69.89
X
44.82
ABMIL+
Deg
Deg
DEMIL+
Deg
Deg
ABMIL+ (Aa)
80.81
63.72
DEMIL++ (A2)
76.65
62.13
Au Ar
82.79
64.14
An Dr =0.01
83.29
64.22
A dr = =0.1
84.77
66.74
Ad AT 0.01 T
88.59
70.4
ResNet18(Sap.)
92.14
71.29
(a) ABMIL (traditona)))
(b) (BMIL(RL)
(c) ABMIL(RL+PSAII)
(d) PSMIL
(c) ResNet18 (Sup.)
Figure 4: T-SNE visualization of the image representation CIFAR-10.
In Figure 4 further demonstrate the capability of representation learning under different condi- tions of the ablation experiments. Figure 4(a) represents initial representation distribution provided by pre-trained ResNetl8. 8, corresponding to the last second column in Table 2 where ABMIL takes pre-extracted features as input without representation learning. Nonetheless, we cannot simply in- volve the feature extractor in training, with the degraded features shown in Figure 4(b) Figure 4(c) demonstrates the enhancement of probability-spar alignment strategy in existing MIL mod- els, where ABMIL ensures effective representation learning for complex data through the constraint of probability-spac alignment. Figure 4(d) provides the clustering effect of our complete solution PSMIL, where we solve the MIL problem totally in probability space. Comparing Figure 4(b) with 4(d) we can see that PSMIL eliminates the feature drift during representation learning, with similar clustering to fully supervised level shown in Figure H(e) See Appendix A.2 for more visualization
results.
4.4 EVALUATION ON EXISTING BENCHMARKS AND LARGE SCALE DATASETS
LARGE-SCALE
We validated that model competitive under traditional datasets and conditions. In existing MIL evaluation, many applications might not have on-the-shelf data augmentation methods. This in- cludes datasets such as handcrafted simple benchmarks (Andrews et al. 2002 Dietterich et 1997)

---

Published as a conference paper at ICLR 2025
and large-scale real-world CAMELYONI6,TCC Lung Cancer datasets et al. 2017). For implementation details see Appendix A.3
Table 4: Results on the small benchmark datasets( accuracy + std-dev). All reimplemented.
Meihods
MUSKI
MUSK2
FOX
TIGER
BLEPHINNT
ABMIL(2008)
0.910 0.118
0.
0.952
0.051
0.042
0.969 + 0.036
Durl-Stream MIL(2021)
0.959 0.053
0.952
0.066
0.939
0.060
0 - 0.053
0.989 1 0. 0.023
TransMIL(202)
0.927 0.093
0.877 = 0.112
0.944 0.050
0.963 = 0.042
0.979 + 0.0030
(EM2L(2024)
0.963 0.073
0.961 = 0.057
0.941 + 0.047
0.965 + 0.035
0.969 + 0.034
RGMIL(2024)
0.968 0.060
0.963 0.048
0.954 + 0.048
0.949 = 0.047
0.963 + 0.032
PSMIL
0.968 0.053
0.966
0.052
0.9420 + 0.054
0.947 1 0.047
0.985 1 0.030
Table 5: Results on Large-scale Datasets.
Statistics directly collected from Zhang et al.
2022 (Fourkioti et al. 2023)
CAMELYONI6
TCGA Lung Cancer
Method
Multi-scale
ACC(T)
AUC(1)
F1(T)
Multi-scale
ACC(1)
AUC(1)
ABMIL
Single(20x)
0.845
0.854
0.780
Single(5x)
0.869
0,941
TransMIL
Single(20x)
0.858
0.906
0.797
Single(5x)
0.883
0,949
DTFDMIL(AFS)
Single(20x)
0.908
0.946
0.882
Single(5x)
0.891
0.951
MMxMMI))
Single(20x)
0.899
0.941
0.865
Single(5x)
0.894
0,961
DSMIL
Single(20x)
0.856
0.899
0.815
Single(5x)
0.888
0.939
CAMIL
Single(20x)
0.910
0.953
0.872
Single(5x)
0.916
0.975
PSMIL
Single(20x)
0.922
0.956
0.921
Single(5x)
0.938
0.986
Annotated
DSMIL
PSMIL
Annotated
DSMIL
PSMIL
6 0
D
Figure 5: Patch-level Localization of two macro tumors in CAMELYON16
In the traditional pipeline, the features provided by preprocessing. Therefore, for PSMIL, only introduced the probability-space attention pooling in all above datasets. Even though these data do not involve complex representation learning model still achieved SOTA-level performance on the existing datasets as shown in Tabel 45 Table 4 we adopted the untrimmed input data with additional instance label, also provided the trimmed version Appendix A..3 And from Figure (5) our solution in probability space could effectively identify the tumor region among the whole slide with significantly less noise than the feature space rule(see yellow boxes), with explicit reference output for each instance(slide patch). For implementation details, see Appendix A.3
CONCLUSION
Multiple-instance Learning (MIL) originated in the machine learning era as pure classification, and they typically follow the restriction of fixed or simplified inputs. In current deep MIL models, the representation learning stage is often ignored. In this paper, we provide comprehensive experiments to verify that current deep MIL. models tend to learn drifted features in complex tasks, and thus they are still restricted by the complexity of input data. As countermeasure, we propose a novel network called Probability-S0 can eliminate drift issues in the feature space by aligning and pooling the instances in the probability space, thereby enhancing the representation learning capability of MIL models for complex tasks. Experiments on public large-scale datasets have also demonstrated the effectiveness addressing multi-instance problems in the probability space.
MIL network (PSMIL). Theoretical analysis and experiments show that
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Stuart Andrews, Ioannis Tsochantaridis, and Thomas Hofmann. Support vector machines for multiple-instance learning. Advances neural information processing systems, 15, 2002.
Babak Ehteshami Bejnordi, Mitko Veta, Paul Johannes Van Diest, Bram Van Ginneken, Nico Karssemeijer, Geert Litjens, Jeroen AWM Van Der Laak, Meyke Hermsen, Quirine Manson, Maschenka Balkenhol, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases women with breast cancer. Jama, 318(22):219-2222 2017.
Forrest Briggs, Xiaoli Fern, and Raviv Raich. Rank-loss support instance machines for miml in- stance annotation. In Proceedings of the 18th ACM SIGKDD international conference Knowl- edge discovery and data mining, pp. 534-542, 2012.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition,
pp. leee, 2009.
Thomas G Dietterich, Richard Lathrop, and Tomás Lozano-Pérez, Solving the multiple instance problem with axis-parallel rectangles. Artificial intelligence, 89(1-2):31-71, 1997.
Zhaolong Du, Shasha Mao, Yimeng Zhang, Shuiping Gou, Licheng Jiao, and Lin Xiong. Rgmil: guide your multiple-instance learning model with regressor. Advances in Neural Information Processing Systems, 36, 2024.
Olga Fourkioti, Matt De Vries, and Chris Bakal. Camil: Context-aware multiple instance learning for cancer detection and subtyping whole slide images, arXiv preprint vrXiv:2355.051/2 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Maximilian Ilse, Jakub Tomezak, and Max Welling. Attention-based deep multiple instance learn- ing. In International conference on machine learning, pp. 2127-2136. PMLR, 2018
Ammar Mansoor Kamoona, Amirali Khodadadian Gostar, Alireza Bab- Hadiiahaaa,, and Reza Ho- seinnezhad. Multiple ssttace-based video anomaly detection using deep temporal encoding- decoding. Expert Systems with Applications, 214:119077 2023.
Salome Kazeminia, Ario Sadafi, Asya Makhro. Anna Bogdanova, Carsten Marr, and Bastian Rieck. Topologicallyyrrgguaaazz multiple instance learning for red blood cell disease classification. CORR, 2023.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 86 86(11):2278-232) 1998.
Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 14318-14328, 2021.
Patrick Lucey, Jeffrey F Cohn, Kenneth M Prkachin, Patricia E Solomon, Sien Chew, and Iain Matthews. Painful monitoring: Automatic pain monitoring using the mebe-memaster shoulder pain expression archive database. Image and Vision Computing, 397 2012.
Oded Maron and Tomás Lozano-Pérez. framework for multiple-instance learning. Advances in neural information processing systems, 10, 1997.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng. et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 4. Granada, 2011.
feature
11

---

Published as a conference paper at ICLR 2025
David Rolnick, Andreas Veit, Serge Belongie, and Nir Shavit. Deep learning is robust to massive label noise. arXiv preprint 170. 2017.
Burr Settles, Mark Craven, and Soumya Ray. Multiple-instance active learning, Advances in neural information processing systems, 20, 2007.
Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Ad- vances neural information processing systems, 34:2136 2147, 2021.
Mengmeng Sheng, Zeren Sun, Zhenhuang Cai, Tao Chen, Yichao Zhou, and Yazhou Yao. Adaptive integration of partial label learning and negative learning for enhanced noisy label learning- In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4820-4828, 2024.
Waqas Sultani, Chen Chen, and Mubarak Shah. Real-world anomaly detection in surveillance videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 6479-6488, 2018.
Wei Tang, Weijia Zhang, and Min-Ling Zhang. Disambiguated attention embedding for multi- instance partial-label learning. Advances Neural Information Processing Systems, 36, 2024.
Dong-Dong Wu, Deng-Bat -Bo Wang, and Min-Ling Zhang. Revisiting consistency regularization for deep partial label learning. In International conference on machine learning, pp. 24212-24225. PMLR, 2022.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint wiXiv:/70.007444 2017.
Hongrun Zhang, Yanda Meng, Yitian Zhao, Yihong Qiao, Xiaoyun Yang. Sarah Coupland, and Yalin Zheng. Difd-mil: Double-tier feature distillation multiple instance learning for histopathol- ogy whole slide image classification. In Pruceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18802-1838 2022.
Min-Ling Zhang, Bin-Bin Zhou, and Xu-Ying Liu. Partial label learning via feature-aware disam- biguation. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1335-1344, 2016.
Zhi-Hua Zhou, Yu- Yin Sun, and Yu-Feng Li. Multi-instance learning by treating instances as non- iid samples. In Proceedings of the 26th annual international conference on machine learning, pp- 1249-1256, 2009.
A APPENDIX
A. REPRODUCIBILITY STATEMENT
The codes have been uploaded to Github, and we also presented the important model weights/logs trained for validation.
12

---

Published as a conference paper at ICLR 2025
A.2 T-SNE VISUALIZATION THE IMAGE REPRESENTATION IN ALL BENCHMARKS
ABaiLLtzadiiiona)) (b) ABMIL (RL+PSAli)
PSMIL
(d) ResNet18 (Sup.)
Figure 6: FMNIST repprsentaation-custeiiigg visualization results.
(a) ABMII (traditional) (b) ABMIL (RL+PSAII)
(c) PSMIL
(d) ResNer18 (Sup.)
Figure 7: SVHN representationncuustrinng visualization results.
(a) (radtintatiio)aa) (b) ABMIL (RL+PSAli)
(c) PSMIL
(d) ResNet18 (Sup.)
Figure 8: CIFAR-10 representationncuustrring visualization results.
(a) ABaiLttaattoona)) (b) ABMIL (RL+PSALi)
(c) PSMIL
(d) ResNet18 (Sup.)
Figure 9: CIFAR-100 represemationn-cuusteinng visualization results.
As demonstrated below, transitioning the MIL solution entirely into the probability space presented more positive impact on rather complex data.
A.3 DATASETS AND IMPLEMENTATION DETAILS
Detailed properties of 5 benchmark datasets used in Table
DATASET
MUSKI
MUSK2
Elephant
Fox
Tiger
dimension
167
167
231
231
231
# of bags
92
102
200
200
200
# of positive bags
47
39
100
100
100
# instances
476
6598
1391
1320
1220
bag size
40
1044
13
13
13
min bag size
2
In Table the algorithms are all deep MIL models, which we re-implemented and ran using the same evaluation codes, with replication results reported. In the implementation process, introduced
13

---

Published as a conference paper at ICLR 2025
two fully connected layers (with ReLU activation function) for all deep MIL models to fine-tune the original inputs.
We borrowed the publicly available evaluation code of RGMIL Github to re- re-implement all the methods listed in Table g. To be clear, this evaluation code uses untrimmed raw input data, where each instance has additional feature dimension that includes the instance label. A more common implementation of the evaluation may trim off the instance label from the features. As a result, all methods perform relatively better different degree in Table than implemented in the trimmed inputs. Generally, considering that the mining of these instance label features is still meaningful, the performance comparison we presented here similar to the trimmed input version, and this implementation version is also supplemented in our Github repo. Metrics see Table below.
Table 6: Performance comparison provided by Kazeminia et
2023 under trimmed input version.
Method
MUSKI
MUSK2
FOX
TIGER
ELEPHANT
ABMIL
0.892+0.044
0.858#0.048
0.615=0.043
0.839+0.022
0.868=0.022
Gated-ABMIL
0,900=0.050
0.863=00042
0.603#0.029
0.845#0.018
0.857=0.027
DPMIL
0.907=0.036
0.926+0.043
0.655#0.052
0.897+0.028
0.894=0.030
DSMIL
0.932+0.024
0.930+0.020
0.729=0.018
0.86940.008
0.925=0.007
BDRMIL
0.926+0.077
0.905+0.092
0.629=0.110
0.86940.066
0.908=0.054
RGMIL
0.940=0.070
0.920*0.106
0.714
0.842+0.088
0.915#0.042
TR-RGMILL
0.946+0.07
0.970#0.042
0.747=0.054
0.961#0.040
0.941
PSMIL
0.962+0.065
0.964+0.057
0.73420.136
0.88440.061
0,918#0.052
In the new synthesized datasets, training set instances are reorganized into the form of multi-instance bags for training. Instances with class label 0 serve background instances, randomly sampled within each multi-instance bag as non-key instances. Each multi-instance bag has a fixed length of 64, with 5 being key instances, accounting for 7.8%. The number of categories of bags generated are the same as the number of classes in each original dataset. For example, in CIFAR-100, a bag labeled 78 contains 5 image instances labeled as 78 (maple-tree images), with the remaining 59 non-key instances sampled from instances with label 0. Key instances are not subject to resampling randomly, meaning the same non-zero image will not appear in different bags. During testing, we directly evaluate instance representation quality based on the classification accuracy of single instance images from the original CIFAR test To avoid overfitting. we set the bags to be relatively many, with the minimum number of bags per class being 100.
FAMELYONIE 16 is a significant publicly available Whole Slide Image (WSI) dataset for lymph node classification and metastasis detection. It includes 270 training and 129 test slides from two medical centers, all meticulously annotated by pathologists. The TCGA Lung Cancer dataset comprises two non-small cell lung cancer subtypes, LUAD and LUSC, with 1053 slides, including 512 LUSC and 541 LUAD. 10 lualquality LUAD slides are discarded. Unlike CAMELYONI6, lacks patch level annotations and independent test set.
In our experiment, we followed a standard evaluation scheme coolddcv-sttnddllonettstt by DSMIL with shown in the logs in supplementary detail and codes on Github. Specifically, for the above two medical tasks, in each fold, the best model and corresponding threshold are saved. After the 5- fold cross-valides volsdvaald,,,nn, 5 best models are obtained which are used to perform inference the unseen test set. A final prediction for a test sample is the majority vote of the 5 models.
A.4 DERIVATION OF PROPOSITION
We provide the derivation of the Proposition 1, which explains how real-time instance predictions and bag predictions impact the optimization process of Φ.
Forward propagation Process: For a single bag with n instances, ÿ denotes its one-hot encoded label, Y denotes its ground-truth bag label(scalar), Y denotes any other label in [0, k except for
14

---

Published as a conference paper at ICLR 2025
Y, using the notations from the primary text:
d softmax
Hbug H d'
Zbag Cbby + b; (similar notation for instance i: CH, + b)
(13)
Pbag softmax (Zbag) (similar notation for instance i:p! softmax x
" log softmax Zhag) (Negative Log-Likelihood Loss)
have the following simplification process:
L y log softmax Zhag)
eapl exp((Zbag)
V log 1s eeeppp2a0 Zbagg
(14)
V (erg(erp( (zbag) log(1" exp((bagg))
7 Zbag + log(1 exp (bbag)
Where 1 denotes vector of ones, log represents the natural logarithm, softmax (7) and exp( 7) denotes the element-wise exponentiation. According to the the differentiation rules of matrix operations, element-wise functions @ and other related methodologies. We have:
exp(
expp ?)*
I (expppppz Zhag) @ di d(zbag))
dL 8 dābag +
1st (Zhag)
expp (Zbag) Tdl ((2bag)
J d2bag
+
1 12'erp(zzonl (2bgg)
(15)
JT džbay + softmax (Zbag) dZzag
= (softmax ZbbggT
- )ddbag
(phag ) dizbag.
In the aforementioned derivation process, number of simplifying techniques are employed, e.g., the differential of constant is zero. Then based on the differentiation rules we have,
dL. (Pbag J) + b)
(Pbag
V\ d(CHbag))
(pbag
(Pbag
(16)
VITCH' dd (pbag
(piag
(pbag J''H' Ja(HT))
(ping CH
Where J Raxn is the Jacobian matrix of d with respect H w. and follows the following computation rules:
a - az) Jij = 1aaaa
ifi j
(17)
if +
af T dž, where f repre-
Considering the relationship between derivatives and differentials df sents scalar function, and / is column vector. have,
15

---

Published as a conference paper at ICLR 2025
V = (Phag CH" JHJ 11" 1 HC (pbag 3) )
(18)
As consequence of the previous results, we have,
V J -= HT of ( Phag)
(19)
express J HCH - Phag)) as 3 E RR.. Thus, we have,
क L 3
B. H;
(20)
Is HCH ( -Pbaglli H
It is clearly that HCT (7) -Pbag) E RR, then we have,
- Σ HCT
( -Poug) Hi.
(21)
To further simplify the above negative gradient formula, we first expand (J²); based formula (14),
(JT). > Jii. Jusr Jus Jns]
0,11, 0:02, ..., a:(1 oi)...., [;an]
ay [a1, 02 ai - 1, ...,. an]
(22)
as [al, a2 ... Ci .... anl + [0, 0, C, .... 0]
a, / (0, O, .... C: C...... 0). (The only non-zero element index i a.)
Considering that the computational formulas for instance logits ž and bag-level logits Zhags we have,
16

---

Published as a conference paper at ICLR 2025
x (J") [HC"]( - -bbag) Hi
[( 0j d + [0,0. 0, a, a.......) O). peag) H,
ag d HCT 10.0, ar 0| Pbag) H
J HCT + +aH pbag) H;
I arl (2bag -by + t - BYT - Pbag) H,
(23)
a:l(b 6 Zbag)
+ (z) ( -bag) -Hi
ai(Z Zbag
=bngT) (t phog) Hi
a,(7)
1/ Pbog ( 2 Zbog ) H;
( =bag)
His where 3 = a;(7 Pbog
Considering that 7 is a one-hot encoded hard label and Φ. is a distribution, we can easily verify that
3 satisfies the constraint: Σj=1 Tj 0; Φr 0: 0; o E R
17


Reviews:
[{"review_id": "dLvVWm6mO7", "reviewer": "Reviewer_wt3B", "summary": "This work observes that the core issues leading to performance degradation in MIL with Attention-based methods stem from selection shift and feature shift. To address these problems, it introduces prototype learning and feature alignment.", "questions": "The authors need to supplement their work by addressing several issues mentioned in the aforementioned weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This work identifies the core issues leading to failures in Attention-based MIL approaches and proposes targeted solutions. It also provides clear theoretical proofs and achieves superior performance.", "weaknesses": "The article employs two core technologies to address the aforementioned issues; however, there are aspects of these technologies that require further clarification. Firstly, introducing prototypes to resolve selection bias is a technique that has been mentioned in \"Rethinking Multiple Instance Learning for Whole Slide Image Classification: A Good Instance Classifier is All You Need.\" Secondly, feature alignment appears to be a specific application of contrastive learning (either bringing enhanced image features closer or aligning probability spaces). If this process were placed before the overall model training (e.g., as a pre-training step), would it achieve similar effects?\nAdditionally, it is necessary to highlight the differences between this approach and traditional contrastive learning methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cD80aAKyjR", "reviewer": "Reviewer_DCds", "summary": "This paper addresses the issue of drift in instance representation learning within multiple instance learning (MIL) models by introducing Probability-Space Multiple Instance Learning (PSMIL). In PSMIL, the authors present two main strategies: probability-space attention pooling and a probability-space alignment objective. Experimental results show that the proposed approach improves performance on challenging MIL tasks, achieving results closer to supervised learning standards, while maintaining competitive performance on bag-level classification benchmarks.", "questions": "Please refer to Weaknesses section", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "+ The approach of addressing drift issue in instance representation learning through pseudo-label inference-based probability-space alignment appears novel to me.\n+ The authors have provided enough visualizations (e.g., Figure 2) along with the ablation study in Table 3 to justify the effectiveness of the proposed components PSAtt, and PSAli.\n+ The strong performance of existing techniques like DTFDMIL is well-explained in the context of the FMNIST dataset.", "weaknesses": "+ The authors should ensure all symbols used in the equations are clearly defined and explained. For example, in Equation 5, it is unclear how ${p^{^\\rightarrow}_s}(x^\\prime)$i s derived. Similarly, in Equation 2, the steps from layerwise differentiation to decomposition and recombination require further clarification.\n+ My main concern is the evaluation, as PS-MIL is tested only on synthetic datasets without including any challenging, real-world datasets. Furthermore,  in Table 2, PSMIL performs significantly lower on the FMNIST dataset compared to DTFDMIL, which requires further indepth justification.\n+ In the ablation study (Table 3), the standalone contribution of PSAtt, independent of PSAli, is unclear. Including results for PSAtt alone would help clarify its individual impact on overall performance.\n+ The impact of hyperparameters, such as $\\lambda$ in Equation 9, is not explored. A detailed study on the effect of  $\\lambda$ would offer insights into the relative importance of $L_{\\text{bag}}$ and $L_{\\text{ins}}$ on performance.\n+ Video anomaly detection is a key application of the MIL approach [1, 2, 3], providing challenging, real-world datasets. The authors should demonstrate the performance of their method on these datasets in comparison with established baselines [1, 2, 3]. This would strengthen the paper, as only simpler synthetic datasets are currently considered, which do not fully represent real-world MIL challenges.\n+ The process of pseudo-label inference in Equation 5 is unclear. Providing additional explanation beyond the equations would aid readers in understanding how pseudo-labels are inferred.\n\n**References**\n1. Wu et al. \"VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detection\". AAAI2024. \n2. Tian et al. “Weakly-supervised Video Anomaly Detection with Robust Temporal Feature Magnitude Learning”. ICCV2021\n3. Sultani et al. “Real-world Anomaly Detection in Surveillance Videos”. CVPR2018", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lccRw4flNk", "reviewer": "Reviewer_v2T9", "summary": "The author contends that deep MIL models suffer from a drift in the representation of instances during training. This drift, which occurs in the feature space, negatively impacts the model's performance. To address this issue, the paper proposes the Probability-Space MIL network. This network introduces probability-space attention pooling and a probability-space alignment objective. By transforming instances from the feature space to the probability space using class prototypes, the network mitigates the effects of feature drift and consequently enhances the overall performance of the MIL model.", "questions": "1- Use another evaluation method, such as localization on WSI datasets, and compare it with the previous approaches. \n\n2-Illustrate the key differences between the proposed approach, DSMIL [1] and DTFDMIL [2].\n\n3-The approach uses additional probability space with the class prototype. Thus, it is worth comparing the proposed approach's scalability and complexity against the previous approaches. \n\n4-We encourage the authors to do more ablation studies for the approach and see how that will impact the performance of the proposed approach. For example, manipulate the hyper-parameter in the equation (11).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1-\tThe problem is described clearly.\n\n2-\tThe investigated problem is important.", "weaknesses": "1-\tLimited downstream evaluation. \n\n2-\tThe PSMIL approach seems similar to other approaches that use additional attention pooling, such as DSMIL [1] and DTFDMIL [2].\n\n3-\t The paper does not discuss the computational complexity and scalability of the proposed approach in detail, which could be a concern for large-scale applications.\n\n4-\tLimited Ablation Studies: While the paper includes some ablation studies, more extensive ablations could strengthen the claims.\n\n[1] Bin Li, Yin Li, and KevinWEliceiri. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14318–14328, 2021.\n\n[2] Hongrun Zhang, Yanda Meng, Yitian Zhao, Yihong Qiao, Xiaoyun Yang, Sarah E Coupland, and Yalin Zheng. Dtfd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18802–18812, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)