Title: RGMIL: Guide Your Multiple-Instance Learning Model with Regressor
Venue: NeurIPS 2023
Accepted: yes
Keywords: Video Analysis

Multiple-Instance Learning

Representation learning
Abstract: In video analysis, an important challenge is insufficient annotated data due to the rare occurrence of the critical patterns, and we need to provide discriminative frame-level representation with limited annotation in some applications. Multiple Instance Learning (MIL) is suitable for this scenario. However, many MIL models paid attention to analyzing the relationships between instance representations and aggregating them, but neglecting the critical information from the MIL problem itself, which causes difficultly achieving ideal instance-level performance compared with the supervised model.
To address this issue, we propose the $\textbf{\textit{Regressor-Guided MIL network} (RGMIL)}$, which effectively produces discriminative instance-level representations in a general multi-classification scenario. In the proposed method, we make full use of the $\textit{regressor}$ through our newly introduced $\textit{aggregator}$, $\textbf{\textit{Regressor-Guided Pooling} (RGP)}$. RGP focuses on simulating the correct inference process of humans while facing similar problems without introducing new parameters, and the MIL problem can be accurately described through the critical information from the $\textit{regressor}$ in our method. 
In experiments, RGP shows dominance on more than 20 MIL benchmark datasets, with the average bag-level classification accuracy close to 1. 
We also perform a series of comprehensive experiments on the MMNIST dataset. Experimental results illustrate that our $\textit{aggregator}$ outperforms existing methods under different challenging circumstances. Instance-level predictions are even possible under the guidance of RGP information table in a long sequence. RGMIL also presents comparable instance-level performance with S-O-T-A supervised models in complicated applications. Statistical results demonstrate the assumption that a MIL model can compete with a supervised model at the instance level, as long as a structure that accurately describes the MIL problem is provided. The codes are available on $\url{https://github.com/LMBDA-design/RGMIL}$.

RGMIL: Guide Your Multiple-Instance Learning Model with Regressor

Abstract

In video analysis, an important challenge is insufficient annotated data due to the rare occurrence of the critical patterns, and we need to provide discriminative frame-level representation with limited annotation in some applications. Multiple Instance Learning (MIL) is suitable for this scenario. However, many MIL models paid attention to analyzing the relationships between instance representations and aggregating them, but neglecting the critical information from the MIL problem itself, which causes difficultly achieving ideal instance-level performance compared with the supervised model. To address this issue, we propose the Regressor-Guided MIL network (RGMIL) , which effectively produces discriminative instance-level representations in a multi-classification scenario. In the proposed method, we make full use of the regressor through our newly introduced aggregator , RegressorGuided Pooling (RGP) . RGP focuses on simulating the correct inference process of humans while facing similar problems without introducing new parameters, and the MIL problem can be accurately described through the critical information from the regressor in our method. In experiments, RGP shows dominance on more than 20 MIL benchmark datasets, with the average bag-level classification accuracy close to 1. We also perform a series of comprehensive experiments on the MMNIST dataset. Experimental results illustrate that our aggregator outperforms existing methods under different challenging circumstances. Instance-level predictions are even possible under the guidance of RGP information table in a long sequence. RGMIL also presents comparable instance-level performance with S-O-T-A supervised models in complicated applications. Statistical results demonstrate the assumption that a MIL model can compete with a supervised model at the instance level, as long as a structure that accurately describes the MIL problem is provided. The codes are available on https://github.com/LMBDA-design/RGMIL .

1 Introduction

Multiple-instance learning (MIL) [3][33] is commonly used for the binary classification tasks in which instances lack individual labels and are grouped into bags. In MIL, a bag is labeled as positive if it contains at least one positive instance, otherwise, it is negative. Its goal is to learn the patterns that characterize positive bags. The models of MIL have been widely applied in various applications, such as anomaly detection [16, 13], pathology diagnosis [24, 4], etc.

At present, many deep MIL models adopt attention-based modules [12] to solve the MIL problems, which aggregate instance-level representations in various ways. Such approaches work during the stage of aggregating instance-level representations and perform well when the given instance-level representations are with good discriminability. However, in some applications, such as facial pain estimation, we face a scenario where there are no discriminative instance-level representations or the instance-level representations need to be explicitly learned. For them, simply learning the aggregation

process is far from enough when the instance-level performance is important. Most of the current methods fail to achieve satisfying instance-level performance in this case.

To address this issue, we propose a new model of multiple-instance learning based on the guidance of regressor, named as Regressor-Guided MIL network (RGMIL), which achieves promising instancelevel performance. Compared with existing methods, our contributions are listed as follows:

RGMIL can address the general multi-class classification problem in MIL scenarios by learning discriminative instance-level features. RGMIL accurately describes the MIL problem through the newly introduced aggregator, Regressor-Guided Pooling (RGP). With a accurate description of the MIL problem, we can fully transfer the learning process into the direct learning of instance-level representations, without the requirement for increasingly complicated aggregation processes.

RGMILdemonstrates outstanding performance and expected functionality on public datasets, showcasing the essential role of RGP. Indicators show that RGP performs well under different challenging circumstances, and instance-level predictions are even possible with the guidance of RGP information in a long sequence. Experimental results validate that the regressor guidance possibly brings similar instance-level performance to a supervised model.

Excepted the simulated data (MMNIST), the effectiveness of RGMIL is also validated on a real pain estimation dataset (UNBC). Specifically, from the perspective of pain estimation, RGMIL can be as effective as supervised learning models.

2 Related Works

2.1 The General Models of MIL

In the classical case of Multi-Instance Learning (MIL), a binary-classification problem is built and solved based on bags, and each bag is formed by several instances. In general, a bag is defined as X = [ x 1 , x 2 , . . . , x t ] , and its label is formulated by

Each bag includes t instances ( X i ∈ R D × t ) and one label ( Y i ∈ { 0 , 1 } ). The task of MIL is to recognize what type of instances-level representation in the bag X i could make the bag label Y i be one (positive). Classical MIL also assumes that there are neither ordering nor dependency of instances within a bag. There are two main architectures in the frameworks of MIL, shown as follows.

Architecture 1 : Instance-level Backbone + Regressor + Aggregator

In this architecture, the MIL model generally takes an instance-level backbone to obtain the instancelevel representations ( fs = [ f 1 , ..., f t ] ) which involves each instance in a bag ( X = [ x 1 , .., x t ] ), where f i is corresponding to x i , f i ∈ R c , i ∈ [1 , t ] . Then, the instance-level representations ( fs ) are fed into the regressor to predict the score or possibility ( p i ) of categories for each instance. Finally, the aggregator is used to aggregate the predicted scores ( P = [ p 1 , ..., p t ] ) and obtain the final bag-score p bag . Suppose ρ express the aggregation function, the aggregator is formulated as p bag = ρ ( P ) , and ρ should follow the permutation-invariant restriction :

where T expresses a arbitrary permutation matrix with appropriate dimension. In this architecture, the max pooling and average pooling are two most widely used aggregators. The max pooling (MXP) takes the highest value in P to produce p bag , while the average pooling takes the average value in P , and both of them comply with Equation ( 2 ).

Architecture 2. : Instance-level Backbone + Aggregator + Regressor

In this architecture, the MIL model produces the bag-level representation ( F ) directly via a backbone+aggregator structure. For example, in a image classification problem, a 2D-CNN backbone is used to get instance-level representation matrix fs ∈ R c × t and then aggregator ρ is applied to get the bag-level representation F = ρ ( fs ) . The function ρ still satisfy Equation ( 2 ). Then, each bag is predicted by the regressor based on the bag-level representation F ∈ R c . In this architecture, the most widely used aggregators are attention-based pooling (ABP) [12], gated attention-based pooling

(G-ABP) [12], and their variations [15][30][32]. Compared with Architecture 1 , Wang et al.[27] advocates Architecture 2 which produce the bag-level representation F , since they often perform better in terms of the bag-level classification.

2.2 Attention-based Pooling Series

At present, Attention-Based Pooling (ABP) and its variations are widely adopted in many mainstream methods of MIL. Generally, ABP is formulated as

where w ∈ R t and V ∈ R t × c are both parameters to be trained. f k ∈ R c is the k th instance representation in a bag. ABP provides the bag representation by weighted sum of representations of different instances. Moreover, many variations of ABP has been proposed, such as Dual-Stream MIL(2021) [15], DTFD-MIL(2022) [30], GAMIL(2023) [32]. Dual-Stream MIL [15] (DSP) used MXPand an MXP-guided ABP as two streams to aggregate representations and obtained the S-O-T-A level performance on MIL benchmarks. DTFD-MIL [30] divided a bag into several pseudo-bags when the bag size was really large and applied ABP on every pseudo-bag twice in a two-stage manner. They still follow the main idea of ABP: weights should rely on representations .

In many applications, such as the whole slide image based pathology diagnosis [9], there is no strong requirement for sufficiently discriminative instance-level representations. The model is provided with bags with the bag length t over large, and is asked to make accurate bag-level predictions. Most of the current models like ABP variations introduced above would only focus in learning of aggregation process or limit the analysis process between instances given the instance-level representations fixed.

2.3 Pain or No Pain

In some real-world applications, it is excepted to obtain the accurate instance-level prediction or representation, such as pain estimation. Most studies of pain estimation based on facial expression [6],[7],[18] rely on the Facial Action Coding System (FACS) [5], which is a rating system that distinguishes 44 facial movements, called Action Units (AU). Painful expressions are associated with some of these AUs. Earlier, Prkachin et al. [19] developed the Prkachin and Solomon Pain Intensity (PSPI) index based on these AUs, which is a 17-point scale reflecting the degree of pain, where 0 indicates no pain and 16 indicates extreme pain. Whereas, due to the imbalanced distribution of the PSPI index, researchers redefined the pain intensity based on the PSPI index. In general, the pain intensity are divided into four categories based PSPI: no pain (0), mild pain (1 to 2), severe pain (3 to 5), and extreme pain (the above 5). At present, there are some existing MIL algorithms, such as MI-DORF[22], MIR[31], but it is difficult for a pure MIL algorithm to provide ideal instance-level performance when compared with supervised deep models.

3 The Proposed Method

3.1 Assumptions

In this paper, we design the model (RGML) with the guidance of two assumptions shown as follows.

Assumption 1 It is realistic for a MIL model to achieve the performance similar to a supervised model at the instance level, as long as it has a well-designed architecture.

When facing MIL problems, we start from the Architecture 2 given in section 2.1. Compared to the fully supervised model, a new component aggregator is added. Here we would not focus on whether the data is in the form of bags or instances. From the perspective of bags, it is still fully supervised. But, it is simply another model with an additional component, and the problem to be solved is not even ambiguous . When a MIL problem is treated in this way, there is reason to make this assumption.

Assumption 2 Performance bottleneck of the current MIL model is the aggregator function ρ .

Input Video

predict

...    ...

...    ...

predict

predict

Backbone

Feature Extractor

RGP-N

RGP-2

Multiply &

Summation

α

1

F

α

2

α t

...  ...

predict

⑦

⑥

⑤

f

1

f

2

...  ...

f

t

⑧

⑨

Regressor

Arbitrary

Instance-level

Feature Extractor

To Be Trained

Normalization

Difference

Backbone

x

1

x

2

...  ...

x t

②

③

④

①

RGP

(a) Model Overview

(b) Procedures in the

𝑖

th Branch

Compared to the supervised learning, the available information obtained by MIL is limited. Therefore, the instance-level performance limit of MIL should be equivalent to the fully supervised mode. Two main architectures introduced above differ from the fully supervised ones only on the presence of the aggregator. We argue that the inability of MIL to achieve the performance of the fully supervised mode is due to the limitation of the learning stage of the new component (aggregator).

So what makes a good aggregator? We assume that if we could describe the key differences between a MIL problem and a supervised problem in ρ well enough, we could get an accurate MIL model. Recall what CNN does. Early in computer vision, representations were provided as fixed to train a pure regressor . CNN took a step forward by its accurate description , which makes it possible to train the backbone+regressor at the same time, and get even better performance. The main task of the new component ρ would also be to provide accurate description .

3.2 Introduction of RGMIL

RGMIL is proposed for a multi-classification scenario, as shown in Fig.1. It is achieved by N branches, each of which learns to distinguish one corresponding non-negative class (with non-zero label) . Thus N -branch RGMIL is suitable to solve an ( N +1 )-class classification problem. To solve the pain estimation problem, it requires only a sequence of frames and the highest pain intensity level ( 0 to N , discrete) of the sequence as input.

During each training step, a bag composed of t images is fed into the model, denoted as X = [ x 1 , x 2 , . . . , x t ] , X ∈ R D × t , with a bag label Y bag ∈ [0 , N ] which indicates the max instance label in the bag. We convert Y bag to a special vector Y , similar to but not the same as one-hot encoding:

Then, the backbone will produce the instance-level representations fs = [ f 1 , f 2 , . . . , f t ] ∈ R c × t corresponding to each image in this bag. The representations fs are fed into all the branches at the same time. For each branch B i , we have a corresponding linear regressor with parameters W i ∈ R c × 2 and b i ∈ R 2 ( i ∈ [1 , N ] ). Parameters in both the RGP and the regressor will be NOT involved in the gradient descent. To be clear, since our goal is to produce discriminative instance-level representations, only the backbone component (instance-level feature extractor) needs to be trained. For the RGP and regressor components, they are just a means of describing the MIL problem. They describe the MIL problem through specific structures, and as long as the structure is clear enough to provide an accurate description , it would be sufficient.

In the branch B i , given the instance-level representation matrix fs ∈ R c × t , the RGP works as follows:

Instance classification logits are provided in Equation 5. Difference values are then calculated to indicate the importance of instances in Equation 6. Normalization like Equation 7 is adopted to prevent the gradient vanishing in Equation 8. Use Equation 8 and 9 to get the final bag representation F i . In a MIL problem, we obey the inference rule that if there are positive instance-level patterns, the bag is positive . That is to say, there is consistency between the discrimination of critical instancelevel patterns and the bag representation. We argue that this is a key element to describe a MIL problem, and we use a shared regressor to express this consistency between them. In summary, the RGP asks for the current result of the regressor , and always tries to aggregate the instance representations based on the current judgments made by the regressor , then dynamically adjusts current judgments in the training process -- Much like what humans would do.

Return to the process of forward propagation. For the branch B i , given the bag representation F i ∈ R c , the corresponding regressor parameters will be used again to produce the prediction Z i ∈ R 2 , which will be used to calculate the loss L i in this branch( i ∈ [1 , N ] ) , formulated as follows:

The loss value L i on the corresponding branch B i is calculated as shown above. For a MIL problem, the bag label Y bag is the maximum instance label in the bag. Based on this, we give the vector Y that describes the presence of instances with different labels in this bag. Y i = 0 means no instance with label i presented in this bag, while Y i = 1 means presence. Since Y bag is the maximum instance label , some bits of Y would be reliable . For Y i where i ∈ [ Y bag , N ] , it gives the correct ground truth, while for Y i where i ∈ [ 1 , Y bag -1 ] it is not reliable. So for the branch B i where i ∈ [ 1 , Y bag -1 ] , it is meaningless to calculate the loss since it may contain false information. We set the loss on these branches to 0, as shown in Equation 11. For other reliable branches , we calculate the cross-entropy loss respectively. The total loss L is the sum of the loss values on all branches.

During test, for the branch B i , after we get the output Z i , we can get the indicator vector ˆ Y :

The branch B i finally produce a number of 0 or 1 based on Z i . When we get the number of 1 in the branch B i that indicates the presence of the corresponding label i , we say the model predicted that there are instances with the label i in the bag . Hence, the indicator vector ˆ Y describes whether the bag contains different labels. And at last,

When different branches output 1, we take the highest branch, because we need to know the maximum label in the bag as shown in Equation 13. Above equations produce the final result ˜ Y , ˜ Y ∈ [ 0 , N ] .

4 Experiments and Analyses

4.1 Bag-Level Performance: Evaluation on Benchmarks

With the branch number N = 1 , five classic MIL benchmark datasets [3][1] are used to evaluate the bag-level performance, and each provides the samples in form of bags, where the bag size t varies widely, ranging from 1 to 1044. Experimental results are shown in Table 1. It is a comprehensive test for the model under different conditions. From Table 1, it is obviously seen that our method (RGMIL) obtains the highest accuracies for five datasets, especially for FOX dataset. More benchmark performances are shown in Section B of our supplementary.

4.2 A Challenging Dataset for Demonstration

Since pain estimation is generally regarded as the classification problems with four classes, we need to set the branch N to 3. It indicates that we construct a 3-branch RGMIL to meet the upcoming challenges. Due to the extremely imbalance of label distribution, the pain data is not really a clear and flexible enough choice to present a demonstration. Thus, we constructed a flexible MIL dataset based on MNIST [14] and denoted it as MMNIST. In MMNIST, samples are presented in form of bags. A bag is composed of t grayscale images (1,28,28) selected from the MNIST dataset. With N = 3 , we take the images with label 0 to 3 to form MMNIST bags. The bag label is assigned based on the label with the largest count. The number of training bags for each bag label is approximately 1000. The labels in the bag are uniformly distributed. For the test, we use a fixed number of 10000 test images .

4.3 Evaluation Methods on the Aggregator ρ

We introduce the error rate ϕ , and the distance γ to describe the aggregator ρ to figure out how good the aggregator ρ describes a MIL problem.

To explain the indicators, we still need to start from a classical MIL binary-classification scenario. Given the instance representations fs = [ f 1 , f 2 , . . . , f t ] ∈ R c × t , the aggregator ρ , the classifier with linear transformation A and the sigmoid function sigm( . . . ) as the activate function, with the total N bags with the bag label Y ∈ [0 , 1] in the dataset. Suppose it be currently in the training phase, the training phase is equivalent to a process that we try to find the solution to the system of N equations, each with the following formula:

Here, we assume that aggregator ρ can be represented in a special formula:

Then, we can transform the Equation 14 into the following formula:

The detailed explanation of this transform are given in Section A of our supplementary. Then, we consider the training process when we apply the gradient-descent algorithm here. Suppose the corresponding weight α k of the k th instance in a positive bag be larger than others', the absolute value of ∂Z ∂ ( Af k ) would be larger than others' . Here we think the instance produces a large gradient. We recognize that Af k is the classification score of the corresponding k th instance. What's more, if its instance label is consistent with the bag label, we call it a critical instance . Given α k greater than 0, the gradient-descent algorithm would just push the parameters Af k to the right direction to produce a correct instance-level prediction . The overall gradient of the parameters in the network will be the sum of different gradients produced by different instances in this bag. In conclusion, if an instance in a bag has a large corresponding weight α and it is a critical instance , then Equation 16 would be

(a)

↑

(10/1)

(b)

↑

(64/1)

(c)

↑

(512/1)

(d)

φ ↓

(512/1)

(e)

γ ↓

(512/1)

MXP -*- ABP

G-ABP

·..- DSP ·=RGP

MXP =*- ABP

G-ABP

DSP =-=RGP

MXP -*= ABP

G-ABP

DSP =·= RGP

0.8

MXP

ABP

G-ABP

DSP

-=RGP

0.25

ABP

G-ABP

DSP

=RGP

Test Set Accwacy

8'0

Test Set Accuwacy

0.8

Test Set Aecwacy

0.8

0.6

0.2

0.6

0.6

0.6

0.15

0.4

0.4

0.4

0.4

0.1

0.2

0.2

0.2

0.2

0.05

10

Epochs

15

20

25

30

10

Epochs

15

20

25

30

10

Epochs

15

20

25

30

1

Epochs

8

10

0

4

Epochs

10

likely to produce a right gradient-descent direction to achieve good performance on instance-level prediction, and that is what we want. If the gradient-descent direction of the equation is beneficial for the instance-level prediction, we call it a right equation , otherwise an erroneous equation . The error rate ϕ is the percentage of erroneous equations to the total number N of equations. ϕ describes the degree of deviation during the training phase . When ϕ > 0 , gradient-descent accumulates errors in the wrong direction. In practice, we approximate ϕ by following steps: In a positive bag, we calculate the values of α for critical instances and take the average as β 1 , and the average value of α for non-critical instances is denoted as β 2 . If β 1 is smaller than β 2 , we consider this to be an erroneous equation . We approximate ϕ by taking the ratio of the number of erroneous equations and the total number of positive bags, and ϕ is evaluated once for every training epoch. For a fully supervised model, it is easy to imagine that its ϕ is 0 from the beginning of train phase, since every equation would produce a right gradient-descent direction. We argue that with a good aggregator ρ , ϕ is supposed to be close enough to 0, to be similar enough to a fully supervised model .

Another indicator is the distance γ . Still imagine what human will do when facing the MIL problem. To predict whether the bag is positive, we usually make decisions only based on the several critical instances in this bag. A well-designed ρ should simulate this characteristic. γ is a metric that measures the difference in weights between critical instances in a positive bag, which describes how similar it is to human decision-making. In practice,we calculate γ as follows: For every positive bag , we calculate the difference between the maximum and minimum weights α of the corresponding critical instances , and take the average of these differences for all positive bags, denoted as γ . γ is also evaluated once for every training epoch. For the max pooling, it is easy to imagine that its distance γ could reach 1, but we do not want the distance to be large. We argue that with a good aggregator ρ , γ is supposed to be close enough to 0, to be similar enough to how we make decisions.

Notice that ϕ and γ are introduced to describe the training process for demonstration, thus all evaluated on the training set . Instance labels are known just when we evaluate ϕ and γ .

4.4 Instance-Level Evaluation on MMNIST: Why RGP Works?

When an experiment with the training bag size ( M ) and the testing bag size ( N ) is implemented, we define it as the M / N mode. We give comprehensive experiments in different modes to test the aggregators in Table 2. For some detail fluctuations see Fig. 2. Results show that RGP shows dominance in different scenarios. As shown in Fig. 2 (d) and (e) under mode 512/1, with the faster convergence of ϕ to 0 and the closer value of γ to 0, RGP exhibits a working logic that is sufficiently similar to human thinking process and a supervised algorithm . For RGP, it represents the consistency between instances and bags through specific structure, so that the learning process can be easily transferred to instance level. Attention-based thoughts are valuable, but simply applying more complicated attention modules to analyze the problem, as done in ABP series, may not lead to breakthrough results on instance level because indicators show that they struggle to catch the points. Another critical fact we found is that if we switch training bag size to 1, which would bring a fully

supervised scenario, the performance is lower than RGP in 10/1 mode. This clearly tells that RGP is not a performance bottleneck, it would enhance the model when the bag size is relatively small . With the guidance of regressor, RGP describes the MIL problem accurately, which make it similar enough to a supervised case to produce discriminative instance-level representations from backbone. We can also say that, when the number of data records(can be in form of bags or instances) is given fixed, MIL model with RGP could even outperform a fully supervised one under some condition. That verifies our Assumption 1 .

We also provide a special experiment under mode 64/10, with the RGP information table presented in Table 3. We see that not only we get a correct indicator vector ˆ Y on bag level, we can also directly provide instance-level predictions by combining weight-related information with the model output ˆ Y : try to take the maximum value in each column.

RGP strictly follows the expectation we gave it, which is a strong constraint inspired by the inference rule. For a positive bag, according to the way RGP works, when we are in the training process, the k th instance representation with a high classification score inside the bag MUST have a larger value of weight α k . Referring to Equation 16 that explains ϕ and γ , the absolute value of its partial derivative ∂Z ∂ ( Af k ) during the training process would be larger. The gradient descent algorithm will increase the classification score, which promotes the increase of its weight α k because it explicitly depends on the classification score in RGP. The gradient descent process produces instance-level representations with large weights and high classification scores in positive bags. For negative bags, the training process will lower the classification scores of all instance representations inside it, but since there are no positive instances in it, the weights of the instance representations would be random. The gradient descent process will uniformly decrease the classification scores of all instances inside the negative bag. In the end of training process, RGMIL will converge to a solution where all the training bags are predicted correctly. In this final state, there is a huge difference in instance weights, and instance representations in positive bags with high classification scores will have significantly larger weights. This final solution we get from RGMIL would comply with our inference rule above. In a MIL problem, any solution that meets our inference rule is legal.

Since we have verified that the final convergence state of RGMIL is a legal solution to the MIL problem, the convergence process implies an improvement in the model's ability to distinguish critical instances. But still, the speed of convergence is crucial to improve instance-level performance. We must accelerate the speed at which the model generates discriminative instance-level representations as much as possible, because when the loss value reaches or approaches 0, the model wouldn't be trained anymore, and too slow a speed can stop the model from reaching the final solution. With low γ value, RGP is able to capture more critical instances, which brings a quicker improvement in instance-level discriminative ability. Based on this, RGMIL could correct previous erroneous inferences to reduce the ϕ value quickly and thus avoid the accumulation of errors on instance level. That leads to a huge improvement on instance-level performance when compared with other methods. After three epochs of training, RGMIL successfully lowered the ϕ -value to 0 in the fourth epoch and maintained it till the end. Multiple experiments have shown that RGMIL can always adjust its inference to lower the ϕ -value to 0 within the first seven training epochs in 512/1 mode.

4.5 Ablation Study of RGP

As shown above, we have provided the derivation of the feasibility of RGP using sigmoid as the activation function, which we refer to as the original version. The derivation is built on the case where there is no trouble in numerical computation or gradient descent. However, in the practical implementation, we still need to explore a more practical way. Here, we still refer to the formula of the RGP forward propagation process in the i th branch ( B i ) mentioned in previous section. In

this implement, we take the difference by using Equation 6, meanwhile using Equation 7 to achieve the normalization for preventing gradient vanishing. Without the normalization equation, if we take the value of the second bit of the softmax output instead of the difference, it would be similar to the original version, which we refer to as the baseline version. The experimental results for different versions are shown in Table 4. Obviously, it is seen that the performance based the different and the normalization is superior to others. Without the training tricks from the equations, it is difficult to reduce γ -value to a low level.

Table 4: Instance-level Average Test Accuracies for first 30 Training Epochs on MMNIST, where 'B' expresses Baseline Version, 'N' expresses Normalization Formula, 'D' expresses Difference Formula.

4.6 Comparison with Supervised Models in Pain Estimation

In this part, pain estimation is considered as a real application to validate the performance of RGMIL, where the benchmark pain dataset (UNBC-McMaster Shoulder Pain dataset [17], UNBC) is used. The UNBC dataset contains 200 video sequences with 48398 frames from 25 subjects, and all frames are well-annotated. To compare with the existing supervised models , we follow the same experimental settings with the S-O-T-A model MSRAN (ICBBT'21) [2]: using 25-fold cross-validation with a leave-one-subject-out strategy and evaluating by four metrics. In experiment, all images are resized as (3, 224, 224), and the training bag size is set as 64. An average of about 6000 bags is used as training data during each fold validation, and ResNet18 [8] is used as backbone. Experimental results are shown in Table 5. From Table 5, it is seen that RGMIL does gain comparable performance to supervised methods. It indicates that current design is effective enough to be practical without causing any performance degradation compared to supervised ones for pain estimation. More details about metrics and settings are given in Section C of supplementary.

4.7 Discussion for More General Scenarios and Limitations

To validate the performance of RGMIL on the general multi-class bag-level classification problems, we test our model on SIVAL [20] dataset, where SIVAL consists of 25 classes of complex objects photographed in different environments and each class contains 60 images. In this experiment, each image is segmented into approximately 30 segments, and each segment is represented by a 30-dimensional feature vector that encodes information. The segments are labeled as containing the object or the background. We separately select 1, 3 and 10 classes as positive classes, and randomly sample from the other classes as the negative class. We set up three Linear+ReLU blocks as feature extractors. The test scenario involves image classification problems ranging from binary to 11-class classification. Here, each image is treated as a bag. Considering that the classes of SIV AL do not have any ordering relationship, all bits of the output indicator vector are reliable, thus all branches are involved to the computation of loss function. Continuing with the current version's approach, the

model's output during test is still obtained by selecting the branch with the highest position among all branches with an output of 1. Table 6 shows the bag-level performance on SIVAL dataset [20]. From Table 6, it is seen that RGP still obtains competitive results on the general bag-level problem.

Moreover, we also validate RGMIL on the general multi-class instance-level classification problems. In this experiment, we still use the 3-branch RGMIL, and adopt a different approach in constructing the data: the images from classes (ranging from 0 to 6) in MMNIST are considered as the negative instances, like the background class, and the images from classes (7, 8 and 9) in MMNIST are as three different positive instances. Table 7 shows the instance-level performance of the constructed four classes classification on MMNIST. Unlike the original MMNIST series, the convergence speed of different methods varies much. Thus, we present the average of 10-time test accuracy for each model after convergence or training after 100 epochs. Each positive class and the negative class account for approximately 25 % of the data. Within one positive bag, the number of images of positive class is approximately 10 % . The processing of loss value and output is the same as the SIV AL experiment above. From Table 7, it is also seen that instance-level results are promising in a more general case.

Although RGMIL is suitable to solve the general multi-classification problem in which class dependencies or ordering need not be considered, the feasibility of training on instance-level with an excessively large bag-length still need to be ensured. As shown in Table 1, it is seen that the instance-performance decays with increasing bag lengths. Obviously, when the bag length is 512, the accuracy is lowest and reduced by 11% than the minimum bag length. In addition, there is memory limitation to handle the excessively large bags when the feature extractor is involved in training.

5 Conclusion

In this paper, we propose a new vision on MIL with a practical multi-classification MIL model (RGMIL). RGMIL extends the classical binary to a multi-classification scenario via optimizing several reliable equations simultaneously . More importantly, a key component (RGP) is introduced in the MIL model. We argue that the difference between a supervised problem and MIL is deterministic and clear : we may not need an over-complicated black box to simulate the difference. An accurate description exists. We describe the characteristic of MIL problems through the structure of RGP, and only the parameters in the feature extractor (backbone) need to be trained to provide the discriminative instance-level representation. This is a method that has good interpretability and conforms to the true inference rule of the MIL problem. We also validate RGMIL in real-world application on the UNBC dataset. Suffice it to say that it is possible for a MIL model to achieve even better instance-level performance than some supervised models. We expect RGP to become an important component of future MIL network architectures and to provide some inspiration.

Acknowledgments

This work is supported in part by the State Key Program of National Natural Science of China under Grant 62234010, the National Natural Science Foundation of China under Grant 61806154, the China Postdoctoral Science Foundation Funded Project under Grant 2019M653565, and the Fundamental Research Funds for the Central Universities No.ZYTS23060.

References

Stuart Andrews, Ioannis Tsochantaridis, and Thomas Hofmann. Support vector machines for multiple-instance learning. Advances in neural information processing systems , 15, 2002.

Shaoxing Cui, Dong Huang, Yue Ni, and Xiaoyi Feng. Multi-scale regional attention networks for pain estimation. In 2021 13th International Conference on Bioinformatics and Biomedical Technology , pages 1-8, 2021.

Thomas G Dietterich, Richard H Lathrop, and Tomás Lozano-Pérez. Solving the multiple instance problem with axis-parallel rectangles. Artificial intelligence , 89(1-2):31-71, 1997.

Michael Gadermayr and Maximilian Tschuchnig. Multiple instance learning for digital pathology: A review on the state-of-the-art, limitations & future potential. arXiv preprint arXiv:2206.04425 , 2022.

Behnood Gholami, Wassim M Haddad, and Allen R Tannenbaum. Relevance vector machine learning for neonate pain intensity assessment using digital imaging. IEEE Transactions on biomedical engineering , 57(6):1457-1466, 2010.

Teena Hassan, Dominik Seuß, Johannes Wollenberg, Katharina Weitz, Miriam Kunz, Stefan Lautenbacher, Jens-Uwe Garbas, and Ute Schmid. Automatic detection of pain from facial expressions: a survey. IEEE transactions on pattern analysis and machine intelligence , 43(6):1815-1831, 2019.

Teena Hassan, Dominik Seuß, Johannes Wollenberg, Katharina Weitz, Miriam Kunz, Stefan Lautenbacher, Jens-Uwe Garbas, and Ute Schmid. Automatic detection of pain from facial expressions: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43(6):1815-1831, 2021.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Lei He, L Rodney Long, Sameer Antani, and George R Thoma. Histology image analysis for carcinoma detection and grading. Computer methods and programs in biomedicine , 107(3):538556, 2012.

Dong Huang, Zhaoqiang Xia, Lei Li, Kunwei Wang, and Xiaoyi Feng. Pain-awareness multistream convolutional neural network for pain estimation. Journal of Electronic Imaging , 28(4):043008-043008, 2019.

Shiluo Huang, Zheng Liu, Wei Jin, and Ying Mu. Bag dissimilarity regularized multi-instance learning. Pattern Recognition , 126:108583, 2022.

Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. In International conference on machine learning , pages 2127-2136. PMLR, 2018.

Ammar Mansoor Kamoona, Amirali Khodadadian Gostar, Alireza Bab-Hadiashar, and Reza Hoseinnezhad. Multiple instance-based video anomaly detection using deep temporal encodingdecoding. Expert Systems with Applications , 214:119079, 2023.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 14318-14328, 2021.

Yang Liu, Jing Liu, Xiaoguang Zhu, Donglai Wei, Xiaohong Huang, and Liang Song. Learning task-specific representation for video anomaly detection with spatial-temporal attention. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 2190-2194. IEEE, 2022.

Patrick Lucey, Jeffrey F. Cohn, Kenneth M. Prkachin, Patricia E. Solomon, Sien Chew, and Iain Matthews. Painful monitoring: Automatic pain monitoring using the unbc-mcmaster shoulder pain expression archive database. Image and Vision Computing , 30(3):197-205, 2012. Best of Automatic Face and Gesture Recognition 2011.

Shasha Mao, Guanghui Shi, Licheng Jiao, Shuiping Gou, Yangyang Li, Lin Xiong, and Boxin Shi. Label distribution amendment with emotional semantic correlations for facial expression recognition, 2021.

Kenneth M Prkachin and Patricia E Solomon. The structure, reliability and validity of pain expression: Evidence from patients with shoulder pain. Pain , 139(2):267-274, 2008.

Rouhollah Rahmani, Sally A Goldman, Hui Zhang, John Krettek, and Jason E Fritts. Localized content based image retrieval. In Proceedings of the 7th ACM SIGMM international workshop on Multimedia information retrieval , pages 227-236, 2005.

Pau Rodriguez, Guillem Cucurull, Jordi Gonzàlez, Josep M Gonfaus, Kamal Nasrollahi, Thomas B Moeslund, and F Xavier Roca. Deep pain: Exploiting long short-term memory networks for facial expression classification. IEEE transactions on cybernetics , 52(5):3314-3324, 2017.

Adria Ruiz, Ognjen Rudovic, Xavier Binefa, and Maja Pantic. Multi-instance dynamic ordinal random fields for weakly supervised facial behavior analysis. IEEE transactions on image processing , 27(8):3969-3982, 2018.

Mohammad Tavakolian and Abdenour Hadid. Deep binary representation of facial expressions: A novel framework for automatic pain intensity recognition. In 2018 25th IEEE International Conference on Image Processing (ICIP) , pages 1952-1956. IEEE, 2018.

Maximilian E Tschuchnig and Michael Gadermayr. Anomaly detection in medical imaging-a mini review. In Data Science-Analytics and Applications: Proceedings of the 4th International Data Science Conference-iDSC2021 , pages 33-38. Springer, 2022.

Jinwei Wang and Huazhi Sun. Pain intensity estimation using deep spatiotemporal and handcrafted features. IEICE TRANSACTIONS on Information and Systems , 101(6):1572-1580, 2018.

Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7794-7803, 2018.

Xinggang Wang, Yongluan Yan, Peng Tang, Xiang Bai, and Wenyu Liu. Revisiting multiple instance neural networks. Pattern Recognition , 74:15-24, 2018.

Xuwu Xin, Xiaowu Li, Shengfu Yang, Xiaoyan Lin, and Xin Zheng. Pain expression assessment based on a locality and identity aware network. IET Image Processing , 15(12):2948-2958, 2021.

Yongluan Yan, Xinggang Wang, Xiaojie Guo, Jiemin Fang, Wenyu Liu, and Junzhou Huang. Deep multi-instance learning with dynamic pooling. In Asian Conference on Machine Learning , pages 662-677. PMLR, 2018.

Hongrun Zhang, Yanda Meng, Yitian Zhao, Yihong Qiao, Xiaoyun Yang, Sarah E Coupland, and Yalin Zheng. Dtfd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18802-18812, 2022.

Yong Zhang, Rui Zhao, Weiming Dong, Bao-Gang Hu, and Qiang Ji. Bilateral ordinal relevance multi-instance regression for facial action unit intensity estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7034-7043, 2018.

Lu Zhao, Liming Yuan, Kun Hao, and Xianbin Wen. Generalized attention-based deep multiinstance learning. Multimedia Systems , 29(1):275-287, 2023.

Zhi-Hua Zhou, Yu-Yin Sun, and Yu-Feng Li. Multi-instance learning by treating instances as non-iid samples. In Proceedings of the 26th Annual International Conference on Machine Learning , pages 1249-1256. ACM, 2009.

Zhen Zhu, Mengde Xu, Song Bai, Tengteng Huang, and Xiang Bai. Asymmetric non-local neural networks for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 593-602, 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new approach for Multiple Instance Learning (MIL) called Regressor-Guided MIL (RGMIL). The contribution of RGMIL is a new aggregation approach Regressor-Guided Pooling (RGP). In this approach, the aggregation part of the MIL model is split into N branches (for N + 1 classes, assuming class 0 is a negative class), where each branch is responsible for classifying a single class. The model is applied to a range of binary MIL problems (MUSK1, MUSK2, FOX, TIGER, ELEPHANT, Webpages, 20NewsGroups, Messidor, and UCSB Breast), and two max-based multi-class problems: MMNIST (proposed in this work) and UNBC shoulder pain estimation. The model mostly outperforms existing methods, and analysis is conducted as to why RGP is an effective pooling strategy.\n\nSTRENGTHS:\n**Originality**\n1. The RGB pooling is an original approach, particularly with the double pass through the regressor.\n2. Section 4.3 is also a novel investigation into the bottlenecks of MIL model training and can help understand why certain models succeed/fail.\n\n**Quality**\n1. The evaluation covers a range of datasets and shows strong results, outperforming most other methods in most cases.\n2. An ablation study is used to show how performance changes when training on different sizes bags but only classifying single instance bags.\n\n**Significance**\n1. Given the strong performance of RGMIL, it would appear useful in MIL settings that use max-based problems, i.e., traditional binary MIL or problems like pain estimation.\n\nWEAKNESSES:\n**Originality**\n1. I believe some elements of the proposed method are already well known concepts, e.g., equation 4 appears to be one hot encoding, and equation 8 is a softmax. Simplifying these in the methodology and focusing on the novel parts of the method will help understanding. \n\n**Quality**\n1. Some elements of the approach are lacking justification (see questions below).\n2. The MMNIST dataset is proposed but then only evaluated in an Instance-level setting. A strong evaluation could involve looking at the performance when the training and test bag sizes are the same (i.e., 10/10, 16/16 etc.) as well as the instance-level setting.\n\n**Clarity**\n1. I found the paper very hard to follow and it took a considerable effort to unpick what the proposed method was actually doing.\n2. It is not clear quite how the method works, particularly with regards to training. In Section 3.2, I am puzzled by the statement *\"Parameters in both the RGP and regressor will NOT be involved in gradient descent\"*. Please see questions below.\n2. The work mentions how CNNs work and how humans would approach a MIL problem as justification for their approach. This makes the narrative harder to follow and detracts from the design of the novel pooling mechanism. I think this comparison would be best placed in the appendix, with more space allocated to discussing the model architecture and explaining how it works.\n3. Figure and Table captions often contain little information, e.g., Table 2 does not state which dataset the model is trained on nor what the different columns means, Figure 2 should have y-axis labels, etc.\n4. Some of the mathematical notation should be revisited to aid understanding. For example, is it necessary to have $M$ as the total number of instances and then $t$ as the number of instances per bag? Equations 5 through 9 jump through different notations, e.g., $fs$ for representations, then $H$, then $F_i$.\n5. I would suggest another thorough proof-read to remove grammatical errors that make the work hard to follow.\n\n**Significance**\n1. My understanding is that this approach is only applicable to multi-class problems that have some max-based ordering of classes, e.g., in pain estimation, if branch 1 and 2 are both positive, then the prediction is class 2. I don't know how this would work for general multi-class problems (see questions below), which limits its significance.\n2. I disagree with the final paragraph of Section 2.2 (comparing t < c and t > c). Instance-level performance is important for interpretability in both cases - just because there are a greater number of instances per bag does not mean instance-level performance becomes less important.\n\nQUESTIONS:\n1. Why does each branch of the aggregator produce two outputs? I can see the two outputs are used to calculate a difference in Equation 6, but don't understand why a single output with a sigmoid activation and a threshold of 0.5 isn't used. This needs to be explained in more detail.\n2. If my understanding is correct, the regressor is used to calculate some form of attention weights (output of equation 8) and also classify the re-weighted instance representations. Is that correct? If so, what is the relationship to other MIL attention methods? It seems somewhat similar to the Additive MIL approach [1], which is not cited.\n3. Is each branch regressor assigned random weights that are then fixed during training of the feature extractor? If so, what is the motivation behind this?\n4. How would this approach work on multi-class problems that don't have some form of max-based ranking, e.g., SIVAL [2] or 4-MNIST-Bags [3]? My confusion lies in what happens when two or more branches all have positive indicator vectors $\\hat{Y}$. In the max-based datasets used in this work, the maximum branch is taken as the overall prediction (equation 13). However, I'm not sure how this would work for other types of multi-class MIL datasets that don't use max-based classes. This limits the significance and scalability of the work.\n\n[1] Javed, Syed Ashar, et al. \"Additive MIL: intrinsically interpretable multiple instance learning for pathology.\" Advances in Neural Information Processing Systems 35 (2022): 20689-20702.  \n[2] Rahmani, Rouhollah, et al. \"Localized content based image retrieval.\" Proceedings of the 7th ACM SIGMM international workshop on Multimedia information retrieval. 2005.  \n[3] Early, Joseph, Christine Evers, and Sarvapali Ramchurn. \"Model Agnostic Interpretability for Multiple Instance Learning.\" In International Conference on Learning Representations. 2022.\n\nLIMITATIONS:\n1. The work does not address the limitations of the novel method or suggest any areas for future work.\n2. The main limitation of the work is that it is very hard to follow and difficult to understand how the novel method works. This is frustrating as the results appear promising.\n3. One potential limitation, discussed in the questions above, is how this could be applied to multi-class datasets that do not follow some form of max-based assumption.\n\nDue to the difficult in understanding the approach and the lack of clarity in the method, I am leaning towards rejection as I don't think the method is reproducible from what is presented. However, there is certainly promise in the approach and the results appear strong.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tries to overcome the drawback of low instance-level prediction performance  in  the existing Multiple Instance Learning (MIL) models. According to the authors, the low instance-level performance is because the existing techniques focus mostly on analyzing the relationship between instances and aggregating them while ignoring learning of effective instance-level representation. To this end, this paper proposes a novel aggregator function called Regressor-Guided Pooling (RGP) that directly learns discriminative instance-level representation by mimicking human inference process. An extensive evaluation is performed using the multiple MIL datasets along with the pain estimation datasets to validate the effectiveness of the proposed aggregator function.\n\nSTRENGTHS:\n1. This paper has made a novel contribution to MIL field by identifying and addressing  key limitations of the proposed MIL techniques. The significance of the novel contribution is demonstrated through non-trivial performance gain  over other competitive baselines.\n2. The motivation for introducing the Regressor-Guided Multiple Instance Learning (RGMIL) framework is very novel, natural, and intuitive.\n3. This paper deals with the multi-classification scenario under the MIL setting which is unique and different from most of the MIL works that focus mostly on the binary-classification scenario. This unique scenario also helped to enhance the novelty of this paper.\n4. The extensive evaluation is conducted considering multiple MIL datasets. In addition to the quantitative result, the authors have done great job explaining why the proosed technique works and how it avoids the limitations of the existing MIL techniques.\n\nWEAKNESSES:\n1. Most of the real-world video anomaly datasets are also used as a crucial testbed for the evaluation of MIL models [1, 2, 3].  Specifically, under video anomaly detection task, the MIL model is trained with video-level annotations (without having explicit access to the frame-level annotation). During the prediction process, the trained MIL model is used to perform frame-level prediction. The evaluation of the proposed MIL model on video-anomaly detection task is missing. I wonder how effective the proposed technique is in terms of making frame-level predictions. Also, in the related work section, the authors may need to explain how MIL techniques aimed to solve the video anomaly detection task are different from their work. \n2. Some of the figures deserve better treatment. For example, the caption in Figure 1 is not very descriptive. Further,  modules in the figures (with texts) are difficult to read and understand. \n3. The intuition behind Equation 6 is difficult to understand and is not very clear. \n\n\nReferences:\n[1]. Sultani et al. “Real-world Anomaly Detection in Surveillance Videos”. CVPR2018.\n[2]. Sapkota et al. “Distributionally Robust Optimization for Deep Kernel Multiple Instance Learning”. AISTATS2021.\n[3]. Tian et al. “Weakly-Supervised Video Anomaly Detection With Robust Temporal Feature Magnitude Learning”. ICCV2021.\n\nQUESTIONS:\n1. How effective the proposed technique is in the video-anomaly detection tasks compared to competitive baselines?\n\nLIMITATIONS:\nSee weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper presents a new model, the Regressor-Guided Multiple Instance Learning network (RGMIL), which addresses the challenge of providing discriminative instance-level representation in multi-classification MIL scenarios. It introduces an aggregator, the Regressor-Guided Pooling (RGP), that enhances the instance-level performance of Multiple Instance Learning (MIL) tasks. RGMIL outperforms existing methods in various datasets and demonstrates the potential for instance-level predictions.

STRENGTHS:
1.  The paper presents a novel model, the Regressor-Guided Multiple Instance Learning network (RGMIL), which brings a fresh perspective to the field of multi-classification MIL scenarios. 
2. The paper is well-structured and the problem and proposed solution are explained clearly. 
3. The experiments are well-designed and the promising results effectively support the authors' claims.
4. The paper is inspiring for future research, including potential applications for multi-classification tasks such as the UCF-Crime dataset.

WEAKNESSES:
1. The paper lacks a clear definition and purpose for the metrics used in Table 4. Additionally, a thorough analysis of the results presented in the table is missing.

QUESTIONS:
See the weaknesses.

LIMITATIONS:
See the weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces _Regressor-guided MIL network (RGMIL)_ as a solution to address the challenges in Multiple instance learning, particularly in the multi-class classification scenario for pain-estimation. RGMIL introduces a novel aggregator, Regressor-Guided Pooling (RGP) in place of currently widely used attention-based aggregators. The design of RGP is based on simulating the correct inference process of humans when facing similar problems. The experiments show RGP out-performing evaluation benchmarks in MIL and works similar to supervised models for pain estimation.\n\nSTRENGTHS:\n- The aggregator introduced in this work is grounded in prior knowledge of Multiple Instance Learning task construction. As a result, rather than opting for a complex black box approach, the design remains simplistic.This work aligns with recent research trends that emphasize leveraging simple yet valuable prior knowledge and inductive biases to design more effective solutions.\n- Regressor-Guided Pooling (RGP), outperforms all MIL evaluation benchmarks including SOTA by a large margin (Table 1) and also works on-par with supervised models for pain estimation (Table 4). This compelling result clearly highlights the potential of the introduced method.\n- The code available has a well-written README and looks easy to reproduce.\n\nWEAKNESSES:\n- **Missing dataset details**: The details of dataset(s) construction for MIL in pain-estimation setting is missing. One of the contributions mentioned in this work is RGMIL being the first weakly supervised deep model for pain estimation, so I feel it is important for this work to also focus on pain estimation datasets and benchmarks.\n    - Firstly, it is not clearly mentioned whether the results presented in section 4.5 is for UNBC pain dataset. The way the dataset is constructed for the MIL setting is also not presented which makes it difficult to understand if the dataset is different or the same as supervised setting. \n    - Secondly, the result is presented on only a single dataset. Given that the contributions of the work also focuses on being the first weakly supervised model for pain estimation, it is important to benchmark on more datasets (for example BioVid dataset presented in [25]).\n\n\n- **Issues in paper structure and writing:** The paper lacks a cohesive flow, making it difficult to follow. The writing style is inconsistent both within the sections and throughout the entire document. This paper needs major restructuring and rewriting to be easy to follow and understand.\n    - The main paper contains many non-essential details and lacks conciseness. Few suggestions improve it: the assumptions section (3.1) should be condensed into smaller paragraphs, with more extensive details provided in the supplementary material. Similarly, sections 4.2, 4.3, and 4.4 are verbose and could benefit by retaining crucial details in the main paper and moving the remaining content to the supplementary.\n    - In its current stage, some important contributions are left behind in supplementary (A1, A2) which supports the method’s effectiveness.\n    - Figure and table captions (except Table 1) are not self-sufficient and needs referencing to the text for understanding. Authors should revise the captions for better stand-alone clarity.\n    - Figure 1(b) is not clearly visible or understandable.\n\n\n- **Missing important citations and explanations:** There are important statements without citations or explanations, based on which the paper makes assumptions for model design (lines 143-144) and presents results on a different dataset (lines 230-231).\n    - line 143-144: 'Experience has shown that when we add the aggregator ⇢ to the backbone + regressor model for MIL problems, the instance-level performance often decreases significantly.' I would suggest the authors to cite works that show that adding aggregator decreases instance-level performance. If it is by observation across paper, the authors should present this in a table or a figure to give strong basis for assumptions.\n    - line 230-231: 'Considering multiple factors, the UNBC pain dataset isn’t really a clear and flexible enough choice to present a demonstration.' What are the multiple factors here is not clear and I would suggest the authors to explain clearly.\n\n\n\n- **Little to no focus on second contribution:** The paper's focus on the first contribution is appreciated, but it leaves little room to highlight the second contribution, leading to confusion in understanding the relevance of pain estimation in this work. It was referenced multiple times but only given a limited space (10 lines: 326-336) in the overall paper. To improve clarity, the authors can choose to implement either of these two suggestions:\n    - Prioritize RGP as the main contribution and dedicate the paper to thoroughly highlighting its significance, while briefly mentioning pain estimation as a valuable benefit of the method only shortly. This also relieves the authors of having to benchmark any other datasets while fully highlighting the method and its effectiveness.\n    - Condense some verbose sections, to be more concise, and move extra details to supplementary. This ensures that the authors have more space to present the benchmark results on pain estimation.\n\nQUESTIONS:\n- My main suggestions for the authors is to address the writing in the paper and to highlight their contributions accordingly. The results looks very compelling, but the paper is quite difficult to follow to appreciate the design or importance of the method. \n- Other minor changes include: captions for standalone clarity, dataset details for reproducibility and clear understanding.\n\nLIMITATIONS:\nThe authors have not mentioned any possible limitations or potential negative societal impact of this work. I would suggest the authors to include some details in settings where this method may not be useful.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}