SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To explain the indicators, we still need to start from a classical MIL binary-classification scenario. Given the instance representations  $\mathbf{f_s} = [\mathbf{f}_1,\mathbf{f}_2,\dots ,\mathbf{f}_t]\in \mathbb{R}^{c\times t}$ , the aggregator  $\rho$ , the classifier with linear transformation  $\mathbf{A}$  and the sigmoid function  $\mathrm{sign}(\ldots)$  as the activate function, with the total  $\mathbf{N}$  bags with the bag label  $\mathbf{Y}\in [0,1]$  in the dataset. Suppose it be currently in the training phase, the training phase is equivalent to a process that we try to find the solution to the system of  $\mathbf{N}$  equations, each with the following formula:
$$
\operatorname {s i g m} (\mathbf {A} \rho (\mathbf {f s})) = \mathbf {Y}; \mathbf {Y} \in \{0, 1 \} \tag {14}
$$
Here, we assume that aggregator  $\rho$  can be represented in a special formula:
$$
\rho (\mathbf {f s}) = \sum_ {k = 1} ^ {t} \alpha_ {k} \mathbf {f} _ {k}; \text {s . t .} \sum_ {k = 1} ^ {t} \alpha_ {k} = 1; \alpha_ {k} \geq 0 \tag {15}
$$
Then, we can transform the Equation 14 into the following formula:
$$
Z = \sum_ {k = 1} ^ {t} \alpha_ {k} \cdot \left(\mathbf {A} \mathbf {f} _ {k}\right) = \left\{ \begin{array}{l l} + \infty ; & \text {i f t h e g i v e n b a g l a b e l i s 1} \\ - \infty ; & \text {o t h e r w i s e} \end{array} \right. \tag {16}
$$
The detailed explanation of this transform are given in Section A of our supplementary. Then, we consider the training process when we apply the gradient-descent algorithm here. Suppose the corresponding weight  $\alpha_{k}$  of the  $k$ th instance in a positive bag be larger than others', the absolute value of  $\frac{\partial Z}{\partial(\mathbf{A}\mathbf{f}_k)}$  would be larger than others'. Here we think the instance produces a large gradient. We recognize that  $\mathbf{A}\mathbf{f}_k$  is the classification score of the corresponding  $k$ th instance. What's more, if its instance label is consistent with the bag label, we call it a critical instance. Given  $\alpha_{k}$  greater than 0, the gradient-descent algorithm would just push the parameters  $\mathbf{A}\mathbf{f}_k$  to the right direction to produce a correct instance-level prediction. The overall gradient of the parameters in the network will be the sum of different gradients produced by different instances in this bag. In conclusion, if an instance in a bag has a large corresponding weight  $\alpha$  and it is a critical instance, then Equation 16 would be
Aggregators	10/1	16/1	32/1	48/1	64/1	512/1	1/1(Full)
MXP	93.47	88.71	81.17	73.02	68.11	65.37
ABP	92.18	89.69	90.84	86.59	89.47	42.21
G-ABP	91.47	89.13	87.47	84.38	86.93	36.99	97.75
DSP	58.54	58.12	58.47	56.13	52.43	42.23
RGP	98.25	97.55	96.37	95.12	94.50	87.28
Table 2: Instance-level Average Test Accuracies for the first 30 Training Epochs on MMNIST. The first row contains the different modes (M/N), and 1/1(Full) expresses the fully supervised case.
Figure 2: Detailed metric fluctuations of Table 2 under different modes (M/N).
likely to produce a right gradient-descent direction to achieve good performance on instance-level prediction, and that is what we want. If the gradient-descent direction of the equation is beneficial for the instance-level prediction, we call it a right equation, otherwise an erroneous equation. The error rate  $\phi$  is the percentage of erroneous equations to the total number  $N$  of equations.  $\phi$  describes the degree of deviation during the training phase. When  $\phi > 0$ , gradient-descent accumulates errors in the wrong direction. In practice, we approximate  $\phi$  by following steps: In a positive bag, we calculate the values of  $\alpha$  for critical instances and take the average as  $\beta 1$ , and the average value of  $\alpha$  for non-critical instances is denoted as  $\beta 2$ . If  $\beta 1$  is smaller than  $\beta 2$ , we consider this to be an erroneous equation. We approximate  $\phi$  by taking the ratio of the number of erroneous equations and the total number of positive bags, and  $\phi$  is evaluated once for every training epoch. For a fully supervised model, it is easy to imagine that its  $\phi$  is 0 from the beginning of train phase, since every equation would produce a right gradient-descent direction. We argue that with a good aggregator  $\rho$ ,  $\phi$  is supposed to be close enough to 0, to be similar enough to a fully supervised model.
Another indicator is the distance  $\gamma$ . Still imagine what human will do when facing the MIL problem. To predict whether the bag is positive, we usually make decisions only based on the several critical instances in this bag. A well-designed  $\rho$  should simulate this characteristic.  $\gamma$  is a metric that measures the difference in weights between critical instances in a positive bag, which describes how similar it is to human decision-making. In practice, we calculate  $\gamma$  as follows: For every positive bag, we calculate the difference between the maximum and minimum weights  $\alpha$  of the corresponding critical instances, and take the average of these differences for all positive bags, denoted as  $\gamma$ .  $\gamma$  is also evaluated once for every training epoch. For the max pooling, it is easy to imagine that its distance  $\gamma$  could reach 1, but we do not want the distance to be large. We argue that with a good aggregator  $\rho$ ,  $\gamma$  is supposed to be close enough to 0, to be similar enough to how we make decisions.
Notice that  $\phi$  and  $\gamma$  are introduced to describe the training process for demonstration, thus all evaluated on the training set. Instance labels are known just when we evaluate  $\phi$  and  $\gamma$ .
4.4 Instance-Level Evaluation on MMNIST: Why RGP Works?
When an experiment with the training bag size (M) and the testing bag size (N) is implemented, we define it as the M/N mode. We give comprehensive experiments in different modes to test the aggregators in Table 2. For some detail fluctuations see Fig. 2. Results show that RGP shows dominance in different scenarios. As shown in Fig. 2 (d) and (e) under mode 512/1, with the faster convergence of  $\phi$  to 0 and the closer value of  $\gamma$  to 0, RGP exhibits a working logic that is sufficiently similar to human thinking process and a supervised algorithm. For RGP, it represents the consistency between instances and bags through specific structure, so that the learning process can be easily transferred to instance level. Attention-based thoughts are valuable, but simply applying more complicated attention modules to analyze the problem, as done in ABP series, may not lead to breakthrough results on instance level because indicators show that they struggle to catch the points. Another critical fact we found is that if we switch training bag size to 1, which would bring a fully
Branch	0	1	3	2	0	2	2	1	1	3	Ŷ
3	0.0151	0.0351	0.4981	0.1183	0.0220	0.0357	0.0365	0.0601	0.0313	0.1497	1
2	0.0132	0.0577	0.0841	0.3062	0.0185	0.1959	0.1977	0.0299	0.069	0.0360	1
1	0.0375	0.1889	0.0879	0.0479	0.0401	0.0793	0.0889	0.2001	0.1732	0.0563	1
Table 3: RGP Information Table (64/10) on MMNIST. The first row contains the 10 images of the test bag. Indicator vector  $\hat{\mathbf{Y}}$  is the model output. The remaining contents are all the weights of corresponding instances from all the branches.
supervised scenario, the performance is lower than RGP in 10/1 mode. This clearly tells that RGP is not a performance bottleneck, it would enhance the model when the bag size is relatively small. With the guidance of regressor, RGP describes the MIL problem accurately, which makes it similar enough to a supervised case to produce discriminative instance-level representations from backbone. We can also say that, when the number of data records (can be in form of bags or instances) is given fixed, MIL model with RGP could even outperform a fully supervised one under some condition. That verifies our Assumption 1.
We also provide a special experiment under mode 64/10, with the RGP information table presented in Table 3. We see that not only we get a correct indicator vector  $\hat{\mathbf{Y}}$  on bag level, we can also directly provide instance-level predictions by combining weight-related information with the model output  $\hat{\mathbf{Y}}$ : try to take the maximum value in each column.
RGP strictly follows the expectation we gave it, which is a strong constraint inspired by the inference rule. For a positive bag, according to the way RGP works, when we are in the training process, the  $k$ th instance representation with a high classification score inside the bag MUST have a larger value of weight  $\alpha_{k}$ . Referring to Equation 16 that explains  $\phi$  and  $\gamma$ , the absolute value of its partial derivative  $\frac{\partial Z}{\partial(\mathbf{A}\mathbf{f}_{k})}$  during the training process would be larger. The gradient descent algorithm will increase the classification score, which promotes the increase of its weight  $\alpha_{k}$  because it explicitly depends on the classification score in RGP. The gradient descent process produces instance-level representations with large weights and high classification scores in positive bags. For negative bags, the training process will lower the classification scores of all instance representations inside it, but since there are no positive instances in it, the weights of the instance representations would be random. The gradient descent process will uniformly decrease the classification scores of all instances inside the negative bag. In the end of training process, RGMIL will converge to a solution where all the training bags are predicted correctly. In this final state, there is a huge difference in instance weights, and instance representations in positive bags with high classification scores will have significantly larger weights. This final solution we get from RGMIL would comply with our inference rule above. In a MIL problem, any solution that meets our inference rule is legal.
Since we have verified that the final convergence state of RGMIL is a legal solution to the MIL problem, the convergence process implies an improvement in the model's ability to distinguish critical instances. But still, the speed of convergence is crucial to improve instance-level performance. We must accelerate the speed at which the model generates discriminative instance-level representations as much as possible, because when the loss value reaches or approaches 0, the model wouldn't be trained anymore, and too slow a speed can stop the model from reaching the final solution. With low  $\gamma$  value, RGP is able to capture more critical instances, which brings a quicker improvement in instance-level discriminative ability. Based on this, RGMIL could correct previous erroneous inferences to reduce the  $\phi$  value quickly and thus avoid the accumulation of errors on instance level. That leads to a huge improvement on instance-level performance when compared with other methods. After three epochs of training, RGMIL successfully lowered the  $\phi$ -value to 0 in the fourth epoch and maintained it till the end. Multiple experiments have shown that RGMIL can always adjust its inference to lower the  $\phi$ -value to 0 within the first seven training epochs in 512/1 mode.
4.5 Ablation Study of RGP
As shown above, we have provided the derivation of the feasibility of RGP using sigmoid as the activation function, which we refer to as the original version. The derivation is built on the case where there is no trouble in numerical computation or gradient descent. However, in the practical implementation, we still need to explore a more practical way. Here, we still refer to the formula of the RGP forward propagation process in the  $i$ th branch  $(\mathbf{B}^{\mathrm{i}})$  mentioned in previous section. In
this implement, we take the difference by using Equation 6, meanwhile using Equation 7 to achieve the normalization for preventing gradient vanishing. Without the normalization equation, if we take the value of the second bit of the softmax output instead of the difference, it would be similar to the original version, which we refer to as the baseline version. The experimental results for different versions are shown in Table 4. Obviously, it is seen that the performance based the different and the normalization is superior to others. Without the training tricks from the equations, it is difficult to reduce  $\gamma$ -value to a low level.
Aggregators	10/1	32/1	64/1	512/1
RGP (B)	94.60	82.64	62.75	66.54
RGP (B + N)	95.31	93.92	90.88	72.70
RGP (B + N + D)	98.25	96.37	94.50	87.28
4.6 Comparison with Supervised Models in Pain Estimation
In this part, pain estimation is considered as a real application to validate the performance of RGMIL, where the benchmark pain dataset (UNBC-McMaster Shoulder Pain dataset [17], UNBC) is used. The UNBC dataset contains 200 video sequences with 48398 frames from 25 subjects, and all frames are well-annotated. To compare with the existing supervised models, we follow the same experimental settings with the S-O-T-A model MSRAN (ICBBT'21) [2]: using 25-fold cross-validation with a leave-one-subject-out strategy and evaluating by four metrics. In experiment, all images are resized as (3, 224, 224), and the training bag size is set as 64. An average of about 6000 bags is used as training data during each fold validation, and ResNet18 [8] is used as backbone. Experimental results are shown in Table 5. From Table 5, it is seen that RGMIL does gain comparable performance to supervised methods. It indicates that current design is effective enough to be practical without causing any performance degradation compared to supervised ones for pain estimation. More details about metrics and settings are given in Section C of supplementary.
Table 4: Instance-level Average Test Accuracies for first 30 Training Epochs on MMNIST, where 'B' expresses Baseline Version, 'N' expresses Normalization Formula, 'D' expresses Difference Formula.
Methods	MAE ↓	MSE ↓	PCC ↑	ICC ↑
Deep Pain(2017)	0.50	0.74	0.78	0.45
DSHF(2018)	-	0.94	0.68	-
DBR(2018)	-	0.69	0.81	-
Multistream CNN(2019)	0.47	0.53	0.70	0.55
MSRAN*(2021)	0.40	0.46	0.78	0.63
LIAN(2021)	0.45	0.66	0.81	0.61
RGMIL	0.31	0.42	0.77	0.62
Table 5: Instance-level Performance Comparison with Supervised Models (*:S-O-T-A model) on UNBC pain dataset, where all results are the mean of 25-fold cross-validation and four metrics are used: MAE, MSE, PCC and ICC. In this experiment, we take consecutive 64 frames in a video as a training bag. To increase the amount of data, we used a sliding window approach with a step size of 8 to produce training bags. Statistics collected from Deep Pain [21], DSHF [25], DBR [23], Multistream CNN [10], MSRAN [2], LIAN [28].
4.7 Discussion for More General Scenarios and Limitations
To validate the performance of RGMIL on the general multi-class bag-level classification problems, we test our model on SIVAL [20] dataset, where SIVAL consists of 25 classes of complex objects photographed in different environments and each class contains 60 images. In this experiment, each image is segmented into approximately 30 segments, and each segment is represented by a 30-dimensional feature vector that encodes information. The segments are labeled as containing the object or the background. We separately select 1, 3 and 10 classes as positive classes, and randomly sample from the other classes as the negative class. We set up three Linear+ReLU blocks as feature extractors. The test scenario involves image classification problems ranging from binary to 11-class classification. Here, each image is treated as a bag. Considering that the classes of SIVAL do not have any ordering relationship, all bits of the output indicator vector are reliable, thus all branches are involved in the computation of loss function. Continuing with the current version's approach, the
model's output during test is still obtained by selecting the branch with the highest position among all branches with an output of 1. Table 6 shows the bag-level performance on SIVAL dataset [20]. From Table 6, it is seen that RGP still obtains competitive results on the general bag-level problem.
# of positive classes	1	3	10
MXP	94.87	89.08	60.95
ABP	97.05	93.93	75.71
G-ABP	97.31	94.28	81.33
DSP	96.79	89.60	75.91
RGP	97.31	94.40	80.19
Table 6: Bag-level Results on the general multi-class image classification problem (the SIVAL dataset), where 1, 3 and 10 express three different multi-classification problems in which 1, 3 and 10 classes are separately used as positive classes, respectively. The shown result is the average of 10-time test accuracy for each model after training convergence.
Moreover, we also validate RGMIL on the general multi-class instance-level classification problems. In this experiment, we still use the 3-branch RGMIL, and adopt a different approach in constructing the data: the images from classes (ranging from 0 to 6) in MMNIST are considered as the negative instances, like the background class, and the images from classes (7, 8 and 9) in MMNIST are as three different positive instances. Table 7 shows the instance-level performance of the constructed four classes classification on MMNIST. Unlike the original MMNIST series, the convergence speed of different methods varies much. Thus, we present the average of 10-time test accuracy for each model after convergence or training after 100 epochs. Each positive class and the negative class account for approximately  $25\%$  of the data. Within one positive bag, the number of images of positive class is approximately  $10\%$ . The processing of loss value and output is the same as the SIVAL experiment above. From Table 7, it is also seen that instance-level results are promising in a more general case.
Aggregators	10/1	16/1	32/1	64/1	256/1	512/1
MXP	76.09	71.78	57.49	58.15	38.63	32.48
ABP	71.76	75.17	71.81	68.33	69.09	62.43
G-ABP	73.12	75.04	72.21	64.58	69.20	60.56
DSP	62.81	69.29	63.75	67.04	59.77	52.91
RGP	88.79	85.73	85.02	83.19	82.50	81.46
Table 7: Instance-level Results on general 4-class problems constructed based MMNIST. The first row contains the different modes (M/N). The shown result is the average of 10-time test accuracy for each model after convergence or 100 epochs.
Although RGMIL is suitable to solve the general multi-classification problem in which class dependencies or ordering need not be considered, the feasibility of training on instance-level with an excessively large bag-length still need to be ensured. As shown in Table 1, it is seen that the instance-performance decays with increasing bag lengths. Obviously, when the bag length is 512, the accuracy is lowest and reduced by  $11\%$  than the minimum bag length. In addition, there is memory limitation to handle the excessively large bags when the feature extractor is involved in training.
In this paper, we propose a new vision on MIL with a practical multi-classification MIL model (RGMIL). RGMIL extends the classical binary to a multi-classification scenario via optimizing several reliable equations simultaneously. More importantly, a key component (RGP) is introduced in the MIL model. We argue that the difference between a supervised problem and MIL is deterministic and clear: we may not need an over-complicated black box to simulate the difference. An accurate description exists. We describe the characteristic of MIL problems through the structure of RGP, and only the parameters in the feature extractor (backbone) need to be trained to provide the discriminative instance-level representation. This is a method that has good interpretability and conforms to the true inference rule of the MIL problem. We also validate RGMIL in real-world application on the UNBC dataset. Suffice it to say that it is possible for a MIL model to achieve even better instance-level performance than some supervised models. We expect RGP to become an important component of future MIL network architectures and to provide some inspiration.
This work is supported in part by the State Key Program of National Natural Science of China under Grant 62234010, the National Natural Science Foundation of China under Grant 61806154, the China Postdoctoral Science Foundation Funded Project under Grant 2019M653565, and the Fundamental Research Funds for the Central Universities No.ZYTS23060.
[1] Stuart Andrews, Ioannis Tsochantaridis, and Thomas Hofmann. Support vector machines for multiple-instance learning. Advances in neural information processing systems, 15, 2002.
[2] Shaoxing Cui, Dong Huang, Yue Ni, and Xiaoyi Feng. Multi-scale regional attention networks for pain estimation. In 2021 13th International Conference on Bioinformatics and Biomedical Technology, pages 1-8, 2021.
[3] Thomas G Dietterich, Richard H Lathrop, and Tomás Lozano-Pérez. Solving the multiple instance problem with axis-parallel rectangles. Artificial intelligence, 89(1-2):31-71, 1997.
[4] Michael Gadermayr and Maximilian Tschuchnig. Multiple instance learning for digital pathology: A review on the state-of-the-art, limitations & future potential. arXiv preprint arXiv:2206.04425, 2022.
[5] Behnood Gholami, Wassim M Haddad, and Allen R Tannenbaum. Relevance vector machine learning for neonate pain intensity assessment using digital imaging. IEEE Transactions on biomedical engineering, 57(6):1457-1466, 2010.
[6] Teena Hassan, Dominik Seuß, Johannes Wollenberg, Katharina Weitz, Miriam Kunz, Stefan Lautenbacher, Jens-Uwe Garbas, and Ute Schmid. Automatic detection of pain from facial expressions: a survey. IEEE transactions on pattern analysis and machine intelligence, 43(6):1815-1831, 2019.
[7] Teena Hassan, Dominik Seuß, Johannes Wollenberg, Katharina Weitz, Miriam Kunz, Stefan Lautenbacher, Jens-Uwe Garbas, and Ute Schmid. Automatic detection of pain from facial expressions: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(6):1815-1831, 2021.
[8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[9] Lei He, L Rodney Long, Sameer Antani, and George R Thoma. Histology image analysis for carcinoma detection and grading. Computer methods and programs in biomedicine, 107(3):538-556, 2012.
[10] Dong Huang, Zhaoqiang Xia, Lei Li, Kunwei Wang, and Xiaoyi Feng. Pain-awareness multistream convolutional neural network for pain estimation. Journal of Electronic Imaging, 28(4):043008-043008, 2019.
[11] Shiluo Huang, Zheng Liu, Wei Jin, and Ying Mu. Bag dissimilarity regularized multi-instance learning. Pattern Recognition, 126:108583, 2022.
[12] Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. In International conference on machine learning, pages 2127-2136. PMLR, 2018.
[13] Ammar Mansoor Kamoona, Amirali Khodadadian Gostar, Alireza Bab-Hadiashar, and Reza Hoseinnezhad. Multiple instance-based video anomaly detection using deep temporal encoding-decoding. Expert Systems with Applications, 214:119079, 2023.
[14] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.
[31] Yong Zhang, Rui Zhao, Weiming Dong, Bao-Gang Hu, and Qiang Ji. Bilateral ordinal relevance multi-instance regression for facial action unit intensity estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7034-7043, 2018.
[32] Lu Zhao, Liming Yuan, Kun Hao, and Xianbin Wen. Generalized attention-based deep multi-instance learning. Multimedia Systems, 29(1):275-287, 2023.
[33] Zhi-Hua Zhou, Yu-Yin Sun, and Yu-Feng Li. Multi-instance learning by treating instances as non-iid samples. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 1249-1256. ACM, 2009.
[34] Zhen Zhu, Mengde Xu, Song Bai, Tengteng Huang, and Xiang Bai. Asymmetric non-local neural networks for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 593-602, 2019.