SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhaolong Du<sup>1</sup>, Shasha Mao<sup>1*</sup>, Yimeng Zhang<sup>1</sup>, Shuiping Gou<sup>1</sup>, Licheng Jiao<sup>1</sup>, Lin Xiong<sup>2</sup>  
<sup>1</sup>Xidian University, <sup>2</sup>SenseTime.
In video analysis, an important challenge is insufficient annotated data due to the rare occurrence of the critical patterns, and we need to provide discriminative frame-level representation with limited annotation in some applications. Multiple Instance Learning (MIL) is suitable for this scenario. However, many MIL models paid attention to analyzing the relationships between instance representations and aggregating them, but neglecting the critical information from the MIL problem itself, which causes difficulty achieving ideal instance-level performance compared with the supervised model. To address this issue, we propose the Regressor-Guided MIL network (RGMIL), which effectively produces discriminative instance-level representations in a multi-classification scenario. In the proposed method, we make full use of the regressor through our newly introduced aggregator, Regressor-Guided Pooling (RGP). RGP focuses on simulating the correct inference process of humans while facing similar problems without introducing new parameters, and the MIL problem can be accurately described through the critical information from the regressor in our method. In experiments, RGP shows dominance on more than 20 MIL benchmark datasets, with the average bag-level classification accuracy close to 1. We also perform a series of comprehensive experiments on the MMNIST dataset. Experimental results illustrate that our aggregator outperforms existing methods under different challenging circumstances. Instance-level predictions are even possible under the guidance of RGP information table in a long sequence. RGMIL also presents comparable instance-level performance with S-O-T-A supervised models in complicated applications. Statistical results demonstrate the assumption that a MIL model can compete with a supervised model at the instance level, as long as a structure that accurately describes the MIL problem is provided. The codes are available on https://github.com/LMBDA-design/RGMIL.
Multiple-instance learning (MIL) [3][33] is commonly used for the binary classification tasks in which instances lack individual labels and are grouped into bags. In MIL, a bag is labeled as positive if it contains at least one positive instance, otherwise, it is negative. Its goal is to learn the patterns that characterize positive bags. The models of MIL have been widely applied in various applications, such as anomaly detection [16, 13], pathology diagnosis [24, 4], etc.
At present, many deep MIL models adopt attention-based modules [12] to solve the MIL problems, which aggregate instance-level representations in various ways. Such approaches work during the stage of aggregating instance-level representations and perform well when the given instance-level representations are with good discriminability. However, in some applications, such as facial pain estimation, we face a scenario where there are no discriminative instance-level representations or the instance-level representations need to be explicitly learned. For them, simply learning the aggregation
*Corresponding author. email:ssmao@xidian.edu.cn
process is far from enough when the instance-level performance is important. Most of the current methods fail to achieve satisfying instance-level performance in this case.
To address this issue, we propose a new model of multiple-instance learning based on the guidance of regressor, named as Regressor-Guided MIL network (RGMIL), which achieves promising instance-level performance. Compared with existing methods, our contributions are listed as follows:
- RGMIL can address the general multi-class classification problem in MIL scenarios by learning discriminative instance-level features. RGMIL accurately describes the MIL problem through the newly introduced aggregator, Regressor-Guided Pooling (RGP). With a accurate description of the MIL problem, we can fully transfer the learning process into the direct learning of instance-level representations, without the requirement for increasingly complicated aggregation processes.
- RGMIL demonstrates outstanding performance and expected functionality on public datasets, showcasing the essential role of RGP. Indicators show that RGP performs well under different challenging circumstances, and instance-level predictions are even possible with the guidance of RGP information in a long sequence. Experimental results validate that the regressor guidance possibly brings similar instance-level performance to a supervised model.
- Excepted the simulated data (MMNIST), the effectiveness of RGMIL is also validated on a real pain estimation dataset (UNBC). Specifically, from the perspective of pain estimation, RGMIL can be as effective as supervised learning models.
2.1 The General Models of MIL
In the classical case of Multi-Instance Learning (MIL), a binary-classification problem is built and solved based on bags, and each bag is formed by several instances. In general, a bag is defined as  $\mathbf{X} = [\mathbf{x}_1,\mathbf{x}_2,\dots ,\mathbf{x}_t]$ , and its label is formulated by
$$
\mathbf {Y} = \max  _ {k} \left\{\mathbf {y} _ {k} \right\}, k \in [ 1, t ]. \tag {1}
$$
Each bag includes  $t$  instances  $(\mathbf{X}_i \in \mathbb{R}^{D \times t})$  and one label  $(\mathbf{Y}_i \in \{0,1\})$ . The task of MIL is to recognize what type of instances-level representation in the bag  $\mathbf{X}_i$  could make the bag label  $\mathbf{Y}_i$  be one (positive). Classical MIL also assumes that there are neither ordering nor dependency of instances within a bag. There are two main architectures in the frameworks of MIL, shown as follows.
Architecture 1: Instance-level Backbone + Regressor + Aggregator
In this architecture, the MIL model generally takes an instance-level backbone to obtain the instance-level representations  $(\mathbf{fs} = [\mathbf{f}_1, \dots, \mathbf{f}_t])$  which involves each instance in a bag  $(\mathbf{X} = [\mathbf{x}_1, \dots, \mathbf{x}_t])$ , where  $\mathbf{f}_i$  is corresponding to  $\mathbf{x}_i$ ,  $\mathbf{f}_i \in \mathbb{R}^c$ ,  $i \in [1, t]$ . Then, the instance-level representations (fs) are fed into the regressor to predict the score or possibility  $(\mathbf{p}_i)$  of categories for each instance. Finally, the aggregator is used to aggregate the predicted scores  $(\mathbf{P} = [\mathbf{p}_1, \dots, \mathbf{p}_t])$  and obtain the final bag-score  $\mathbf{p}_{\mathrm{bag}}$ . Suppose  $\rho$  express the aggregation function, the aggregator is formulated as  $\mathbf{p}_{\mathrm{bag}} = \rho(\mathbf{P})$ , and  $\rho$  should follow the permutation-invariant restriction:
$$
\rho (\mathbf {P}) = \rho (\mathbf {P T}), \tag {2}
$$
where  $\mathbf{T}$  expresses a arbitrary permutation matrix with appropriate dimension. In this architecture, the max pooling and average pooling are two most widely used aggregators. The max pooling (MXP) takes the highest value in  $\mathbf{P}$  to produce  $\mathbf{p}_{\mathrm{bag}}$ , while the average pooling takes the average value in  $\mathbf{P}$ , and both of them comply with Equation (2).
Architecture 2.: Instance-level Backbone + Aggregator + Regressor
In this architecture, the MIL model produces the bag-level representation  $(\mathbf{F})$  directly via a backbone+aggregator structure. For example, in a image classification problem, a 2D-CNN backbone is used to get instance-level representation matrix  $\mathbf{fs} \in \mathbb{R}^{c \times t}$  and then aggregator  $\rho$  is applied to get the bag-level representation  $\mathbf{F} = \rho(\mathbf{fs})$ . The function  $\rho$  still satisfy Equation (2). Then, each bag is predicted by the regressor based on the bag-level representation  $\mathbf{F} \in \mathbb{R}^c$ . In this architecture, the most widely used aggregators are attention-based pooling (ABP) [12], gated attention-based pooling
$(G - ABP)$  [12], and their variations [15][30][32]. Compared with Architecture 1, Wang et al.[27] advocates Architecture 2 which produce the bag-level representation  $\mathbf{F}$ , since they often perform better in terms of the bag-level classification.
2.2 Attention-based Pooling Series
At present, Attention-Based Pooling (ABP) and its variations are widely adopted in many mainstream methods of MIL. Generally, ABP is formulated as
$$
\mathbf {F} = \sum_ {k = 1} ^ {t} a _ {k} \mathbf {f} _ {k}, \quad a _ {k} = \frac {\exp \left\{\mathbf {w} ^ {\top} \tanh  \left(\mathbf {V} \mathbf {f} _ {k} ^ {\top}\right) \right\}}{\sum_ {j = 1} ^ {t} \exp \left\{\mathbf {w} ^ {\top} \tanh  \left(\mathbf {V} \mathbf {f} _ {j} ^ {\top}\right) \right\}} \tag {3}
$$
where  $\mathbf{w} \in \mathbb{R}^t$  and  $\mathbf{V} \in \mathbb{R}^{t \times c}$  are both parameters to be trained.  $\mathbf{f}_k \in \mathbb{R}^c$  is the  $k$ th instance representation in a bag. ABP provides the bag representation by weighted sum of representations of different instances. Moreover, many variations of ABP has been proposed, such as Dual-Stream MIL(2021) [15], DTFD-MIL(2022) [30], GAMIL(2023) [32]. Dual-Stream MIL [15] (DSP) used MXP and an MXP-guided ABP as two streams to aggregate representations and obtained the S-O-T-A level performance on MIL benchmarks. DTFD-MIL [30] divided a bag into several pseudo-bags when the bag size was really large and applied ABP on every pseudo-bag twice in a two-stage manner. They still follow the main idea of ABP: weights should rely on representations.
In many applications, such as the whole slide image based pathology diagnosis [9], there is no strong requirement for sufficiently discriminative instance-level representations. The model is provided with bags with the bag length  $t$  over large, and is asked to make accurate bag-level predictions. Most of the current models like ABP variations introduced above would only focus in learning of aggregation process or limit the analysis process between instances given the instance-level representations fixed.
2.3 Pain or No Pain
In some real-world applications, it is expected to obtain the accurate instance-level prediction or representation, such as pain estimation. Most studies of pain estimation based on facial expression [6],[7],[18] rely on the Facial Action Coding System (FACS) [5], which is a rating system that distinguishes 44 facial movements, called Action Units (AU). Painful expressions are associated with some of these AUs. Earlier, Prkachin et al. [19] developed the Prkachin and Solomon Pain Intensity (PSPI) index based on these AUs, which is a 17-point scale reflecting the degree of pain, where 0 indicates no pain and 16 indicates extreme pain. Whereas, due to the imbalanced distribution of the PSPI index, researchers redefined the pain intensity based on the PSPI index. In general, the pain intensity are divided into four categories based PSPI: no pain (0), mild pain (1 to 2), severe pain (3 to 5), and extreme pain (the above 5). At present, there are some existing MIL algorithms, such as MI-DORF[22], MIR[31], but it is difficult for a pure MIL algorithm to provide ideal instance-level performance when compared with supervised deep models.
In this paper, we design the model (RGML) with the guidance of two assumptions shown as follows.
Assumption 1 It is realistic for a MIL model to achieve the performance similar to a supervised model at the instance level, as long as it has a well-designed architecture.
When facing MIL problems, we start from the Architecture 2 given in section 2.1. Compared to the fully supervised model, a new component aggregator is added. Here we would not focus on whether the data is in the form of bags or instances. From the perspective of bags, it is still fully supervised. But, it is simply another model with an additional component, and the problem to be solved is not even ambiguous. When a MIL problem is treated in this way, there is reason to make this assumption.
Assumption 2 Performance bottleneck of the current MIL model is the aggregator function  $\rho$ .
Figure 1: Overview of Regressor-Guided multiple-instance Learning Network (RGMIL)
Compared to the supervised learning, the available information obtained by MIL is limited. Therefore, the instance-level performance limit of MIL should be equivalent to the fully supervised mode. Two main architectures introduced above differ from the fully supervised ones only on the presence of the aggregator. We argue that the inability of MIL to achieve the performance of the fully supervised mode is due to the limitation of the learning stage of the new component (aggregator).
So what makes a good aggregator? We assume that if we could describe the key differences between a MIL problem and a supervised problem in  $\rho$  well enough, we could get an accurate MIL model. Recall what CNN does. Early in computer vision, representations were provided as fixed to train a pure regressor. CNN took a step forward by its accurate description, which makes it possible to train the backbone+regressor at the same time, and get even better performance. The main task of the new component  $\rho$  would also be to provide accurate description.
3.2 Introduction of RGMIL
RGMIL is proposed for a multi-classification scenario, as shown in Fig.1. It is achieved by  $N$  branches, each of which learns to distinguish one corresponding non-negative class (with non-zero label). Thus  $N$ -branch RGMIL is suitable to solve an  $(N + 1)$ -class classification problem. To solve the pain estimation problem, it requires only a sequence of frames and the highest pain intensity level (0 to  $N$ , discrete) of the sequence as input.
During each training step, a bag composed of  $t$  images is fed into the model, denoted as  $\mathbf{X} = [\mathbf{x}_1,\mathbf{x}_2,\dots ,\mathbf{x}_t],\mathbf{X}\in \mathbb{R}^{D\times t}$ , with a bag label  $\mathbf{Y}_{\mathrm{bag}}\in [0,N]$  which indicates the max instance label in the bag. We convert  $\mathbf{Y}_{\mathrm{bag}}$  to a special vector  $\mathbf{Y}$ , similar to but not the same as one-hot encoding:
$$
\mathbf {Y} \in \mathbb {R} ^ {N}; \mathbf {Y} _ {\mathbf {i}} = \left\{ \begin{array}{l l} 1, & \text {i f f} \mathbf {Y} _ {\text {b a g}} = \mathbf {i} \\ 0, & \text {o t h e r w i s e} \end{array} \right. (\mathbf {i} \in [ 1, N ]). \tag {4}
$$
Then, the backbone will produce the instance-level representations  $\mathbf{f}_{\mathrm{s}} = [\mathbf{f}_1,\mathbf{f}_2,\dots ,\mathbf{f}_t]\in \mathbb{R}^{c\times t}$  corresponding to each image in this bag. The representations fs are fed into all the branches at the same time. For each branch  $\mathbf{B}^{\mathrm{i}}$  we have a corresponding linear regressor with parameters  $\mathbf{W_i}\in \mathbb{R}^{c\times 2}$  and  $\mathbf{b_i}\in \mathbb{R}^2 (\mathbf{i}\in [1,N])$ . Parameters in both the RGP and the regressor will be NOT involved in the gradient descent. To be clear, since our goal is to produce discriminative instance-level representations, only the backbone component (instance-level feature extractor) needs to be trained. For the RGP and regressor components, they are just a means of describing the MIL problem. They describe the MIL problem through specific structures, and as long as the structure is clear enough to provide an accurate description, it would be sufficient.
In the branch  $\mathbf{B}^{\mathrm{i}}$ , given the instance-level representation matrix  $\mathbf{f}\mathbf{s}\in \mathbb{R}^{c\times t}$ , the RGP works as follows:
$$
\mathbf {H} = \mathbf {f s} ^ {\top}; \mathbf {P} _ {k} = \mathbf {W} _ {\mathbf {i}} ^ {\top} \mathbf {H} _ {k} + \mathbf {b} _ {\mathbf {i}}; (k \in [ 1, t ], \mathbf {P} \in \mathbb {R} ^ {t \times 2}) \tag {5}
$$
$$
\widehat {P} = \mathbf {P} ^ {\top}; \mathbf {p} = \widehat {P} _ {2} - \widehat {P} _ {1}; (\mathbf {p} \in \mathbb {R} ^ {t}) \tag {6}
$$
$$
\mathbf {w} = \frac {\mathbf {p} - E [ \mathbf {p} ]}{\sqrt {\operatorname {V a r} (\mathbf {p}) + \epsilon}}; \left(\mathbf {w} \in \mathbb {R} ^ {t}, \epsilon \rightarrow 0\right) \tag {7}
$$
$$
\vec {\alpha} \in \mathbb {R} ^ {t}; \vec {\alpha} _ {k} = \frac {\exp (\mathbf {w} _ {k})}{\sum_ {j = 1} ^ {t} \exp (\mathbf {w} _ {j})}; (\vec {\alpha} _ {k} \in (0, 1)) \tag {8}
$$
$$
\mathbf {F} _ {\mathbf {i}} = \sum_ {k = 1} ^ {t} \vec {\alpha} _ {k} \mathbf {H} _ {k}; \left(\mathbf {F} _ {\mathbf {i}} \in \mathbb {R} ^ {c}\right) \tag {9}
$$
Instance classification logits are provided in Equation 5. Difference values are then calculated to indicate the importance of instances in Equation 6. Normalization like Equation 7 is adopted to prevent the gradient vanishing in Equation 8. Use Equation 8 and 9 to get the final bag representation  $\mathbf{F_i}$ . In a MIL problem, we obey the inference rule that if there are positive instance-level patterns, the bag is positive. That is to say, there is consistency between the discrimination of critical instance-level patterns and the bag representation. We argue that this is a key element to describe a MIL problem, and we use a shared regressor to express this consistency between them. In summary, the RGP asks for the current result of the regressor, and always tries to aggregate the instance representations based on the current judgments made by the regressor, then dynamically adjusts current judgments in the training process —— Much like what humans would do.
Return to the process of forward propagation. For the branch  $\mathbf{B}^{\mathrm{i}}$ , given the bag representation  $\mathbf{F}_{\mathrm{i}} \in \mathbb{R}^{c}$ , the corresponding regressor parameters will be used again to produce the prediction  $Z^{\mathrm{i}} \in \mathbb{R}^{2}$ , which will be used to calculate the loss  $\mathbf{L}^{\mathrm{i}}$  in this branch ( $\mathrm{i} \in [1, N]$ ), formulated as follows:
$$
Z ^ {\mathbf {i}} = \mathbf {W} _ {\mathbf {i}} ^ {\top} \mathbf {F} _ {\mathbf {i}} + \mathbf {b} _ {\mathbf {i}}; (Z ^ {\mathbf {i}} \in \mathbb {R} ^ {2}) \tag {10}
$$
$$
\mathbf {L} = \sum_ {\mathrm {i} = 1} ^ {\mathrm {N}} \mathbf {L} ^ {\mathrm {i}}; \text {w h e r e} \mathbf {L} ^ {\mathrm {i}} = \left\{ \begin{array}{l l} - \log \left\{\frac {\exp \left(Z _ {1 +} ^ {\mathrm {i}} \mathbf {Y} _ {\mathrm {i}}\right)}{\exp \left(Z _ {1} ^ {\mathrm {i}}\right) + \exp \left(Z _ {2} ^ {\mathrm {i}}\right)} \right\} & \text {i f f i} \geqslant \mathbf {Y} _ {\text {b a g}} \\ 0 & \text {o t h e r w i s e} \end{array} \right. \tag {11}
$$
The loss value  $\mathbf{L}^{\mathrm{i}}$  on the corresponding branch  $\mathbf{B}^{\mathrm{i}}$  is calculated as shown above. For a MIL problem, the bag label  $\mathbf{Y}_{\mathrm{bag}}$  is the maximum instance label in the bag. Based on this, we give the vector  $\mathbf{Y}$  that describes the presence of instances with different labels in this bag.  $\mathbf{Y}_{\mathrm{i}} = 0$  means no instance with label i presented in this bag, while  $\mathbf{Y}_{\mathrm{i}} = 1$  means presence. Since  $\mathbf{Y}_{\mathrm{bag}}$  is the maximum instance label , some bits of  $\mathbf{Y}$  would be reliable. For  $\mathbf{Y}_{\mathrm{i}}$  where  $\mathbf{i}\in [\mathbf{Y}_{\mathrm{bag}},\mathbf{N}]$  , it gives the correct ground truth, while for  $\mathbf{Y}_{\mathrm{i}}$  where  $i\in [1,\mathbf{Y}_{\mathrm{bag}} - 1]$  it is not reliable. So for the branch  $\mathbf{B}^{\mathrm{i}}$  where  $i\in [1,\mathbf{Y}_{\mathrm{bag}} - 1]$  , it is meaningless to calculate the loss since it may contain false information. We set the loss on these branches to 0, as shown in Equation 11. For other reliable branches, we calculate the cross-entropy loss respectively. The total loss  $\mathbf{L}$  is the sum of the loss values on all branches.
During test, for the branch  $\mathbf{B}^{\mathrm{i}}$ , after we get the output  $Z^{\mathrm{i}}$ , we can get the indicator vector  $\hat{\mathbf{Y}}$ :
$$
\hat {\mathbf {Y}} \in \mathbb {R} ^ {N}; \quad \hat {\mathbf {Y}} _ {\mathbf {i}} = \underset {\mathbf {k}} {\operatorname {a r g m a x}} \left\{Z _ {\mathbf {k}} ^ {\mathbf {i}} \right\} - 1; (\mathbf {k} \in \{1, 2 \}) \tag {12}
$$
The branch  $\mathbf{B}^{\mathrm{i}}$  finally produce a number of 0 or 1 based on  $Z^{\mathrm{i}}$ . When we get the number of 1 in the branch  $\mathbf{B}^{\mathrm{i}}$  that indicates the presence of the corresponding label  $\mathbf{i}$ , we say the model predicted that there are instances with the label  $\mathbf{i}$  in the bag. Hence, the indicator vector  $\hat{\mathbf{Y}}$  describes whether the bag contains different labels. And at last,
$$
\tilde {\mathbf {Y}} = \underset {\mathbf {k}} {\operatorname {a r g m a x}} \left\{\hat {\mathbf {Y}} _ {\mathbf {k}} \right\} \tag {13}
$$
$$
\mathrm {s . t .} \hat {\mathbf {Y}} _ {\mathbf {k}} = 1; \mathbf {k} \in [ 1, N ]
$$
When different branches output 1, we take the highest branch, because we need to know the maximum label in the bag as shown in Equation 13. Above equations produce the final result  $\widetilde{\mathbf{Y}},\widetilde{\mathbf{Y}}\in [0,\mathbf{N}]$
4 Experiments and Analyses
4.1 Bag-Level Performance: Evaluation on Benchmarks
With the branch number  $N = 1$ , five classic MIL benchmark datasets [3][1] are used to evaluate the bag-level performance, and each provides the samples in form of bags, where the bag size  $t$  varies widely, ranging from 1 to 1044. Experimental results are shown in Table 1. It is a comprehensive test for the model under different conditions. From Table 1, it is obviously seen that our method (RGMIL) obtains the highest accuracies for five datasets, especially for FOX dataset. More benchmark performances are shown in Section B of our supplementary.
Methods	MUSK1	MUSK2	FOX	TIGER	ELEPHANT
MI-Net(2018)	0.887 ± 0.041	0.859 ± 0.046	0.622 ± 0.038	0.830 ± 0.032	0.862 ± 0.034
ABMIL(2018)	0.892 ± 0.040	0.858 ± 0.048	0.615 ± 0.043	0.839 ± 0.022	0.868 ± 0.022
Gated-ABMIL(2018)	0.900 ± 0.050	0.863 ± 0.042	0.603 ± 0.029	0.845 ± 0.018	0.857 ± 0.027
DP-MINN(2018)	0.907 ± 0.036	0.926 ± 0.043	0.655 ± 0.052	0.897 ± 0.028	0.894 ± 0.030
Non-Local(2018)	0.921 ± 0.017	0.910 ± 0.009	0.703 ± 0.035	0.857 ± 0.013	0.876 ± 0.011
Asymmetric Non-Local(2019)	0.912 ± 0.009	0.822 ± 0.084	0.643 ± 0.012	0.733 ± 0.068	0.883 ± 0.014
Dual-Stream MIL(2021)*	0.932 ± 0.023	0.930 ± 0.020	0.729 ± 0.018	0.869 ± 0.008	0.925 ± 0.007
BDR(2022)	0.926 ± 0.079	0.905 ± 0.092	0.629 ± 0.110	0.869 ± 0.066	0.908 ± 0.054
GAMIL(2023)	0.933 ± 0.065	0.910 ± 0.085	0.685 ± 0.093	0.894 ± 0.070	0.915 ± 0.058
RGMIL	0.968 ± 0.060	0.985 ± 0.039	0.954 ± 0.048	0.951 ± 0.045	0.965 ± 0.032
Table 1: The classification accuracy (mean ± std) of RGMIL and compared methods on five benchmark datasets (*for S-O-T-A method). The results are the average of five independent experiments with a 10-fold cross-validation. Statistics collected from MI-Net[27], ABMIL[12], Gated-ABMIL [12], DP-MINN[29], Non-Local[26], Asymmetric Non-Local[34], DSMIL[15], BDR[11] and GAMIL[32].
4.2 A Challenging Dataset for Demonstration
Since pain estimation is generally regarded as the classification problems with four classes, we need to set the branch  $N$  to 3. It indicates that we construct a 3-branch RGMIL to meet the upcoming challenges. Due to the extremely imbalance of label distribution, the pain data is not really a clear and flexible enough choice to present a demonstration. Thus, we constructed a flexible MIL dataset based on MNIST [14] and denoted it as MMNIST. In MMNIST, samples are presented in form of bags. A bag is composed of t grayscale images (1,28,28) selected from the MNIST dataset. With  $\mathbf{N} = \mathbf{3}$ , we take the images with label 0 to 3 to form MMNIST bags. The bag label is assigned based on the label with the largest count. The number of training bags for each bag label is approximately 1000. The labels in the bag are uniformly distributed. For the test, we use a fixed number of 10000 test images.
4.3 Evaluation Methods on the Aggregator  $\rho$
We introduce the error rate  $\phi$ , and the distance  $\gamma$  to describe the aggregator  $\rho$  to figure out how good the aggregator  $\rho$  describes a MIL problem.