SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wei Tang $^{1,2}$ , Weijia Zhang $^{3}$ , Min-Ling Zhang $^{1,2*}$
$^{1}$  School of Computer Science and Engineering, Southeast University, Nanjing 210096, China
<sup>2</sup> Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China
$^{3}$  School of Information and Physical Sciences, The University of Newcastle, Callaghan, NSW 2308, Australia
tangw@seu.edu.cn, weijia.zhang@newcastle.edu.au, zhangml@seu.edu.cn
In many real-world tasks, the concerned objects can be represented as a multi-instance bag associated with a candidate label set, which consists of one ground-truth label and several false positive labels. Multi-instance partial-label learning (MIPL) is a learning paradigm to deal with such tasks and has achieved favorable performances. Existing MIPL approach follows the instance-space paradigm by assigning augmented candidate label sets of bags to each instance and aggregating bag-level labels from instance-level labels. However, this scheme may be suboptimal as global bag-level information is ignored and the predicted labels of bags are sensitive to predictions of negative instances. In this paper, we study an alternative scheme where a multi-instance bag is embedded into a single vector representation. Accordingly, an intuitive algorithm named DEMIPL, i.e., Disambiguated attention Embedding for Multi-Instance Partial-Label learning, is proposed. DEMIPL employs a disambiguation attention mechanism to aggregate a multi-instance bag into a single vector representation, followed by a momentum-based disambiguation strategy to identify the ground-truth label from the candidate label set. Furthermore, we introduce a real-world MIPL dataset for colorectal cancer classification. Experimental results on benchmark and real-world datasets validate the superiority of DEMIPL against the compared MIPL and partial-label learning approaches.
Significant advancements in supervised machine learning algorithms have been achieved by utilizing large amounts of labeled training data. However, in numerous tasks, training data is weakly-supervised due to the substantial costs associated with data labeling [1-6]. Weak supervision can be broadly categorized into three types: incomplete, inexact, and inaccurate supervision [7]. Multi-instance learning (MIL) and partial-label learning (PLL) are typical weakly-supervised learning frameworks based on inexact supervision. In MIL, samples are represented by collections of features called bags, where each bag contains multiple instances [8-19]. Only bag-level labels are accessible during training, while instance-level labels are unknown. Consequently, the instance space in MIL contains inexact supervision, signifying that the number and location of positive instances within a positive bag remain undetermined. PLL associates each instance with a candidate label set that contains one ground-truth label and several false positive labels [20-35]. As a result, the label space in PLL embodies inexact supervision, indicating that the ground-truth label of an instance is uncertain.
(a) Instance-level distribution of three MIPL samples
Figure 1: A brief illustration of DEMIPL, where  $\mathcal{S}$  and  $\widehat{Y}$  are candidate label sets and predicted labels, respectively. The ground-truth labels are shown in red.
(b) Aggregated bag-level features and predicted labels by DEMIPL
multi-instance bags
negative instances
positive instances
★ features of multi-instance bags
However, inexact supervision can exist in instance and label space simultaneously, i.e., dual inexact supervision [36]. This phenomenon can be observed in histopathological image classification, where an image is typically partitioned into a multi-instance bag [37-40], and labeling ground-truth labels incurs high costs due to the need for specialized expertise. Consequently, utilizing crowd-sourced candidate label sets will significantly reduce the labeling cost [41]. For this purpose, a learning paradigm called multi-instance partial-label learning (MIPL) has been proposed to work with dual inexact supervision. In MIPL, a training sample is represented as a multi-instance bag associated with a bag-level candidate label set, which comprises a ground-truth label along with several false positive labels. It is noteworthy that the multi-instance bag includes at least one instance that is affiliated with the ground-truth label, while none of the instances belong to any of the false positive labels.
Due to the difficulty in handling dual inexact supervision, to the best of our knowledge, MIPLGP [36] is the only viable MIPL approach. MIPLGP learns from MIPL data at the instance-level by utilizing a label augmentation strategy to assign an augmented candidate label set to each instance, and integrating a Dirichlet disambiguation strategy with the Gaussian processes regression model [42]. Consequently, the learned features of MIPLGP primarily capture local instance-level information, neglecting global bag-level information. This characteristic renders MIPLGP susceptible to negative instance predictions when aggregating bag-level labels from instance-level labels. As illustrated in Figure 1(a), identical or similar negative instances can simultaneously occur in multiple multi-instance bags with diverse candidate label sets, thereby intensifying the challenge of disambiguation.
In this paper, we overcome the limitations of MIPLGP by introducing a novel algorithm, named DEMIPL, i.e., Disambiguated attention Embedding for Multi-Instance Partial-Label learning, based on the embedded-space paradigm. Figure 1(b) illustrates that DEMIPL aggregates each multi-instance bag into a single vector representation, encompassing all instance-level features within the bag. Furthermore, DEMIPL effectively identifies the ground-truth label from the candidate label set.
Our contributions can be summarized as follows: First, we propose a disambiguation attention mechanism for learning attention scores in multi-instance bags. This is in contrast to existing attention-based MIL approaches that are limited to handling classifications with exact bag-level labels [13, 14, 43]. Second, we propose an attention loss function that encourages the attention scores of positive instances to approach one, and those of negative instances to approach zero, ensuring consistency between attention scores and unknown instance-level labels. Third, we leverage the multi-class attention scores to map the multi-instance bags into an embedded space, and propose a momentum-based disambiguation strategy to identify the ground-truth labels of the multi-instance bags from the candidate label sets. In addition, we introduce a real-world MIPL dataset for colorectal cancer classification comprising 7000 images distributed across seven categories. The candidate labels of this dataset are provided by trained crowdsourcing workers.
Experiments are conducted on the benchmark as well as real-world datasets. The experimental results demonstrate that: (a) DEMIPL achieves higher classification accuracy on both benchmark and real-world datasets. (b) The attention loss effectively enhances the disambiguation attention mechanism, accurately discerning the significance of positive and negative instances. (c) The momentum-based disambiguation strategy successfully identifies the ground-truth labels from candidate label sets, especially in scenarios with an increasing number of false positive labels.
The remainder of the paper is structured as follows. First, we introduce DEMIPL in Section 2 and present the experimental results in Section 3. Finally, we conclude the paper in Section 4.
Figure 2: The framework of DEMIPL, where  $\mathcal{L}_a$  and  $\mathcal{L}_m$  are the attention loss and momentum-based disambiguation loss, respectively.
2.1 Notations and Framework of DEMIPL
Let  $\mathcal{X} = \mathbb{R}^d$  represent the instance space, and let  $\mathcal{V} = \{l_1, l_2, \dots, l_k\}$  represent the label space containing  $k$  class labels. The objective of MIPL is to derive a classifier  $f: 2^{\mathcal{X}} \to \mathcal{Y}$ .  $\mathcal{D} = \{(X_i, S_i) \mid 1 \leq i \leq m\}$  is a training dataset that consists of  $m$  bags and their associated candidate label sets. Particularly,  $(X_i, S_i)$  is the  $i$ -th multi-instance partial-label sample, where  $X_i = \{\pmb{x}_{i,1}, \pmb{x}_{i,2}, \dots, \pmb{x}_{i,n_i}\}$  constitutes a bag with  $n_i$  instances, and each instance  $\pmb{x}_{i,j} \in \mathcal{X}$  for  $\forall j \in \{1, 2, \dots, n_i\}$ .  $S_i \subseteq \mathcal{V}$  is the candidate label set that conceals the ground-truth label  $Y_i$ , i.e.,  $Y_i \in S_i$ . It is worth noting that the ground-truth label is unknown during the training process. Assume the latent instance-level labels within  $X_i$  is  $\pmb{y}_i = \{y_{i,1}, y_{i,2}, \dots, y_{i,n_i}\}$ , then  $\exists y_{i,j} = Y_i$  and  $\forall y_{i,j} \notin \mathcal{V} \setminus \{Y_i\}$  hold. In the context of MIPL, an instance is considered a positive instance if its label is identical to the ground-truth label of the bag; otherwise, it is deemed a negative instance. Moreover, the class labels of negative instances do not belong to the label space.
The framework of the proposed DEMIPL is illustrated in Figure 2. It consists of three main steps. First, we extract instances in the multi-instance bag  $\mathbf{X}_i$  and obtain instance-level feature  $\mathbf{H}_i$ . Next, we employ the disambiguation attention mechanism to integrate the multi-instance bag into a single feature vector  $\mathbf{z}_i$ . Finally, we use a classifier to predict the classification confidences  $\mathbf{P}_i$  of the multi-instance bag. To enhance classification performance, we introduce two loss functions for model training: the attention loss  $\mathcal{L}_a$  and the momentum-based disambiguation loss  $\mathcal{L}_m$ . During the training process, the attention mechanism and the classifier work collaboratively.
2.2 Disambiguation Attention Mechanism
Based on the embedded-space paradigm, a key component of DEMIPL is the disambiguation attention mechanism. The attention mechanisms are common models [44-46], which can calculate attention scores to determine the contribution of each instance to the multi-instance bag [13, 14]. The attention scores are then utilized to aggregate the instance-level features into a single vector representation.
For a multi-instance bag  $X_{i} = \{\pmb{x}_{i,1},\pmb{x}_{i,2},\dots ,\pmb{x}_{i,n_{i}}\}$ , we employ a neural network-based function parameterized by  $h$  to extract its feature information:
$$
\boldsymbol {H} _ {i} = h \left(\boldsymbol {X} _ {i}\right) = \left\{\boldsymbol {h} _ {i, 1}, \boldsymbol {h} _ {i, 2}, \dots , \boldsymbol {h} _ {i, n _ {i}} \right\}, \tag {1}
$$
where  $\pmb{h}_{i,j} = h(\pmb{x}_{i,j}) \in \mathbb{R}^{d'}$  is the feature of the  $j$ -th instance within  $i$ -th bag. For the MIPL problems, we propose a multi-class attention mechanism. First, we calculate the relevance of each instance to all classes, and then transform the relevance into the contribution of each instance to the bag-level feature by a learnable linear model. The attention score  $a_{i,j}$  of  $\pmb{x}_{i,j}$  is calculated as follows:
$$
a _ {i, j} = \frac {1}{1 + \exp \left\{- \boldsymbol {W} ^ {\top} \left(\tanh  \left(\boldsymbol {W} _ {v} ^ {\top} \boldsymbol {h} _ {i , j} + \boldsymbol {b} _ {v}\right) \odot \operatorname {s i g m} \left(\boldsymbol {W} _ {u} ^ {\top} \boldsymbol {h} _ {i , j} + \boldsymbol {b} _ {u}\right)\right) \right\}}, \tag {2}
$$
where  $\boldsymbol{W}^{\top} \in \mathbb{R}^{1 \times k}$ ,  $\boldsymbol{W}_v^{\top}$ ,  $\boldsymbol{W}_u^{\top} \in \mathbb{R}^{k \times d'}$ , and  $\boldsymbol{b}_v$ ,  $\boldsymbol{b}_u \in \mathbb{R}^k$  are parameters of the attention mechanism.  $\tanh(\cdot)$  and  $\operatorname{sigm}(\cdot)$  are the hyperbolic tangent and sigmoid functions to generate non-linear outputs for the models, respectively.  $\odot$  represents an element-wise multiplication. Consequently, the bag-level feature is aggregated by weighted sums of instance-level features:
$$
\boldsymbol {z} _ {i} = \frac {1}{\sum_ {j = 1} ^ {n _ {i}} a _ {i , j}} \sum_ {j = 1} ^ {n _ {i}} a _ {i, j} \boldsymbol {h} _ {i, j}, \tag {3}
$$
where  $z_{i}$  is the bag-level feature of  $X_{i}$ . To ensure that the aggregated features accurately represent the multi-instance bag, it is necessary to maintain the consistency between attention scores and instance-level labels, that is, the attention scores of positive instances should be significantly higher than those of negative instances. To achieve this, the proposed attention loss is shown below:
$$
\mathcal {L} _ {a} = - \frac {1}{m} \sum_ {i = 1} ^ {m} \sum_ {j = 1} ^ {n _ {i}} a _ {i, j} \log a _ {i, j}. \tag {4}
$$
Different from existing attention-based MIL approaches where most of them can only handle binary classification, DEMIPL produces multi-class attention scores using Equation (2). Furthermore, unlike loss-based attention [14] that extends binary attention score to multi-class using a naive softmax function, DEMIPL utilizes a learnable model with the attention loss to encourage attention scores of negative and positive instances to approach 0 and 1, respectively. The ambiguity in attention scores is reduced since the differences in attention scores between positive and negative instances are amplified. As a result, the disambiguated attention scores can make bag-level vector representations discriminative, thereby enabling the classifier to accurately identify ground-truth labels.
2.3 Momentum-based Disambiguation Strategy
After obtaining the bag-level feature, the goal is to accurately identify the ground-truth label from the candidate label set. Therefore, we propose a novel disambiguation strategy, namely using the momentum-based disambiguation loss to compute the weighted sum of losses for each category. Specifically, the proposed momentum-based disambiguation loss is defined as follows:
$$
\mathcal {L} _ {m} = \frac {1}{m} \sum_ {i = 1} ^ {m} \sum_ {c = 1} ^ {k} w _ {i, c} ^ {(t)} \ell \left(f _ {c} ^ {(t)} \left(\boldsymbol {z} _ {i} ^ {(t)}\right), \mathcal {S} _ {i}\right), \tag {5}
$$
where  $(t)$  refers to the  $t$ -th epoch.  $\pmb{z}_i^{(t)}$  is the bag-level feature of multi-instance bag  $\mathbf{X}_i$  and  $f_c^{(t)}(\cdot)$  is the model output on the  $c$ -th class at the  $t$ -th epoch.  $\ell(\cdot)$  is the cross-entropy loss, and  $w_{i,c}^{(t)}$  weights the loss value on the  $c$ -th class at the  $t$ -th epoch.
Following the principle of the identification-based disambiguation strategy [47], the label with the minimal loss value on the candidate label set can be considered the ground-truth label. We aim to assign a weight of 1 to the single ground-truth label and a weight of 0 to the rest of the candidate labels. However, the ground-truth label is unknown during the training process. To overcome this issue, we allocate weights based on the magnitude of class probabilities, ensuring that larger class probabilities are associated with higher weights. Specifically, we initialize the weights by:
$$
w _ {i, c} ^ {(0)} = \left\{ \begin{array}{c l} \frac {1}{| \mathcal {S} _ {i} |} & \text {i f} Y _ {i, c} \in \mathcal {S} _ {i}, \\ 0 & \text {o t h e r w i s e}, \end{array} \right. \tag {6}
$$
where  $\frac{1}{|\mathcal{S}_i|}$  is the cardinality of the candidate label set  $S_{i}$ . The weights are updated as follows:
$$
w _ {i, c} ^ {(t)} = \left\{ \begin{array}{c l} \lambda^ {(t)} w _ {i, c} ^ {(t - 1)} + (1 - \lambda^ {(t)}) \frac {f _ {c} ^ {(t)} \left(\boldsymbol {z} _ {i} ^ {(t)}\right)}{\sum_ {j \in \boldsymbol {S} _ {i}} f _ {j} ^ {(t)} \left(\boldsymbol {z} _ {j} ^ {(t)}\right)} & \text {i f} Y _ {i, c} \in \mathcal {S} _ {i}, \\ 0 & \text {o t h e r w i s e ,} \end{array} \right. \tag {7}
$$
where the momentum parameter  $\lambda^{(t)} = \frac{T - t}{T}$  is a trade-off between the weights at the last epoch and the outputs at the current epoch.  $T$  is the maximum training epoch.
It is worth noting that the momentum-based disambiguation strategy is a general form of the progressive disambiguation strategy. Specifically, when  $\lambda^{(t)} = 0$ , the momentum-based disambiguation strategy degenerates into the progressive disambiguation strategy [47]. When  $\lambda^{(t)} = 1$ , the momentum-based disambiguation strategy degenerates into the averaging-based disambiguation strategy [21], which equally treats every candidate label.
2.4 Synergy between Attention Mechanism and Disambiguation Strategy
Combining the attention loss and disambiguation loss, the full loss function is derived as follows:
$$
\mathcal {L} = \mathcal {L} _ {m} + \lambda_ {a} \mathcal {L} _ {a}, \tag {8}
$$
where  $\lambda_{a}$  serves as a constant weight for the attention loss. In each iteration, the disambiguation attention mechanism aggregates a discriminative vector representation for each multi-instance bag. Subsequently, the momentum-based disambiguation strategy takes that feature as input and yields the disambiguated candidate label set, i.e., class probabilities. Meanwhile, the attention mechanism relies on the disambiguated candidate label set to derive attention scores. Thus, the disambiguation attention mechanism and the momentum-based disambiguation strategy work collaboratively.
Benchmark Datasets We utilize four benchmark MIPL datasets stemming from MIPLGP literature [36], i.e., MNIST-MIPL, FMNIST-MIPL, Birdsong-MIPL, and SIVAL-MIPL from domains of image and biology [48-51]. Table 1 summarizes the characteristics of both the benchmark and real-world datasets. We use  $\#$  bags,  $\#$  ins,  $\#$  dim, avg.  $\#$  ins,  $\#$  class, and avg.  $\#$  CLs to denote the number of bags, number of instances, dimension of each instance, average number of instances in all bags, number of class labels, and the average size of candidate label set in each dataset.
Real-World Dataset We introduce CRC-MIPL, the first real-world MIPL dataset for colorectal cancer classification (CRC). It comprises 7000 hematoxylin and eosin (H&E) staining images taken from colorectal cancer and normal tissues. Each image has dimensions of  $224 \times 224$  pixels and is categorized into one of the seven classes based on the tissue cell types. CRC-MIPL is derived from a larger dataset used for colorectal cancer classification, which originally contains 100000 images with nine classes [52]. The adipose and background classes exhibit significant dissimilarities compared to the other categories. Therefore, we choose the remaining seven classes to sample 1000 images per class. These classes include debris, lymphocytes, mucus, smooth muscle, normal colon mucosa, cancer-associated stroma, and colorectal adenocarcinoma epithelium.
We employ four image bag generators [53]: Row [54], single blob with neighbors (SBN) [54], k-means segmentation (KMeansSeg) [55], and scale-invariant feature transform (SIFT) [56], to obtain a bag of instances from each image, respectively. The candidate label sets of CRC-MIPL are provided by three crowdsourcing workers without expert pathologists. Each of the workers annotates all 7000 images, and each worker assigned candidate labels with non-zero probabilities to form a label set per image. A higher probability indicates a higher likelihood of being the ground-truth label, while a probability of zero implies the label is a non-candidate label. After obtaining three label sets for each image, we distill a final candidate label set as follows. A label present in two or three label sets is selected as a member of the final candidate label set. If the final candidate label set consists of only one or no label, we pick the labels corresponding to the highest probability in each label set. The average length of the final candidate label set per image is 2.08. More detailed information on the MIPL datasets can be found in the Appendix.
Compared Algorithms For comparative studies, we consider one MIPL algorithm MIPLGP [36] and four PLL algorithms, containing one feature-aware disambiguation algorithm PL-AGGD [32] and three deep learning-based algorithms, namely PRODEN [47], RC [57], and LWS [58].
Table 1: Characteristics of the Benchmark and Real-World MIPL Datasets.
Dataset	#bags	#ins	#dim	avg. #ins	#class	avg. #CLs	domain
MNIST-MIPL	500	20664	784	41.33	5	2,3,4	image
FMNIST-MIPL	500	20810	784	41.62	5	2,3,4	image
Birdsong-MIPL	1300	48425	38	37.25	13	2,3,4,5,6,7	biology
SIVAL-MIPL	1500	47414	30	31.61	25	2,3,4	image
CRC-MIPL-Row	7000	56000	9	8	7	2.08	image
CRC-MIPL-SBN	7000	63000	15	9	7	2.08	image
CRC-MIPL-KMeansSeg	7000	30178	6	4.311	7	2.08	image
CRC-MIPL-SIFT	7000	175000	128	25	7	2.08	image
Implementation DEMIPL is implemented using PyTorch [59] on a single Nvidia Tesla V100 GPU. We employ the stochastic gradient descent (SGD) optimizer with a momentum of 0.9 and weight decay of 0.0001. The initial learning rate is chosen from a set of  $\{0.01, 0.05\}$  and is decayed using a cosine annealing method [60]. The number of epochs is set to 200 for the SIVAL-MIPL and CRC-MIPL datasets, and 100 for the remaining three datasets. The value of  $\lambda_{a}$  is selected from a set of  $\{0.0001, 0.001\}$ . For the MNIST-MIPL and FMNIST-MIPL datasets, we utilize a two-layer CNN in work of Ilse et al. [13] as a feature extraction network, whereas for the remaining datasets, no feature extraction network is employed. To ensure the reliability of the results, we conduct ten runs of random train/test splits with a ratio of  $7:3$  for all datasets. The mean accuracies and standard deviations are recorded for each algorithm. Subsequently, we perform pairwise t-test at a significance level of 0.05.
To map MIPL data into PLL data, we employ the Mean strategy and the MaxMin strategy as described in the MIPLGP literature [36]. The Mean strategy calculates the average values of each feature dimension for all instances within a multi-instance bag to yield a bag-level vector representation. The MaxMin strategy involves computing the maximum and minimum values of each feature dimension for all instances within a multi-instance bag and concatenating them to construct bag-level features. In the subsequent section, we report the results of PRODEN, RC, and LWS using linear models, while the results with multi-layer perceptrons are provided in the Appendix.
3.2 Experimental Results on the Benchmark Datasets
Table 2 presents the classification results achieved by DEMIPL and the compared algorithms on the benchmark datasets with varying numbers of false positive labels  $r$ . Compared to MIPLGP, DEMIPL demonstrates superior performance in 8 out of 12 cases, with no significant difference observed in the remaining 1 out of 12 cases. MIPLGP performs better than DEMIPL on the SIVAL-MIPL dataset, primarily due to the unique characteristics of the SIVAL-MIPL dataset. The dataset encompasses 25 highly diverse categories, such as apples, medals, books, and shoes, resulting in distinctive and discriminative features within multi-instance bags. Each instance's feature includes color and texture information derived from the instance itself as well as its four cardinal neighbors, which enhances the distinctiveness of instance-level features. This suggests that instances with similar features are rarely found across different multi-instance bags. Therefore, by following the instance-space paradigm, MIPLGP effectively leverages these distinctive attributes of the dataset.