Combining Supervised Learning and Reinforcement Learning for Multi-Label Classification Tasks with Partial Labels (2024)

Abstract: Traditional supervised learning heavily relies on human-annotated datasets, especially in data-hungry neural approaches. However, various tasks, especially multi-label tasks like document-level relation extraction, pose challenges in fully manual annotation due to the specific domain knowledge and large class sets. Therefore, we address the multi-label positiveunlabelled learning (MLPUL) problem, where only a subset of positive classes is annotated. We propose Mixture Learner for Partially Annotated Classification (MLPAC), an RL-based framework combining the exploration ability of reinforcement learning and the exploitation ability of supervised learning. Experimental results across various tasks, including documentlevel relation extraction, multi-label image classification, and binary PU learning, demonstrate the generalization and effectiveness of our framework.

heading: 1 Introduction
text: Multi-Label Classification (MLC) task treats a problem that allows instances to take multiple labels, and traditional Supervised Learning (SL) methods on MLC heavily rely on human-annotated data sets, especially neural approaches that are datahungry and susceptible to over-fitting when lacking training data. However, in many MLC tasks that generally have dozens or hundreds of sizes of class sets, incompleteness in the acquired annotations frequently arises owing to the limited availability of expert annotators or the subjective nature inherent in human annotation processes. (Kanehira and Harada, 2016; Cole et al., 2021; Tan et al., 2022; Ben-Baruch et al., 2022). Therefore, we focus on the fundamentally important problem, typically termed Multi-Label Positive-Unlabelled Learning (MLPUL) (Kanehira and Harada, 2016; Teisseyre, 2021), which involves learning from a multi-label
* Correspondence to Zilong Zheng.
dataset in which only a subset of positive classes is definitely annotated, while all the remaining classes are unknown (which could be positives or negatives). For instance, as shown in Fig. 1A&B, human annotators find it hard to completely annotate all the relations due to the confusion of understanding relation definitions and long-context semantics in document-level relation extraction (DocRE) task (Huang et al., 2022; Tan et al., 2022).
Positive and unlabelled (PU) classification has received extensive attention in binary settings, with several recent MLPUL approaches adapting traditional binary PU loss functions to address multilabel classification tasks (Kanehira and Harada, 2016; Wang et al., 2022, 2024). These methods typically operate under the assumption that the prior distribution of positive labels can be inferred from fully labeled training samples or closely unbiased estimations. In a specific instance, (Wang et al., 2022) supposed that the actual positive classes are three times the number of observed labels in the DocRE task, and their model’s performance is heavily influenced by the prior (Fig. 1C). However, estimating the prior distribution of labels in real-world scenarios poses significant challenges, as it is rarely feasible to ensure a comprehensive data set encompassing all label types (Chen et al., 2020; Hu et al., 2021; Yuan et al., 2023). Additionally, Li et al. (2023) noted that many long-tail label types tend to be omitted from training annotations. Consequently, we focus on addressing MLPUL without prior knowledge of class distribution. Moreover, MLC generally faces the challenge of imbalanced positive and negative labels, which is severely exacerbated by missing positive class annotations under MLPUL, as shown in Fig. 1B. Previous works typically adopted the re-balance factor to re-weight the loss functions, containing positive up-weight and negative under-weight (Li et al., 2020). We simply attempt these approaches and find they partly improve the model performance but still perform
13553
unsatisfactorily when only a very small set (10%) of positive class annotations is available (Fig. 1D).
Previous works (Silver et al., 2016; Feng et al., 2018; Nooralahzadeh et al., 2019) have demonstrated the powerful exploration ability of Reinforcement Learning (RL). Furthermore, RL has shown great success on distant or partial annotations recently (Feng et al., 2018; Luo et al., 2021; Chen et al., 2023). Inspired by these successful RL attempts, we believe that the exploratory nature of RL has the potential ability to discover additional positive classes while mitigating the overfitting issues typically encountered in supervised learning, especially when the observed label distribution is severely biased, which holds promise in addressing MLPUL. Besides, recent works have shown that supervised learning can be remarkably effective for the RL process (Emmons et al., 2021; Park et al., 2021; Badrinath et al., 2024).
Based on this intuition, we introduce a novel framework termed Mixture Learner for Partially Annotated Classification (MLPAC), which combines the exploratory capacity of RL in tandem with the exploitation capabilities of supervised learning. Specifically, we design a policy network (as a multi-label classifier) and a critic network, along with two types of reward functions: global rewards calculated by a recall function, which evaluates the all-classes prediction performance for each instance, and local rewards provided by the critic network, which assesses the prediction quality of each individual class for a given instance. The local rewards are expected to narrow the exploration space of traditional RL and offer a preliminary yet instructive signal to guide the learning process, while the global rewards encourage the policy network to explore a broader spectrum of positive classes, consequently mitigating distribution bias stemming from imbalanced labels and incomplete
annotations. In addition, inspired by the traditional actorcritic RL algorithm (Bahdanau et al., 2016), we iteratively train the policy network and the critic network, which achieves dynamic reward estimation in our setting. The absence of fully annotated samples in both training and validation sets precludes the direct attainment of perfectly accurate rewards. Hence, we introduce label enhancement through collaborative policy and critic network efforts during iterative training, boosting label confidence and enhancing the critic network’s reward estimation accuracy.
Moreover, our RL framework is concise and flexible, guaranteeing its generalization and adaptation to many tasks. Beyond the experiments on document-level relation extraction task (§4.1) in Natural Language Process (NLP) field, we also conduct sufficient experiments in multi-label image classification task (§4.2) in Computer Vision (CV) field and general PU learning setting in binary case (§4.3) to verify the generalization and effectiveness of our framework. All experimental results demonstrate the advantage and significant improvement of our framework.

heading: 2 Related Work
text: Multi-label Positive-Unlabelled Learning Methods Label correlation modeling, rank-based weighted loss function and enhanced incomplete labels are typical technologies that previous MLPUL methods adopted. Label correlation is usually learned or calculated from the label matrix of the training data under partial positive and negative samples under multi-label partially observed labeling (Rastogi and Mortaza, 2021; Kumar and Rastogi, 2022; Jiang et al., 2023; Yu et al., 2024). In the MLPUL setting, Teisseyre (2021) leveraged classifier chains to model high-order label corre-
lation with the assumption that prior probabilities could be estimated by predictions from the previous models in the chain. Because label correlations may bring correlation bias under the severe scarcity of positive labels, we did not explicitly model the correlations in our model.
Kanehira and Harada (2016) extended binary PU classification to the multi-label setting, dealing with multi-label PU ranking problem. They modeled MLPUL as cost-sensitive learning by designing a weighted rank loss for multi-label image classification tasks. Following this, Wang et al. (2022) proposed shift and squared ranking loss PU learning for the document-level relation extraction task by modifying the prior terms in the rank-based loss function. Both works supposed the knowledge of class prior distribution, but the prior is difficult to estimate in reality. Recently, encouraged by the binary PU methodology without class prior (Chen et al., 2020; Hu et al., 2021), Yuan et al. (2023) adopted the variational binary PU loss and additionally dealt with the catastrophic imbalanced positive and negative label distribution that MLPUL faced by introducing an adaptive re-balance factor and adaptive temperature coefficient in the loss function. Our approach also pays attention to solving the imbalanced problem in MLPUL and designs a framework without the knowledge of class priors. Different from their methods specifically designed for image classification, our framework is effective for a wide range of MLPUL tasks and is further adapted for imbalanced binary PU learning cases.
Besides the above methodologies, Chen et al. (2020); Yuan et al. (2023) proposed regularization term based on Mixup (Zhang et al., 2018) to enhance incomplete labels, alleviating the overfitting problem and increasing model robustness. The concurrent work to us, Wang et al. (2024) designs positive-augmentation and positive-mixup strategies to improve rank-based learning methods
(Wang et al., 2022) for PU document-level relation extraction. These strategies are not uniquely applicable to their framework, and our method can also integrate them, which we leave to future work.
Reinforcement Learning under Weak Supervision There are many previous works leveraging Reinforcement Learning (RL) to solve tasks only with weak supervision (Feng et al., 2018; Zeng et al., 2018; Luo et al., 2021; Chen et al., 2023). In the NLP field, to precisely leverage distant data, Qin et al. (2018); Feng et al. (2018) train an agent as a noisy-sentence filter, taking performance variation on development or probabilities of selected samples as a reward and adopting policy gradient to update. Nooralahzadeh et al. (2019) expand their methods to NER task. Recent work of Chen et al. (2023) also conducts RL to remove the noisy sentence so as to improve the fault diagnosis system. Notably, RL learning on distantly supervised learning aims to filter false positives, whereas our goal is to identify false negatives. A closer related work to us is Luo et al. (2021), in which an RL method is designed to solve the PU learning problem. But unlike us, their agent is a negative sample selector, aiming to find negatives with high confidence and then couple them with partial positives to train a classifier. More related works under different weak supervision settings can be found in Appendix A.

heading: 3 RL-based Framework
text: We propose a novel RL framework to solve the MLPUL task. We formulate the multi-label prediction as the action execution in the Markov Decision Process (MDP) (Puterman, 1990). We design both local and global rewards for actions to guide the action decision process. The policy-based RL method is adopted to train our policy network. The overall RL framework is illustrated in Figure 2. We introduce the details in the following subsections.

heading: 3.1 Problem Setting
text: We mathematically formulate the MLPUL task. Given a multi-label dataset X = {xi}, each xi is labeled with a partially annotated multi-hot vector yi = [y1i , · · · , yci , · · · y |C| i ], where y c i ∈ {0, 1} denotes whether class c ∈ C is TRUE for instance xi and |C| is the cardinality of set C. In terms of partial annotation, we assume that {c; yci = 1} is a subset of gold positive classes regarding to xi, i.e., {c; yci = 0} is the set of UNKNOWN classes (which could be actually positive or negative). Typically, in the multi-label setting, the size of the label set is dozens or hundreds; thus |{c; yci = 1}| << |{c; yci = 0}| ≤ |C|. In some complicated tasks, such as Relation Extraction, we further define a special label <None> to xi if yi = [yci = 0] |C| c=1. It should be mentioned that we do not have any fully annotated data, both the training and validation sets being partially annotated.
A straightforward approach (referred to as the “negative mode”) for tackling this task involves treating all unlabeled classes as FALSE (set all yci = 0 to y c i = −1) and subsequently reducing the multi-label problem to a number of independent binary classification tasks by employing conventional supervised learning (SL). However, due to incomplete positive labels and severely imbalanced label distribution, the negative mode is susceptible to overfit biased annotated labels, resulting in high precision but low recall on the test set. Based on the negative mode, we introduce an RL framework (MLPAC) that combines RL and SL to mitigate distribution bias and encourage the multi-label classifier to predict more potential positive classes.

heading: 3.2 Modeling
text: Typically, basic RL is modeled as an MDP (S,A, π, T , R) which contains a set of environment and agent states S, a set of actions A of the agent, the transition probabilities T from a state to another state under action a, and the reward R. The goal of an RL agent is to learn a policy π that maximizes the expected cumulative reward. In our problem setting, we formalize the multi-label positive-unlabelled learning as a one-step MDP problem: we do not consider state transitions because our action execution does not change the agent and environment. Our setting highly resembles the setting of contextual bandits (Chu et al., 2011), where actions only affect the reward but not the state. Our RL framework’s policy πθ is a
multi-label classifier constructed by a neural network with parameter θ. We define the constituents in detail.
States A state s includes the potential information of an instance to be labeled. In our setting, this information consists of instance features, which are essentially continuous real-valued vectors derived from a neural network.
Actions Due to the multi-label setting, our agent is required to determine the label of each class c for one instance. There are two actions for our agent: setting the current class as TRUE (ŷci = 1) or FALSE (ŷci = −1). It is necessary to execute |C| (size of the class set) actions to label an instance completely.
Policy Our policy network outputs the probability πθ(ŷ c i |xi) = P (a = ŷci |s = xi) for each action condition on the current state. We adopt the model structure commonly utilized in previous supervised studies as the architecture for our policy network.
Rewards Recall that our primary objective is to acquire a less biased label distribution compared to the supervised negative mode training approach using the partially annotated training dataset. We anticipate that our MLPAC possesses the capacity for balanced consideration of both exploitation and exploration. Exploitation ensures that our agent avoids straying from local optima direction and avoids engaging in excessively invalid exploratory behavior, while exploration motivates our agent to explore the action space somewhat randomly and adapt its policy, preventing overfitting to partial supervision. Inspired by the actor-critic RL algorithm (Bahdanau et al., 2016), we design our rewards function containing two parts: a local reward provided by a trainable critic network, which provides immediate value estimation of each action and a global reward regarding the overall performance of all classes predictions for each instance.
Specifically, inspired by Luo et al. (2021), the local reward calculates the reward of each class prediction for each instance according to the critic network confidence:
rci (Vλ,xi, c) (1)
=    C(−1, log p c Vλ (xi) 1−pcVλ (xi) , 1) if ŷci = 1, C(−1, log 1−p c Vλ (xi)
pcVλ (xi)
, 1) if ŷci = −1.
where pcVλ(xi) and 1−p c Vλ (xi) are the probabilities of class c being TRUE and FALSE respectively for an
instance xi, calculated by a critic network Vλ with parameter λ, ŷci denotes the prediction of policy network, and C(−1, ·, 1) is a clamping function: a) C(−1, x, 1) = −1 if x < −1; b) C(−1, x, 1) = 1 if x > 1; c) otherwise, C(−1, x, 1) = x.
We train the critic network in a supervised fashion (under negative mode) to guide the policy network’s exploration direction through the local reward function, thus equipping our framework with the ability of exploitation. From Equation 2, the action of our policy network could get a positive reward if the critic network has the same prediction tendency and a negative reward otherwise. The policy network is rewarded to “softly fit” the distribution learned by the critic network. To improve the accuracy of value estimation, we perform label enhancement to train our critic network (described in Section 3.4). Consequently, the local rewards offer a preliminary yet instructive signal to guide the learning process in our MLPAC framework, thereby preventing the MLPAC from engaging in excessively invalid exploratory behavior within the large action space, thereby enhancing the overall learning efficiency. Nevertheless, relying solely on these local rewards may potentially lead the MLPAC system to converge to a biased “negative mode” solution. To mitigate this risk, we introduce global rewards to stimulate more comprehensive exploration during the learning process.
As for global reward, we employ a straightforward yet highly effective scoring function, which is computed based on the recall metric. In detail, for the whole classes prediction ŷi of xi with the observed ground truth yi, the recall score is:
recall(yi, ŷi) (2)
= |{yci = 1 ∧ ŷci = 1, yci ∈ yi, ŷci ∈ ŷi}|
|{yci = 1, yci ∈ yi}| To enhance recall scores, our policy network is encouraged to predict a greater number of classes as TRUE, thereby alleviating the catastrophic label imbalanced challenge 1.
Note that in our reward design, the terms “local” and “global” are both used to characterize the “goodness” of the predictions of input instances (i.e., state-action pairs). They are both immediate rewards in the considered RL framework, as we formalize MLPUL as a one-step MDP. To calculate the final reward of the whole predictions of an
1Note that we do not punish the action of wrongly setting a class as TRUE. Thus, the policy network is pleased to predict all classes as TRUE if we only leverage the global reward function
instance, the local rewards of all predicted classes c are summed out, eventually weighted summed with global reward:
R(xi, ŷi, Vλ,yi) (3)
= 1 |C| ∑
c∈C rci (Vλ,xi, c) + w ∗ recall(yi, ŷi),
where w is a weight controlling the scale balance between local reward and global reward.

heading: 3.3 Inference
text: The final predictions of each instance are decided according to the probabilities that our policy network output. We simply set classes whose probabilities are more than 0.5 (πθ(ŷci |xi) > 0.5) to as TRUE (i.e., ŷci = 1).

heading: 3.4 Learning
text: We iteratively train our critic network and policy network in an end-to-end fashion. Since the critic network plays a critical role in guiding policy network learning, we employ label enhancement techniques during the training of the critic network to enhance the precision of value estimations. It is important to emphasize that we intentionally exclude the enhanced labels from participation in the calculation of the recall reward. This decision is motivated by the desire to maintain the precision of the global reward and prevent potential noise introduced by the enhanced labels.
It is widely acknowledged that the training process in RL can experience slow convergence when confronted with a vast exploration space. Inspired by previous RL-related works (Silver et al., 2016; Qin et al., 2018), we initiate our process by conducting pre-training for both our policy and critic networks before proceeding with the RL phase. Typically, pre-training is executed through a supervised method. In our settings, a range of trivial solutions for MLPUL can serve as suitable candidates for pre-training. In most cases, we simply utilize the negative mode for pre-training. However, as previously mentioned, the negative mode tends to acquire a biased label distribution due to severe label imbalance. Thus, we implement an earlystopping strategy during the pre-training phase to prevent convergence. The following introduces the detailed learning strategies and objectives.
Objective for Value Model Generally, a welldesigned supervised objective urges models to
Algorithm 1: Partially Annotated Policy Gradient Algorithm Input: Observed data X , partial labels Y ,
pre-trained policy network πθ0 , critic network Vλ, REINFORCE learning rate α, confidence threshold γ, sample steps T
Output: Optimal parameters θ∗ 1 e← 0, θ∗ ← θ0, enhanced annotation set Ȳ ← Y 2 while e < total training epoches do 3 Training set for critic network: (X , Ȳ) 4 Training set for policy network: (X ,Y) 5 for Xbatch ∈ X in total batches do 6 Update λ by minimizing Equation 4 with Ȳbatch for critic network 7 for step t < sample steps T do 8 For each xi ∈ Xbatch, sample ŷi w.r.t. ŷi ∼ πθ(ŷi|xi) 9 Compute R(xi, ŷi, Vλ,yi)
according to Equation 3
10 Update policy network using θ ← θ + α∇θJPG(θ) 11 Ȳ ← {[ȳci ] |C| c=1}, where ȳci = 1 if (yci = 1
or ( πθ(ŷ c i = 1|xi) > γ and ŷci =
Vλ(xi)c = 1 ) )
12 if eval(πθ) > eval(πθ∗) then 13 θ∗ ← θ 14 e← e+ 1 15 return θ∗
learn expected outputs by learning from annotated data. This process typically refers to the exploitation, where the supervised model fits the distribution of label annotations. We denote the supervised objective by a general formulation:
LSUP (θ) = ∑
xi∈X p(xi)D(yi, ŷi), (4)
where D is a task-specific distance metric measuring the distance between annotation yi and prediction ŷi. Recall that we treat all the unknown classes as negatives to perform supervised learning.
Objective for Policy Model As stated in previous work (Qin et al., 2018), policy-based RL is more effective than value-based RL in classification tasks because the stochastic policies of the policy network are capable of preventing the agent from getting stuck in an intermediate state. We
leverage policy-based optimization for RL training. The objective is to maximize the expected reward:
JPG(θ) = Eπθ [R(θ)] (5) ≈ ∑
xi∈batch p(xi)
∑
ŷi∼πθ(ŷi|xi) πθ(ŷi|xi)R(ŷi.xi),
The policy network πθ can be optimized w.r.t. the policy gradient REINFORCE algorithm (Williams, 1992), where the gradient is computed by
∇θJPG(θ) = ∑
xi∈batch p(xi) (6)
∑
ŷi∼πθ(ŷi|xi) ∇θ ln(πθ(ŷi|xi))R(ŷi,xi),
where p(xi) is a prior distribution of input data. Specific to uniform distribution, p(x) = 1|xbatch| .
Overall Training Procedure The overall training process is demonstrated in Algorithm 1, where ŷci = Vλ(xi)c = 1 refers to the prediction of class c that critic network outputs for the sample xi being TRUE (pcVλ(xi) > 0.5). There are several strategies that need to be clarified in the overall training procedure. We empirically prove the effectiveness of these strategies. The Computation of local rewards is based on the annotated positive classes and a randomly selected subset of unknown classes rather than considering the entire class set of an instance, which is intended to emphasize the impact of positive classes within the computation of local rewards. The enhanced labels are determined by both the policy network and critic network to guarantee high confidence of enhanced positive labels. We fix the critic network once it converges to enhance training efficiency.

heading: 4 Experiments
text: The Multi-Label Positive Unlabelled Learning (MLPUL) problem is common and essential in the NLP field. There are many tasks in NLP that face incomplete annotation problems, such as finegrained entity typing, multi-label text classification, and document-level relation extraction. To verify the effectiveness of our proposed RL framework, we experiment with the positive-unlabelled document-level relation extraction (DocRE) task as a representative MLPUL task in NLP. We also conduct experiments in multi-label image classification (MLIC) tasks and binary PU learning setting
to verify the generalization and effectiveness of our framework.
Besides comparing with previous state-of-the-art (SOTA) models in each specific task, we construct some simple baseline methods: • Negative Mode: As mentioned in Section 3.1,
all unknown labels are treated as negative labels, performing conditional supervised learning. • Pos Weight: Based on negative mode, we upweight positive labels’ loss in the supervised loss. • Neg Weight: Based on negative mode, we perform negative sampling in the supervised loss. We train the model on 1 NVIDIA A100 GPU. The total number of training epochs is 30. We iteratively train our critic and policy network for the first 10 epochs, and then we only train the policy network for the last 20 epochs. The threshold γ of choosing enhancement labels is 0.95 in most cases, and we find our framework performs robustly to γ varying from 0.5 to 0.95. We tune the hyperparameter reward weight w in Eq.3 and sampling number T with different experiment settings, and w is dynamically adjusted during training2. All the above hyper-parameters are determined according to validation set performance. We leverage the F1 score and MAP score (for MLIC) to evaluate all the models. We conduct experiments on selected datasets with varying ratios of positive class annotations from 10% to 100%. We randomly keep a ratio of annotated relations and treat all the leaving classes as unknown. Part of the results are shown in the experimental tables. Detailed descriptions of these simple baselines, evaluation metrics, data statistics, and full experimental results can be found
2Intuitively, the w of recall reward should be dropped along with the training epochs because our critic network provides more and more accurate local rewards beneficial by data enhancement before convergence.
in Appendix B, C, D, and E respectively.

heading: 4.1 Document-level Relation Extraction
text: Document-level Relation Extraction (DocRE) is a task that focuses on extracting fine-grained relations between entity pairs within a lengthy context. Align to our formulation, an input xi is an entity pair, and yi represents relation types between the entity pair. An entity pair may have multiple relations or have no relation in DocRE. Beyond the experiments trained on an incomplete annotated DocRED training set and tested on an almost fully annotated Re-DocRED test set (as shown in Fig. 1C), we choose the Re-DocRED, which is the most complete annotated dataset in DocRE, for experiments on varying ratio of positive annotations. The size of class set C is 97 (contains <None>).
Configuration and Baselines We adopt the fully supervised SOTA, DREEAM (Ma et al., 2023), as our critic and policy network architectures in this experiment. We keep the same training method with an Adaptive Thresholding Loss (ATL) of DREEAM for our critic network. The hyperparameter w of reward weight in Eq. (3) is set to 10. The sample steps T in RL is set to 10. We compare our method to SSR-PU (Wang et al., 2022) and show the results of concurrent work PM3 (Wang et al., 2024) for reference. Both of these two models perform rank-based PU loss with an assumption of label distribution prior.
Results Experimental results in Re-DocRED are shown in Table 1. Compared to previous work and our simple baselines, our MLPAC demonstrates its advantage in all annotation ratios. Besides, our framework achieves more balanced precision-recall scores, suggesting its ability to deal with imbalanced label challenges and predict more positive
labels. It is worth noting that our framework also achieves improved performance with the full annotated dataset because the full annotations of ReDocRED still miss some actual relations, as mentioned in Li et al. (2023). Furthermore, our framework does not rely on label prior estimation, while previous rank-based methods are sensitive to a certain extent (fix-prior vs. vary-prior).

heading: 4.2 Multi-Label Image Classification
text: Multi-label image classification is the task of predicting a set of labels corresponding to objects, attributes, or other entities present in an image. Following previous work (Kanehira and Harada, 2016; Ben-Baruch et al., 2022), we utilize MS-COCO dataset (Lin et al., 2014) containing 80 classes.
Configuration and Baselines Our policy and critic networks adopt the same architecture as PASL (Ben-Baruch et al., 2022). We rerun previous works (ERP, ROLE, P-ASL) with official code (Cole et al., 2021; Ben-Baruch et al., 2022) for fair comparisons with our methods under the same training data samples. We tune the hyper-parameter w between {5, 7, 12} in this task. The sample steps T in RL is set to 50. Detailed descriptions of compared models can be found in Appendix D.
Results Experimental results are shown in Section 4.2. Our MLPAC still performs competitively in the MLIC task. Furthermore, we ran our model three times and found very small standard deviations of F1 scores and mAP, which demonstrates the high robustness and stability of our framework. Standard deviations of three runs and more experimental results can be found in Appendix E.2.

heading: 4.3 Binary PU Learning Setting
text: To verify the generalization and wide adaptation of our RL framework, we conduct binary PU learning
with the same setting following Su et al. (2021) that concerns positive/negative imbalanced problems in binary image classification. The imbalanced datasets are constructed from CIFAR103 by picking only one category as positives and treating all other data as negatives.
Configuration and Baselines Of note, our framework can integrate any supervised model architecture. For a fair comparison, we take the same architecture of Kiryo et al. (2017); Su et al. (2021) as our critic and policy networks. We compared to nnPU (Kiryo et al., 2017) and ImbalancednnPU (Su et al., 2021). More details are in Appendix D. We tune the hyper-parameter w between {10, 20, 50}. The action sampling number T is 100.
Results We show F1 scores with varying ratios of annotated positives in Fig. 3. Our MLPAC achieves significant improvements over Negative
3A multi-class dataset containing ten categories. https: //web.cs.toronto.edu/
Mode and previous work, It is worth mentioning that our framework still demonstrates its superiority though ImbalancednpU is specifically designed for binary PU learning with imbalanced settings.

heading: 4.4 Analysis
text: We conduct ablation studies to analyze our MLPAC framework both on modeling and training strategy.
Rewards Design: To show the effectiveness of combining exploitation and exploration and the benefit of local and global rewards, we train our framework in the 10% annotations setting without local rewards and global rewards, respectively. Additionally, we replace the recall scores with precision w. Prec or F1 scores w. F1 as our global rewards to show the effects of different global reward designs. Experimental results are shown in Table 3. It can be observed that it is hard for an RL framework to achieve comparable performance without local rewards to guide exploitation. The reason is that the action space of multi-label classification is too large to find the global optimal directions. Without our global reward, the recall evaluation score drops a lot (72.78 vs. 45.94), which demonstrates the advantage of the global reward in alleviating imbalance distribution. Both the two variants of global reward damage the performance, revealing the advance of taking the exactly accurate evaluation as rewards in the partially annotated setting.
Training Strategy: To verify the effectiveness of our training procedure, we attempt different training strategies shown in Tabel 4. w/o. Iterative training means that we fix the critic network after pretraining and only train the policy network in the RL training procedure. w/o. Data enhancement means that we still iteratively train our critic and policy network but do not enhance pseudo labels for the critic network. w/o. Action sampling means that we leverage the whole action sequence to calculate local rewards without sampling operation illustrated in Section 3.4. Supervised self-training means that we conduct self-training of the critic network. It is obvious that our training method makes remarkable achievements. More analysis experiments are in Appendix E.1.

heading: 5 Conclusion
text: In this work, we propose an RL framework to deal with partially annotated multi-label classification tasks. We design local rewards assessed by a critic
network and global rewards assessed by recall functions to guide the learning of our policy network, achieving both exploitation and exploration. With an iterative training procedure and a cautious data enhancement, our MLPAC has demonstrated its effectiveness and superiority on different tasks.
Limitation
We have considered the label correlations in our challenge. However, in our setting, label correlations may bring correlation bias to the severe scarcity of positive labels. Therefore, we did not explicitly model the correlations in our current framework. We would like to explore the potential of leveraging the label correlations to enhance our framework in future work.

heading: Acknowledgements
text: The authors thank the reviewers for their insightful suggestions on improving the manuscript. This work presented herein is supported by the National Natural Science Foundation of China (62376031).

heading: A Related Work
text: Different Settings under Weak Supervision Many previous works, such as partial conditional random field, focus on single-label multiclass tasks with partial supervision (Mayhew et al., 2019; Effland and Collins, 2021; Li et al., 2021; Zhou et al., 2022). Recently, there have been various settings of multi-label classification tasks without fully annotated training sets: Multi-label positive unlabelled learning, which is our concern, supposing a multilabel dataset has properties by which (1) assigned labels are definitely positive and (2) some labels are absent but are still considered positive (Kanehira and Harada, 2016; Wang et al., 2022; Yuan et al., 2023); Partially observed labeling, which is also termed as Multi-label classification with missing labels supposes partial positive and negative classes are labeled (Durand et al., 2019; Zhang et al., 2022; Ben-Baruch et al., 2022; Abdelfattah et al., 2022); Partially positive observed labeling, which supposes only part of positive classes are observed with the assumption that at least one positive label per instance should be observed. Single positive labeling, which supposes one and only one positive class per instance is observed (Cole et al., 2021; Kim et al., 2022; Jouanneau et al., 2023). Distantly supervised learning, which supposes annotated samples contain both false positives and false negatives, devoted to dealing with label noise problems (Ye and Luo, 2020; Sun et al., 2023; Zeng et al., 2024).

heading: B More technology details
text: B.1 Pos Weight and Neg Weight In the “Pos Weight” method, we impose a large weight wp to the positives. We set wp as the times
of spositive targets to unlabeled targets in each training batch. Previous study (Li et al., 2020) had stated that negative sampling can be considered as a type of negative weighting method. And this work experimentally find that negative sampling even work better. In our experiments, we undersampling the unlabeled targets as the “Neg Weight” method. Unlabeled targets 10 times the number of positive targets are retained in each training batch.
B.2 Evaluation Metrics We compute the F1 scores based on TP (True Positive), FP (False Positive), and FN (False Negative).
Recall = TP/(TP + FN),
P recision = TP/(TP + FP ), F1 = 2 ∗Recall ∗ Precision Recall + Precision .
(7)
We choose the widely used evaluation metric mAP on multi-label image classification. Nc is the number of images containing class c, Precision(k, c) is the precision for class c when retrieving k best predictions and rel(k, c) is the relevance indicator function that is 1 if the class c is in the groundtruth of the image at rank k. We also compute the performance across all classes using mean average precision (mAP), where C is the number of classes.
APc = 1
Nc
N∑
k=1
Precision(k, c) ∗ rel(k, c), (8)
mAP = 1
C
∑
c
APc (9)

heading: C Data Statistics
text: Document-level Relation Extraction Given a document x containing entities Ex = {ei}|Ex|i=1 , DocRE aims to predict all possible relations between every entity pair. Each entity e ∈ Ex is mentioned at least once in x. with all its propernoun mentions denoted asMe = {mi}|Me|i=1 . Each entity pair (es, eo) can hold multiple relations, comprising a set y = Rs,o ⊂ R, where R is a pre-defined relation set. We let the set R include ϵ, which stands for no-relation. To better formulate, we denote the target of DocRE for each document as a set of multi-hot vectors representing labels of relation-existing entity pair {y(es, eo) = yso = [yso1 , ..., ysoi , ..., yso|R|], s ∈ {1, |ED|}, o ∈ {1, |ED|}}, where ysoi ∈ {0, 1} and∑|R|
i=1 yi = |Rs,o|.
In this task, we have trained our model based on the training set in the Re-DocRED dataset and validated our model by the Re-DocRED test set and DocGNRE test set. There are 3053 documents (including 59359 entities, and 85932 relations) in the Re-DocRED training set and 500 documents (including 9779 entities, and 17448 relations) in the Re-DocRED test set. DocGNRE test set provides a more accurate and complete test set with the addition of 2078 triples than ReDocRED.
To simulate partial annotation, we randomly kept a ratio of annotated relations. As mentioned in the introduction, Re-DocRED still misses some actual relation annotations. Hence, we also conduct experiments on the full training set to compare our framework with previous fully-supervised work. The size of label set C is 97 (contains <None>) in this task. Supervised learning on DocRE generally faces two challenges: i) lots of actual relation triples are not labeled by annotators because of the long context and fine-grained relation types; ii) the number of no-relation entity pairs are far larger than the number of relation-existing entity pairs, which cause an unbalanced problem.
Multi-label Image Classification Following previous work (Kanehira and Harada, 2016; BenBaruch et al., 2022), Ben-Baruch et al. (2022) (PASL), which deals with partial annotations containing both positive and negative classes, we utilize MS-COCO dataset (Lin et al., 2014) containing 80 classes. We keep the original split with 82081 training samples and 40137 test samples. We simulate the partial annotations following the operations in P-ASL. But different from them, we only retain the positive classes in their partial annotations and take all the rest of the classes as UNKNOWN. Specifically, We utilize the MS-COCO dataset to simulate the ‘Random per annotation’ scheme. We omit each annotation no matter the positive or negative label with probability p. With our setting, the retained positive labels are considered as positives and the rest are all unlabeled. We will not directly exploit unavailable negative annotations.
Binary PU Learning Setting With our formulation, an instance xi is an image, and the label of an instance in binary classification settings can be denoted as yi = [y1i , y 2 i ] where y 1 i is the label of positive and y2i is the label of negative. The prediction for each image is conducted by setting y corresponding to the higher score as 1 and the other as 0. Hence, there are 50,000 training data
and 10,000 test data as provided by the original CIFAR10. To make the training data into a partially annotated learning problem, we randomly sample a ratio of positives as annotated data and all the leaving training data as an unknown set.

heading: D Detailed Experimental Configuration
text: Document-level Relation Extraction We randomly sample three different versions of datasets and report the average results over them. Detailed scores are in Table 5 in Appendix E.1. The training hyper-parameters are the same as DREEAM (Ma et al., 2023). We only train the policy network for the last 20 epochs. It takes about 6 hours.
Multi-Label Image Classification For a fair comparison, our critic and policy networks have the same architecture as P-ASL. The training hyperparameters are the same as that in (Ben-Baruch et al., 2022). Due to the different partially annotated settings, we rerun P-ASL utilizing their codebase but with our datasets. P-ASL+Negative means training a model taking all UNKNOWN as negative classes to predict label distribution as prior. PASL+Counting means counting partially labeled positive classes as distribution prior. We also rerun EPR and ROLE methods from (Cole et al., 2021) with our datasets, utilizing their official code. We tune the hyper-parameter w between {5, 7, 12} in this task. Following previous work (Ridnik et al., 2021; Huynh and Elhamifar, 2020), we use both F1 scores and mAP as evaluation metrics in this task. Detailed methodology of the re-weight approach and the detailed formula of metric calculations can be found in Appendix B.
Binary PU Learning We consider a 13-layer CNN with ReLU as the backbone and Adam as the optimizer. Kiryo et al. (2017) designed an algorithm nnPU for PU learning with balanced binary classification data, while Su et al. (2021) proposed ImbalancednnPU considering imbalanced setting. We take these two previous state-of-the-art models as our compared baselines. We rerun nnPU and ImbalancennPU with their provided codes and configurations and report the results. The negative reward sampling is 20% for all settings. The threshold γ to choose enhancement labels is 0.8. We keep the values of other hyper-parameters the same as Su et al. (2021). Following previous work, we evaluate all the methods with F1 scores. Unless stated otherwise, the hyper-parameters specified in
our framework remain the same in the following experiments.

heading: E More experiments
text: We indeed considered two different scenarios to evaluate the robustness of our method.
Different dataset versions: To simulate partial annotation, we randomly reserve a ratio of positive classes and treat other classes as unknown. To fully verify the effectiveness and stability of our model, we randomly constructed three versions of data sets (refer to Table 5 for detailed results on DocRE). It can be seen that our method achieves consistent improvement in all the dataset versions. Different runs in the same dataset: For multi-label image classification, we repeat three runs in the same dataset. The standard deviations of F1/MAP scores of Table 10 in the paper are as follows.
E.1 Document-level Relation Extraction
Training curve. In Fig. 4, we display the reward and loss curves of our model in three annotation rations, 10%, 50%, and 100%. Our experimental settings were conducted under partially annotated multi-label tasks, but we also compute metrics on ground Truth during the experiment.
All experiments on different ratios of annotated labels To fully verify the effectiveness and robustness of our model, we randomly constructed three versions of data sets and tested the DREEAM model, Pos Weight, Neg Weight, and our MLPAC model on all data sets respectively. The results are shown in Table 6.
Experiments on selecting action sampling ratios (Take annotations ratio=50% as an example) In order to select the action sampling ratio hyperparameter, we conducted comparative experiments from 0.1 to 0.9, and finally found that the model performed best when the hyperparameter was 0.4.
The results are shown in Table 5. Critic network performance of Our MLPAC We iteratively train our critic network and policy network. After multiple rounds of iterations, the performance of the critic network has been greatly improved. The performance of the critic network of our MLPAC is shown in Table 7.
Case study. In Table 12, we show an example on the prediction of each method. Our MLPAC predicts more true positives.
E.2 Multi-label Image Classification The experimental results on extra evaluation metrics and annotation ratios. In Table 8 and Table 9, we show Precision and Recall of CIFAR10 and the results of other annotation ratios on MsCOCO. Table 10 shows the stability of our method MLPAC. The standard deviation was computed from three different runs on the MS-COCO dataset.
The experimental results on synthetic classification. According to the data construction, any category of data in CIFAR10 dataset can be chosen as the positive set. To further make our experiments convincing, we show the results of different data construction in Appendix E.2.

heading: Future Work
text: We would like to explore the potential of leveraging the label correlations to enhance our framework in future work