Noise-Robust Semi-Supervised Learning for Distantly Supervised Relation Extraction
Authors: Xin Sun
Qiang Liu
Shu Wu
Zilei Wang
Liang Wang
Venue: EMNLP 2023 Findings (2023)
Keywords: Relation Extraction
Distant Supervision
Semi-Supervised-Learning

Abstract: Distantly supervised relation extraction (DSRE) aims to extract relational facts from texts but suffers from noisy instances.
 To mitigate the influence of noisy labels, current methods typically use the Multi-Instance-Learning framework to extract relations for each bag. However, these approaches are not capable of extracting relation labels for individual sentences. Several studies have focused on sentence-level DSRE to solve the above problem. These studies primarily aim to develop methods for identifying noisy samples and filtering them out to mitigate the impact of noise. However, discarding noisy samples directly leads to the loss of useful information. To this end, we propose SSLRE, a novel  Semi-Supervised-Learning Relation Extraction framework for sentence-level DSRE. We discard only the labels of the noisy samples and utilize these instances without labels as unlabeled samples. Our SSLRE framework utilizes a weighted K-NN graph to select confident samples as labeled data and the rest as unlabeled. We then design a robust semi-supervised learning framework that can efficiently handle remaining label noise present in the labeled dataset, while also making effective use of unlabeled samples. Based on our experiments on two real-world datasets, the SSLRE framework we proposed has achieved  significant enhancements in sentence-level relation extraction performance compared to the existing state-of-the-art methods. Moreover, it has also attained a state-of-the-art level of performance in bag-level relation extraction with ONE aggregation strategy.

Noise-Robust Semi-Supervised Learning for Distantly Supervised Relation Extraction
Xin Sun 1,, Qiang Liu Shu Wu Zilei Wang', Liang Wang
!University of Science and Technology of China
CRIPAC & MAIS, Institute of Automation, Chinese Academy of Sciences University of Chinese Academy of Sciences sunxin800@mail. ustc. ustc.edu.com on, qiang. liunnlpr. ac
Abstract
of Obama born United States
Sudazelde'
Distantly supervised relation extraction (DSRE) aims to extract relational facts from texts but suffers from noisy instances. To mitigate the influence of noisy labels, current methods typically use the Multi-Instancee Learning framework to extract relations each bag. However, these approaches not capable of extracting relation labels for individual sentences. Several studies have focused on sentence-level DSRE to solve the above problem. These studies primarily aim to develop methods for identifying noisy samples and filtering them out to mitigate the impact of noise. However, discarding noisy samples directly leads to the loss of useful information. To this end, we propose SSLRE, a novel Somi-Supperiised-Leaniiin Extraction framework for sentence-level DSRE, We discard only the lahels of the noisy samples and utilize these instances without labels unlabeled samples. Our SSLRE framework utilizes weighted K-NN graph to select confident samples as labeled data and the rest as unlabeled. We then design robust semi-supervised learning framework that can efficiently handle remaining label noise present in the labeled dataset, while also making effective use of unlabeled samples. Based on experiments two real-world datasets, the SSLRE framework we proposed has achieved significant enhancements sentence-level relation extraction performance compared to the existing state-of-the-ae methods. Moreover, it has also attained state-of-the-art level of performance in bag-level relation extraction with ONE aggregation strategy.
Bug
Oboma the 44th presiomnt of the United States.
Basphyyne of
Place-ofiirh
Obemna is honsshodd name
Figure I: Bag-level RE maps bag of sentences bag labels. Sentence-level RE maps each sentence to specific relation.
techniques require extensive labeled training data, which can be difficult to obtain manually. To ad- dress this issue, Distant Supervision (DS) was pro- posed (Mintz al., 2009) automatically gener- ate labeled text corpus by aligning plain texts with knowledge bases (KB). For instance, sentence contains both the subject (s) and object (o) of a triple (s, rr,)) o) ((uubject, relation, object)), then the DS method considers (s, r, o) as valid sam- ple for that sentence. Conversely, if no relational triples apply, then the sentence is labeled as "NA". Distantly supervised datasets usually face high label noise training data due to the annotation process. mitigate the impact of noisy labels caused by distant supervision, contemporary tech- niques (Lin et al., 2016; Alt et al., 2019; Chen et al., 2021b; Li et al., 2022; Dai et al., 2022) usually em- ploy
Relation
(MIL) framework or modify MIL train the relation extraction model.
A Although MIL-baased techniques can identify bag relation labels, they are not proficient in pre- cisely mapping each sentence in a bag with explicit sentence labels (Feng et al., 201 8; Jia et al., 2019; Ma et al., 2021). Several studics have focused on improving sentence-level DSRE and have empir- ically demonstrated the inadequacy of bag-level methods on sentence-level evaluation. (Feng et al., 2018; Qin et al., 2018) apply reinforcement learn- ing to train a sample selector, (Jia et al., 2019) iden-
1 Introduction
Relation extraction (RE) is a fundamental pro- cess for constructing knowledge graphs, as aims to predict the relationship between entities based on their context. However, most supervised RE
"Corresponding author.

---

tify confident samples by frequent patterns. (Ma al., 2021) utilizes negative training to separate noisy data from the training data.
2 Related work
2.1 Distantly Supervised Relation Extraction
However, these methods has two main issues: (1) These works simply discard all noisy samples and train relation extraction model with selected sam- ples. However, filtering out noisy samples directly results in the loss of useful information. (Gao et al.. 2021a) notes that the DSRE dataset has a noise- rate exceeding 50%. Discarding all these samples would result in a significant loss of information. (2) The confident samples selection procedure is not impeccable, and there may still exist small amount of noise in the chosen confident samples. Directly training classifier in the presence of label noise is known to result in noise memorization.
Relation extraction (RE) is fundamental pro- cess for constructing knowledge graphs(Zhang et al., 2023a; Xia et al., 2022: Zhang al., 2023b). To generate large-scale auto-labeled data without hu- man effort, (Mintz et al., 2009) first use Distant Supervision to label sentences mentioning two en- tities with their relations in KGs, which inevitably brings wrongly labeled instances. To tackle the predicament of noise, most of the existing studies on DSRE are founded the multi-instance learn- ing framework. This approach is leveraged han- dle noisy sentences in each bag and train models by capitalizing on the constructed trustworthy bag- level representations. Usually, these methods em- ploy attention mechanisms to assign less weights to the probable noisy sentences in the bag(Lin et al.. 201 6; Han et al., 201 8b; Alt et al., 2019; Shen et al., 2019; Ye and Ling, 2019; Chen et al., 2021a; Li et al., 2022), apply adversarial training reinforce- ment learning to remove the noisy sentences from the bag (Zeng et al., 2015; Qin et al., 2018; Shang and Wei, 2019; Chen et al., 2020; Hao et al., 2021). However, the studies (Feng et al., 2018 8; Jia et al., 2019; Ma et al., 2021; Gao et al., 2021a) indicate that the bag-level DSRE methods are ineffective for sentence-level relation extraction.
To address the two issues, this work proposes a novel semi-supervise--leamiing framework for sentence-level DSRE, First, we construct K-NN graph for all samples using the hidden features. Then, we identify confident samples from the graph structure and consider the remaining samples as noisy. For issue (1): Our method discards only the noisy labels and treats corresponding samples as un- labeled data. We then utilize this unlabeled data by pseudo labeling within our robust supi-super learning framework to learn a better feature repre- sentation for relation. For issue (2): Despite our initial selection of confident samples, there may still be noise in the labeled dataset. we have de- veloped noise-robust semi-supervised learning framework that leverages mixup supervised con- trastive learning to learn from the labeled dataset and curriculum pscudo labeling to learn from the unlabeled dataset.
This work focuses extracting relations at the sentence level, (Feng al., 2018) applied reinforce- ment learning to identify confident instances based on the reward of noisy labels. (Jia et al., 2019) involve building initial reliable sentences based on several manually defined frequent relation pat- terns. (Ma et al., 2021) assigning complementary labels that cooperate with negative training to fil- ter out noisy instances. Unlike previous studies, our method only discard noisy labels and keep the unlabeled samples. We use pseudo labeling ef- fectively utilize unlabeled samples, which helps to learn better representation.
To summarize the contribution of this work:
We propose a noise-robust Semi- Supervised- Learning framework SSLRE for DSRE task, which effectively mitigate the impact of noisy labels.
We propose to use graph structure informa- tion (weighted K-NN) to identify the confi- dent samples and effectively convert noisy samples as useful training data by utilizing pseudo labeling.
2.2 Semi-Supervised-LLermii
In SSL, portion of training dataset is labeled and the remaining portion is unlabeled. SSL has seen great progress in recent years. Since (Bachman et al., 2014) proposed a consistency regularization based method, many approaches have migrated it into the semi-supervised learning field. MixMatch (Berthelot et al., 2019) proposes to combine con- sistency regularization with entropy minimization.
The proposed framework achieves significant improvement over previous methods in terms of both sent-level and legelevel relation ex- traction performance.

---

Mean Teacher (Tarvainen and Valpola, 2017) and Dual Student (Ke al., 2019) are also based on consistent learning, aiming for the same outputs for different networks. Recently, FixMatch (Sohn et al., 2020) provides a simple yet effective weak-to- strong consistency regularization framework. Flex- match (Zhang et al., 2021) provides curriculum pseudo pesudo labels learning to combat the imbal- ance of pseudo labels.
which consists of two main steps. Firstl, we se- leet confident samples from the distantly super- vised dataset using a weighted K-NN approach built by all sample representations. We use the selected samples as labeled data and the remain- ing samples as unlabeled data (as detailed in Sec- tion 3.1). Secondly, we employ our robust Semi- Supervised Learning framework to learn from the Semi-Supervised datasets (as described in Section 3.2). Appendix A delineates the full algorithm. Specifically, we denote the orig- inal dataset in this task as D where
Our SSLRE framework differs from these frame- works in two main ways. Firstly, our labeled dataset still contains a small amount of noise due to the fact that confident sample identification can- not achieve perfection. Therefore, we utilize mixup contrastive supervised learning to combat this noise. Secondly, current SSL methods generate and utilize pseudo labels with the same head, which causes error accumulation during the training stage. To address this issue, we propose utilizing a pscudo classifier head, which decouples the generation and utilization of pseudo labels by two parameter- independent heads to avoid error accumulation.
((1,,3). (82, I2),. (SNNDN)), Us E (1,..., is the label of the ith input sentence 8t. The labeled dataset (identified confident samples) is denoted X = {(11, (22, 92). ... (Zn: Bn)} and the unlabeled dataset (the noisy samples without labels) is denoted as U { u2. , where m + n N.
3.1 Confident Samples Identification with Weighted K-NN
2.3 Learning with Noisy Data
Our Ser-SSeeriieed-eariiiii module requires us to divide the noisy dataset into a labeled dataset and an unlabeled dataset. Inspired by (Lee et al., 2019; Bahri et al., 2020), we utilize neighborhood information of the hidden feature spaces to iden- tify confident samples We employ supervised con- trastive learning to warm up our model and obtain the representations for all instances. thy that deep neural networks tend to initially fit the training data with clean labels during an early training phase, prior to ultimately memorizing the examples with false labels (Arpit et al., 2017; Liu et al., 2020). Consequently, we only warm up our model for single epoch. Given two sentences and sj, we obtain their din-dimensional rep- resentations as 2; 0(s)) and zj 0(sj), where is the sentence encoder. We then calculate their representation similarity using the cosine distance
In both computer vision and natural language pro- cessing, learning with noisy data is widely dis- cussed problem. Existing upproaches include but not limit to estimating the noise transition ma- trix (Chen and Gupta, 2015; Goldberger and Ben- Reuven, 2016), leveraging a robust loss function (Lyu and Tsang, 2019; Ghosh et al., 2017; Liu and Guo, 2019), introducing regularization (Liu et al., 2020; Iscen et al., 2022), selecting noisy samples by multi-network learning or multi-round learning (Han et al., 2018a; Wu al., 2020), weightinng examples (Liu and Tao, 2014), generating pscudo labels (Li et al., 2020; Han et al., 2019), and so on. addition, some advanced ofate-of-the-an meth- ods combine serveral techniques, e.g., Dividemix (Li et al., 2020) and ELR+ (Liu et al., 2020).
notewor-
In this paper, address the issue of noisy la- bels in distant relation extraction. Our approach first involves constructing a K-NN graph identify confident samples based on their graph structure, and then use noise-robust mixup supervised con- trastive learning to train with the labeled samples.
2
d(z,, 22)
|
Then, we build a weighted K-NN graph for all samples based on the consine distance. To quantify the agreement between s; and ÿ,, We first use the label distribution in the K-neighborhooo to approx- imate clean posterior probabilities,
3 Methodology
(c(*)) = Enjex; d(zi,zz)
Σ BVENs
d(zs.2)) 1(ūx =
To achieve sentence-level relation extraction in DSRE, we propose a framework called SSLRE,
(2)

---

X
Φ
L
KNN
u
PD
uu,
8,
Foward
Backward
Labeled
Unlabeled instance
Hidden Testure
Mixed
Week
String
Propagation
Propagation
Instance
festure
augmented
augmented
festure
feature
Figure 2: An overview of the proposed framework, SSLRE. D, N, denote the original noisy dataset, labeled dataset and unlabeled dataset. indicates the encoder. &n and 8, mean forward with lower and higher dropout rate, respectively.. and classifier head and pseudo classifier head. L, is the mixup supervised contrastive loss defined in Eq. (12), and L is the unsupervised loss defined in Eq. (6).
where N₁ represents the of K closest neigh- bors to sj. We then use the cross-entropy loss to calculate the disagreement between (s;) and j/i- Denoting the set of confident examples belonging to the e-th class as Xe we have
sentence by applying dropout during the forward process. Additionally, we propose using a high
dropout rate for strong augmentation and a low dropout rate for weak augmentation. The sentence encoder is denoted as 6, with forward propagation using a high dropout rate denoted as 0, and forward propagation using a low dropout rate denoted Ow.
X { ((()s, Ui) < Te ct, [C],
(3)
where Yo is threshold for the e-th class, which is dynamically defined to ensure a class-balanced set identified confident examples. To achieve this goal, we use the a fractile of per-class agreements between the original label Js and max, (e(s;) across all classes to determine how many exam- ples should be selected for each class, i.c. Σn 1l (max. (cr Ti) (ū) c), [C]. Finally, we can get the labeled set and unlabeled set as
3.2.2 Unsupervised Learning with Pseudo Labeling
In this part, we propose two modules to learn from the unlabeled dataset: (1) To generate and utilize pseudo labels independently, we propose pseudo classifier head. (2) Utilize Curriculum Pseudo La- beling to perform consistent learning while combat- ing the unbalance of the generated pscudo labels.
X' = -
Pseudo classifier head: pseudo labeling is one of the prevalent techniques in semi-supervised learning. The existing approaches generate and utilize pseudo labels with the same head. How- ever, this may cause training bias, ultimately am- plifying the model's errors as self-training con- tinues. (Wang et al., 2022).
(4)
U = {si (:) x3.
3.2 Noise-Robust Semi-Supervised learning
Despite selecting confident samples from the dis- tantly supervised dataset, there still remains a small amount of noise in the labeled dataset. Naively training a classifier in the presence of label noise leads to noise memorization (Liu et al., 2020), which degrades the performance. We propose noise-robust semi-super learning framework mitigate the influence of remaining noise.
reduce this bias when using pseudo labels, we propose utilizing a two-classifier model consisting of an encoder 0 with both classifier head and pseudo clas- sifier head vb. We optimize the classifier head ø using only labeled samples and without any unre- liable pseudo labels from unlabeled samples. Un- labeled samples
3.2.1 Data Augmentation with Dropout
used solely for updating en- coder 0 and pseudo classifier head v/. particu- lar, the classifier head & generates pscudo labels (ó a 8.. (m)) for unlabeled samples (which have
Inspired by SimCSE (Gao et al., 2021b), we aug- ment training samples by embedding processing. particular, We obtain different embeddings of a

---

no gradient), the loss of unlabeled samples is cal- culated by E((v) o (u)(,, (Ø (w)))))), where l denotes cross entropy loss. This decouples the generation and utilization of pseudo labels by two parameter-independent heads to mitigate error ac- cumulation.
To make representation learning robust, we add Mixup (Berthelot al., 2019) to supervised con- trastive learning. Mixup strategies have demon- stated excellent performance in classification frameworks and have futher shown promising re- sults to prevent label noise memorization. Inspired by this success, we propose mixup supervised con- trastive learning, novel adaptation of mixup data augmentation for supervised contrastive learning. Mixup performs convex combinations pairs of samples as
Curriculum Pseudo Labeling: Due to the highly unbalanced dataset, using constant cut-off for all classes in Pseudo labeling results in almost all selected samples (those with confidence greater than the cut-off) being labeled as 'NA', which is the dominant class.
Inspired by Flexmatch (Zhang et al., 2021), we use Curriculum Pseudo Labeling (CPL) to combat unbalanced pseudo labels. We first generate pseudo labels for iteration /.
Tj X + (1 (1- (1-)).r
(10)
where E [0, 1] ~ Beta(Um:: Om)) and x; denotes the training example that combines two mini-batch examples Ta and 2z.- A linear relation in the con- trastive loss is imposed
P (Ør (u.))
(5)
These labels are then used as the target of strongly- augmented data. The unsupervised loss term has the form
L = (Z)( + (1- (1-A)(())
(11)
where Ca and Ls have the same form as C; in Eq. (9).
Cu.t
(max (px) > Ti (arg max (px)))
The supervised loss is the sum of Eq. (11) for
BB
(6)
each mixed instance:
e ((1b o (ws))
2N
where
C. = Σ MIX
(12)
(c)
T((c)=
0a
Mixup supervised contrastive learning helps to learn a robust representation for relations, but it cannot map the representation to a class. To learn the map function from the learned representation to relation class, classification learning using cross entropy loss is also employed as
and (c) represents the numbers of the samples whose predictions fall into class and above the threshold, formulated as
(c(c)) Σ 1 (max Ds > T)- 1arg max Ps- = c).
(8)
cLs
L (xx)
Σ
E((e (x.).
3.2.3 Mixup Supervised Contrastive Learning We target learning robust relation representation in the presence of remaining label noise. particular, we adopt the contrastive lcarning approach and randomly sample N sentences and inference the sentences twice with same dropout rate to get two view. Then we normalize the embedding by I.2 normalization. The resulting minibatch { ) consists of 2N normalized sentence embedding and corresponding labels. We perform supervised contrastive learning on labeled samples
(13)
3.3 Training Objective
Combining the above analyses, the total objective loss is
C- + NuLus A.CS
(14)
Experiments
4.1 Datasets
We evaluate our SSLRE framework on three DSRE datasets, including NYT10 (Riedel et al., 2010), NYT10m (Gao al., 2021a), and wiki20m (Gao et al., 2021a).
2N
L, (zi. = 2N m4
Lifs Il w=
J=1
(9)
exp (ze 23/ 23/T)
NYT10 dataset is created by aligning information from FreeBase with the New York Times (NYT)
.log
2N
L exp (zi- 2T/T)

---

corpus. However, (1) it contains many duplicated instances in the dataset; (2) there is no public vali- dataion set; (3) The distantly supervised test set quite noisy since the anotated errors. (Gao et al., 2021a) notes that 53% samples of the NYT10 test set are wrongly labeled. We only use it to per- form held-out evaluation experiment in Table 5 with some strong baselines.
ATT (Lin et al., 2016) produces bag-level repre- sentation as a weighted average over embeddings of sentences and determines weights by attention scores between sentences and relations.
For sentence-level baselines, RL-DSRE (Feng et al., 2018) apply reinforcement learning train sample selector by feedback from the manually de- signed reward function. ARNOR (Jia et al., 2019) selects the reliable instances based on reward of attention score on the selected patterns, SENT (Ma et al., 2021) filters noisy instances based on nega- tive training.
NYT1Om build a manually annotated test set for NYTI0 for more accurate evaluation. Futhermore, it removes all the duplicated instances and create a new relation ontology by merging semantically similar relations and delete the relations that only show up in train set or the test set.
4.4 Results
wiki20m is processed version of Wiki20 (Han et al., 2020), which is constructed by aligning the English Wikipedia corpus with Wikidata (Vran- dečié and Krötzsch, 2014). It has a manually anno- tated test set for evaluation.
We first evaluate our SSLRE framework in the NYT10m and WIKI20m dataset. Table 1 shows the overall performance in terms of sentence-level evaluation. From the results, we can observe that (1) Our SSLRE framework demonstrates superior performance on both datasets, surpassing all strong baseline models significantly in terms of Fl score. In comparison to the most robust baseline models in two distantly supervised datasets, SSLRE dis- plays significant enhancement in performance (i.e., +6.3% Fl and +3.4% F1). (2) The current sentence-level DSRE models (SENT, ARNOR) fail to outperform the of-t-oofthhe-r art MIL-based tech- niques in terms of F1 score on the aforementioned datasets. This could be attributed to the loss of information resulting from the elimination of sam- plcs. Unlike the MIL-based methods that employ all samples for training, these models only utilize selected samples. Moreover, the selection proce- dure may not always be reliable. (3) The perfor- mance of state-of-the-art MIL-based methods is not substantially superior to that of the Bert base- line. This suggests that the MIL modules, which are specifically crafted for this task, do not exhibit significant effectiveness when evaluated at the sen- tence level.
4.2 Evaluation and Parameter Settings
To guarantee the fairness of evaluation. We take both sentence-level evaluation and bag-level evalu- ation in our experiments. Further details of the eval- uation methods are available in the appendix C. To achieve bag-level evaluation under sentence-level training, we use at-least-one (ONE) aggregation stragy (Zeng et al., 2015), which first predicts rela- tion scores for each sentence in the bag, and then takes the highest score for each relation. The de- tails of the hyper- ssperpparameters are available in the appendix D.
4.3 Baselines
order to prove the effectiveness of the SSLRE, we compare our method with state-of-the-art meth- ods sentence-level DSRE framework and bag-level DSRE framwork.
For bag-level methods baselines, RESIDE (Vashishth et al., 2018) exploits the information of entity type and relation alias add a soft limitation for DSRE. DISTRE (Alt et al., 2019) combines the selective attention to its Transformer-baat model. CIL (Chen et al., 2021a) utilize contrastive instance learning under MIL framwork, HiCLRE (Li et al., 2022) propose hierarchical contrastive learning framwork, PARE (Rathore et al., 2022) propose concatenate all sentences in the bag to at- tend every token in the bag. Besides, we combine Bert with different aggregation strategies: ONE, which is mentioned section 4.2; AVG averages the representations of all the sentences in the bag;
Table 2 presents the results of bag-level evalu- ation of SSLRE with ONE strategy on NYT10m and WIKI20m datasets. We compared our SSLRE framework with state-of-the-an art methods for bag- level relation extraction, and found that our ap- proach outperformed all strong baselines. Specif- ically, SSLRE achieved a 5.5% improvement in AUC compared the best baseline model PARE, and a 10.1% improvement in micro Fl score com- pared with best baseline model HICLRE on the NYT10m dataset. Despite being trained without a

---

NYT10m
wiki20m
Models
preec
pRec.
jFl1
μPrrc
pRec.
JoF1
Bert-ATT
49.1
58.2
53.3
71.3
77.8
74.4
Bert-ONE
51.0
60.7
55.5
72.8
73.2
73.0
Bert-AVG
52.0
55.8
53.9
80.7
76.9
78.7
RESIDE
45.5
48.7
47.0
69.9
72.3
70.9
DISTRE
55.1
51.0
52.9
80.3
73.6
76.8
CIL
58.1
49.3
53.4
81.8
73.1
77.0
HiCLRE
54.6
60.7
57.5
81.6
74.6
77.9
PARE
53.1
58.1
55,4
77.3
74.2
75.7
RL-DSRE
48.6
57.1
52.5
69.9
74.3
72.0
ARNOR
53.2
59.1
55.9
76.8
76.9
76.8
SENT
57.2
56.3
56.7
79.8
78.2
78.9
SSLRE
57.4
72.0
63.8
81.9
81.0
81.5
Table 1: Sentence-level evaluation results on NYT10m and wiki20m. Bold and underline indicate the best and the second best scores.
NYT10m
wiki20m
Models
AUC
μF1
Macro_Fi FI
AUC
μF1
Macro_ _1
Bert-ATT
49.5
52.9
24.5
88.0
80.9
80.7
Bert-ONE
56.7
54.1
35.7
89.9
81.3
82.0
Bert-AVG
57.1
56.2
33.9
88.9
82.6
81.1
RESIDE
36.8
44.2
11.2
80.5
75.1
74.2
HiCLRE
61.0
61.2
32.0
89.1
82.3
81.1
CIL
57.4
59.6
29.4
89.3
81.8
82.4
PARE
61.1
59.8
37.2
90.3
83.2
82.6
SSLRE-OF ESLRE-ONE
66.5
71.3
36.9
91.6
83.3
84.1
Table 2: Bag-level evaluation results on nyt10m and wiki20m. SSLRE-ONE represents the SSLRE with ONE aggregation strategy
MIL framework, our SSLRE framework achieves state-of-the-art performance bag-level relation extraction. This finding suggests that sentence- level training can also yield excellent results on bag- label prediction. This observation also consistent with (Gao al., 202 la; Amin et al., 2020).On the wiki20m dataset, we note a consistent improvement on as well, although it is not as evident as in the case of NYT10m. We surmise that this could be attributed to the fact that the wiki20m dataset is relatively less noisy when compared to NYT10m
4.5 Ablation Study
We conducted ablation study experiments on the NYT10m dataset assess the effectiveness of dif- ferent modules in our SSLRE framework. We specifically removed each of the argued contribu- tions one by one to evaluate their effectiveness. For unsupervised learning part, we remove the pseudo classifier head and CPL one at time. For super- vised learning part, we switch from mixup super- vised learning to supervised contrastive learning and cross entropy of confident samples identifications methods, we alternate between using random (randomly selec- tion) and NI.I-based selection instead of our K- NN method. The NLI method involves performing zero-shot relation extraction through Natural Lan-
objective. In terms
We also compared our framework to several strong baselines using held-out evaluation on the NYT10 dataset, which is detailed appendix B.

---

guage Inference (NLI(Sainz al., 2021), then identify the confident samples based on the level of agreement between the distant label and NLI soft label.
Pre.
Rec.
20%
0.9851
0,8970
30%
0.9732
0.8758
40%
0.9531
0,8723
50%
0.9254
0.8655
Methods
Prec.
Rec.
F1
60%
0.8735
0.8172
SSLRE
57.4
72.0
63.8
Table 4: The effect of KNN.
w/o pseudo head w/o CPL
Unsupervised Learning
56.2
70.6
62.5
50.8
69.6
58.7
beled data as clean as possible after selecting confi- dent samples. Even with noisy label rate 60%. our weighted kNN method still achieves a preci- sion of 87.35% for the identified confident samples. This indicates that only 12.65% of the labeled data is noisy, which is significantly smaller than the 60% noise rate. Additionally, our weighted kNN method demonstrates recall of over 80% for con- fident samples. Although a few clean samples may not selected, they can still be utilized through techniques.
Supervised
SupCon CE
55.4
67.5
60.9
Learning
51.8
70.3
59.6
Conf-Samples
random NLI
58.6
56.0
57.3
Identification
56.4
65.3
60.5
Table 3: Ablation study of SSLRE on NYTI0m
Table. 3 shows the ablation study results. We conclude that (1) Unsupervised learning apart ef- fectively utilize the unlabeled samples. Remov- ing pseudo classifier head and CPL leads to a de- crease 1.3% and 5.1% on micro-F1, respectively. (2) When dealing with noisy labeled data in su- pervised learning, Mixup Contrastive Supervised Learning proves to be more robust than both Su- pervised Contrastive Learning (-2.9%) and Cross Entropy (-4.2%). (3) Our K-NN-based confident samples identification method outperforms the ran- dom method by 6.5% and the NLI method by 3.3%. This indicates that our K-NN method can effec- tively select confident samples.
4.7 Analysis Dropout rate
Figure 3 shows the performance of SSLRE under different dropout rate for strong augmentation. we can observe that: (1) Increasing the dropout rate appropriately improves the model's performance, with SSLRE achieving the best result (63.8 on FI) when the strong augmentation dropout rate is set 0.4. (2) Augmentation using a very high dropout rate harms the performance, as it results in a loss of a significant amount of information. However, this does not significantly degrade the performance since we only use strong augmentation to predict seeuo-llbbels which only affects L..
4.6 Analysis on KNN
We conducted an evaluation of the performance of weighted k-nearest neighbors (kNN) in terms of its ability to select confident samples. To elaborate, we intentionally corrupted the labels of instances in the nyt10m test set with random probability of 20%. 40%, and 60%. Our objective was as- sess whether our weighted kNN method could ef- fectively identify the uncorrupted (confident) in- stances. We trained our model on the corrupted nyt10m test set for 10 epochs, considering its rel- atively smaller size compared to the training set, which required more epochs to converge. In order to evaluate the ability of the weighted kNN in iden- tifying confident samples, reported the recall and precision metrics. The results are shown 4: is worth noting that precision the more im- portant metric because our goal is to make the la-
0.75
0.73
0.53
0.57
0.
0.8
Figure 3: Strong augmentation with different dropout rate

---

for semi-supervised learning and show its effective- ness through the performance results. Nonetheless, we have not conducted explicit experiments in- vestigate the interpretability, which needs further investigation.
A Acknowledgment
(a) Sup
(b) SSL.
This work is sponsored by National Natural Sci- ence Foundation of China (NSFC) (62206291, 62141608, and 62236010).
Figure 4: t-SNE visualization of representations with Pseudo (abeling(SSL) and wSth SSL achieves better cluster results comparing with Sup.
4.8 t-SNE analysis
References
To demonstrate that preserving unlabeled samples facilitate the learning of superior represen- tation compared to discarding them, we utilized sentence representations obtained from theta as the input to conduct dimension reduction via t- SNE and acquire two-dimensional representations. We focused on four primary categories of relation classes, which are "location/ocatinn/ooniiins" "/business/prrsom/compaa"" "Moaraannnnooootttoooaaaaaaiiii 'Mocaaton//ddministra tive division/ccuntry".. and "/pcople/persoonnattion ality". As depicted in Figure 4, leveraging unla- beled samples via Pseudo labeling enhances the clustering of identical relation data points and ef- fectively separates distinct classes from one another. Appendix F shows the t-SNE results all classes.
Christoph Alt, Marc Hübner, and Leonhard Hennig. 2019. Fine-tuning pre-trained transformer language models to distantly supervised relation extraction. ArXīv, abs/1906.088646.
Saadullah Amin, Katherine Ann Dunfield, Anna Vechkaeva, and Guenter Neumann. 2020. A data- driven approach for noise reduction in distantly su- pervised biomedical relation extraction. In Proceed- ings of the 19th SIGBioMed Workshop on Biomedical Language Processing, pages 187-194, Online. Asso- ciation for Computational Linguistics.
Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kan- wal. Tegan Maharaj, Asja Fischer, Aaron Courville. Yoshua Bengio, et al. 2017. closer look at mem- orization in deep networks. In International confer- ence machine learning, pages 233 233-242, PMLR.
5 Conclusion
Philip Bachman, Ouais Alsharif, and Doina Precup. 2014. Learning with pseudo-ensembles. ArXiv, abs/1412.4864.
In this paper, we propose SSLRE, a novel sentence-level framework that is grounded in Semi- Supervised Learning for the DSRE task. Our SSLRE framework employs mixup supervised con- trastive learning to tackle the noise present in se- lected samples, and leverages unlabeled samples through Pseudo Labeling, which effectively utilize the information contained within noisy samples. Experimental results demonstrate that SSLRE out- performs strong baseline methods in both sentence- level and bag-level relation extraction.
Dara Bahri, Heinrich Jiang, and Maya Gupta. 2020. Deep k-nn for noisy labels. In International Confer- Machine Learning, pages 550. PMLR.
David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. 2019. Mixmatch: A holistic approach to semi- supervised learning. ArXiv, abs/1905.02249.
Jhih-Was Wei Chen, Tsu-Jui Fu, Chen-Kang Lee, and Wei- Yun Ma. 2020. H-fnd: Hierarchical false-negative denoising for distant supervision relation extraction. In Findings.
Limitations
Tao Chen, Haizhou Shi, Siliang Tang, Zhigang Chen, Fei Wu, and Yue Ting Zhuang. 202 la. Cil: Con- trastive instance learning framework for distantly su- pervised relation extraction. ArXiv, abs/2106. 1o./5506.10855..
In order to augment textual instances, we leverage dropout during forward propagation. This necessi- tates propagating each instance twice to generate the augmented sentence embeddings. However, the demand for GPU resources higher compared to previous methods. Furthermore, we adjust the dropout rate to regulate the augmentation intensity
Tao Chen, Haochen Shi, Liyuan Liu, Siliang Tang. Jian Shao, Zhigang Chen, and Yueting Zhuang. 202 Empower distantly supervised relation extraction with collaborative adversarial training. In AAAI Con- ference on Artificial Intelligence.

---

Xinlei Chen and Abhinav Kumar Gupta. 2015, Webly supervised learning of convolutional networks. 2015 IEFE International Conference Computer Vision (ICCV), pages 1431-.
Xu Han, Zhiyuan Liu, and Maosong Sun. 2018b. Neural knowledge acquisition via mutual attention between knowledge graph and text. In AAAI Conference on Artificial Intelligence.
Qin Dai, Benjamin Heinzerling, and Kentaro Inui. 2022. Cross- booss-stitching text and knowledge graph encoders for distantly supervised relation extraction. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 6947- 6958, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Kailong Hao, Botao Yu, and Wei Hu. 2021. Know- ing false negatives: An adversarial training method for distantly supervised relation extraction. In Con- ference on Empirical Methods in Natural Language Processing.
Ahmet Iscen, Jack Valmadre, Anurag Amab, and Cordelia Schmid. 2022. Learning with neighbor con- sistency for noisy labels. 2022 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 4662-4671.
Jacob Devlin, Ming-Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Wei Jia, Dai Dai, Xinyan Xiao, and Hua Wu. 2019. Arnor: Attention regularization based noise reduc- tion for distant supervision relation classification. Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguisties, pages 1399- 1408.
Jun Feng, Minlie Huang, Li Zhao, Yang Yang, and Xiaoyan Zhu. 2018. Reinforcement learning for re- lation classification from noisy data. Proceedings of the aaai conference on artificial intelligence, vol- ume 32.
Zhanghan Ke, Daoye Wang, Qiong Yam, Jimmy S. J. Ren, and Rynson W. H. Lau. 2019. Dual student: Breaking the limits of the teacher semi-supervised learning. 2019 IEEF/CVF International Conference Computer Vision (ICCV), pages 6727-6735.
Tianyu Gao, Xu Han, Keyue Qiu, Yuzhuo Bai, Zhiyu Xie, Yankai Lin, Zhiyuan Liu, Peng Li. Maosong Sun, and Jie Zhou. 2021a. Manual evaluation mat- ters: Reviewing test protocols of distantly supervised relation extraction. Findings.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. CoRR, abs/1412.6980.
Kimin Lee, Sukmin Yun. Kibok Lee, Honglak Lee. Bo Li, and Jinwoo Shin. 2019. Robust inference via generative classifiers for handling noisy labels. In Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pages 3763-3772. PMLR.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021b. SimCSE: Simple contrastive learning of sentence embeddings. In Empirical Methods in Natural Lan- guage Processing (EMNLP).
Aritra Ghosh, Himanshu Kumar, and P. Shanti Sastry. 2017. Robust loss functions under label noise for deep neural networks. In AAAI Conference Artifi- cial Intelligence.
Dongyang Li, Taolin Zhang, Nan Hu, Chengyu Wang. and Xiaofeng He. 2022. Hiclre: A hierarchical con- trastive learning framework for distantly supervised relation extraction. In Findings.
Jacob Goldberger and Ehud Ben-Reuven. 2016. Train- ing deep netrak-networks using a noise adaptation layer. In International Conference on Learning Rep- resentations.
Junnan Li, Richard Socher, and Steven C. H. Hoi. 2020. Dividemix: Learning with noisy labels semi-supervised learning. ArXiv, abs/2062.07399.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Wai-Hung Tsang, and Masashi Sugiyama. 2018 8a. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Neural Information Processing Systems.
Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016. Neural relation extraction with selective attention over instances. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2124-2133.
Janngfan Han, Ping Luo, and Xiaogang Wang. 2019. Deep self-lcarnng from noisy labels. 2019
IEEE/CVF International Conference on Computer Vision (ICCV), pages 5137-5146.
Ruri Liu, Shasha Mo. Jianwei Niu, and Shengda Fan. 2022. CETA: A consensus enhanced training ap- proach for denoising in distantly supervised rela- tion extraction. In Proceedings of the 29th Inter- national Conference Computational Linguistics, pages 2247-2258, Gyeongju, Republic of Korea. In- ternational Committee on Computational Linguistics.
Xu Han, Tianyu Gao, Yankai Lin, Hao Peng. Yaoliang Yang, Chaojun Xiao, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2020. More data, more relations, more context and more openness: A review and out- look for relation extraction. In AACI.

---

Sheng Liu, Jonathan Niles-Weed. Narges Razavian, and Carlos Fernandez- Baaannde--Gradd... 2020. Early-Leaming reg- ularization prevents memorization of noisy labels. ArXiv, abs/2007.00155
Tao Shen, Guodong Long, Tao Shen. Tianyi Zhou, Lina Yao, Huan Huo, and Jing Jiang. 2019. Self-attention enhanced selective gate with entity-aware embedding for distantly supervised relation extraction. ArXiv, abs/1911 1189.
Tongliang Liu and Dacheng Tao. 2014. Classification with noisy labels by importance reweighting. IEFE Transactions on Pattern Analysis and Machine Intel- ligence, 38:447-461.
Kihyuk Sohn, David Berthelot, Chun-Liang Li. Zizhao Zhang. Nicholas Carlini, Ekin Dogus Cubuk, Alexey Kurakin, Han Zhang, and Colin Raffel. 2020. Fix- match: Simplifying semi-supervvsed learning with consistency and confidence. ArXiv, abs/2001.07685.
Yang Liu and Hongyi Guo. 2019. Peer loss functions: Learning from noisy labels without knowing noise rates. ArXiv, abs/1910.03231.
Antti Tarvainen and Harri Valpola. 2017. Mean teachers are better role models: targets improve semi-supervised deep learning re- sults. In NIPS
Yueming Lyu and Ivor Wai-Hum Vai-Hung Tsang. 2019. Curricu- lum loss: Robust learning and generalization against label corruption. ArXiv, abs/1905.11 10045.
consistency
Ruotian Ma, Tao Gui, Linyang Li, Qi Zhang, Xuan-Jing Huang. and Yaqian Zhou. 2021. Sent: Sentence-level distant relation extraction via negative training. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguisties and the 1 11th International Joint Conference on Natural Longuage Processing (Volume 1: Long Papers), pages 6201 6213.
Shikhar Vashishth, Rishabh Joshi, Sai Suman Prayaga, Chiranjib Bhattacharyya, and Partha Talukdar. 2018. RESIDE: Improving distantly-supervisee neural re- lation extraction using side information. In Proceed- ings of the 2018 Conference on Empirieal Methods in Natural Language Processing, pages 1257-1266, Brussels, Belgium. Association for Computational Linguisties.
Mike Mintz, Steven Bills, Rion Snow, and Dan Juraf- sky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Con- ference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 1003 1011.
Denny Vrandečić and Markus Krötzsch. 2014. Wiki- data: A free collaborative knowledge hase. Commu- nications of the ACM, 57:78-85 85.
Xudong Wang, Zhi-Li Wu, Long Lian, and Stella X. Yu. 2022. Debiased learning from naturally imbalanced pseudo-labels. 2022 IEEF/CVF Conference Com- puter Vision and Pattern Recognition (CVPR), pages 14627-14637.
Pengda Qin, Weiran Xu, and William Yang Wang. 2018. Robust distant supervision relation extraction via deep reinforcement learning. In Proceedings of the 56th Annual Meeting of the Association for Compu- lational Linguistics (Volume 1; Long Pupers), pages 2137-2147.
Pengxiang Wu, Songzhu Zheng, Mayank Goswami, Dimitris N. Metaxas, and Chao Chen. 2020. A topo- logical filter for learning with label noise. ArXiv, abs/2012.04483.
Vipul Rathore, Kartikeya Badola, Parag Singla, and Mausam 2022. PARE: A simple and strong base- line for monolingual and multilingual distantly super- vised relation extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 340-354, Dublin, Ireland. Association for Computational Lin- guistics.
Yuwei Xia, Menggi Zhang, Qiang Liu, Shu Wu, and Xiao Yu Zhang. 2022. MetaTKG: Learning evo- lutionary meta- ketalknoo for temporal knowledge graph reasoning. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 2230-7240, Abu Dhabi, United Arah Emirates. Association for Computational Lin- guistics.
Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions with- out labeled text. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 163. Springer.
Zhiquan Ye and Zhenhua Ling. 2019. Distant supervi- sion relation extraction with intra-bag and inter-bag attentions. ArXiv, abs//104.000
Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao. 2015. Distant supervision for relation extraction via piecewise convolutional neural networks, In Proceed- ings of the 2015 conference on empirical methods in natural language processing, pages 1753-1762.
Oscar Sainz, Oier Lopez de Lacalle, Gorka Labaka, An- der Barrena, and Eneko Agirre. 2021. Label verbal- ization and entailment for effective zero and shw-shot relation extraction. In Proceedings of the 202/ Con- ference on Empirical Methods in Natural Language Processing, pages 1199-1212.
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu. Jindong Wang, Manabu Okumura, and Takahiro Shi- nozaki. 2021. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. ArXiv. abs/21 10.0026..
Yu-Ming Shang and Wei Wei. 2019, Are noisy sen- tences useless for distant supervised relation extrac- tion? In AAAI Conference on Artificial Intelligence.

---

Mengqi Zhang, Yuwei Xia, Qiang Liu, Shu Wu, and Liang Wang. 2023a. Learning latent relations for temporal knowledge graph reasoning. In Proceed- ings of the st Annual Meeting the Association for Compulational Linguistics (Volume I: Long Papers), pages 12617-12631, Toronto, Canada. Association for Computational Linguistics.
1.0
0.8
0.4
PARE
Menggi Zhang, Yuwei Xia, Qiang Liu, Shu Wu, and Liang Wang. 2023b. Learning long- and short-term representations for temporal knowledge graph rea- soning. In Proceedings of the ACM Web Conference 2023, WWW "23, page 2412-2422, New York, NY, USA. Association for Computing Machinery.
HICURE
BERT ONE
0.0
BERT-AVG
0.0
0.2
0.4
0.0
0.8
1.0
Figure 5: PR-curve
A Algorithm
Algorithm provides the pseudo-code of the over- all framework.
this bag is labeled N/A. We report AUC, Micro- F1 and Macro-F1 for bag level evaluation.
The underlying encoder for sentence are imple- mented by BERT-base (Devlin et al., 2019), which generates 768 hidden units for each token's ' context- aware representation. During the training stage, we set the learning rate of the model to 2 x 10 and the batch size to 64, which was determined through grid search
B Held-out evaluation
Models
Prec.
Rec.
F1
SENT
45.3
50.1
47.5
HiCLRE
44.3
50.6
47.2
batch sizes in [16, 32, 64) and learning rates in (le-5, 2c-5, 5c-5). We train the model for epochs and use Adam (Kingma and Ba, 2014) 1) as the optimizer. The Mixup parameter Om set to 1, the classifier loss weight Ae is set to 0.2, the fractile alpha is set to 0.8, the unsupervised loss weight X is set to 1, and the CPL threshold is set to 0.95. We set the dropout rate for weak augmentation to 0.2 and the dropout rate for strong augmentation 0.4. Further analysis the strong augmentation dropout rate is presented in Section 4.7.
SSLRE(Ours)
49.2
54.3
51.6
Table 5: Held-out evaluation on NYTIO
Table 5 shows the held-out evaluation results on NYT10 dataset.
C Evaluation Settings
Sentence-level evaluation: Different from bag- level evaluation used by MIL-based model, sentence-level(or instance-level) evaluation ac- cesses model performance directly all of the individual instances in the dataset, which require the model to accurately predict the relation for each sentence. Following (Jía et al., 2019; Ma et al., 2021; Liu et al., 2022), we report micro- Precision((PPeec.). meca)-)ecally).ec.) and micro- F1(dF1) for sentence-level evaluation.
E PR-curve
We report the P-R curve on NYT10m dataset as Figure 5:
D Parameter Settings
Bag evaluation accesses the performance of bag relation label extraction. Since manually annotated data are at the sentence-level, following (Gao al., 2021a), level annotations in the following way: For each bag, one sentence in the bag has human-laheled relation, this bag is labeled with this relation; no sentence in the bag is annotated with any relation,
Bag-level evaluation:
Sup
(b) SSL
construct bag-
Figure 6: t-SNE visualization of representations with Pseudo (abeling(SSL) and wSthout(Sup). SSL achieves better cluster results comparing with Sup, especially on color green and light purple.

---

F Additional t-SNE analysis
Figure shows the t-SNE results on all classes of the sentence representation.
Algorithm 1: SSLRE Algorithm
input :Noisy Dataset D
output:
Warm up and for one epoch using
supervised contrastive learning and get 0'.
Gain features: = (x(z))
Build weighted K-NN graph using Eq. (1)
Gain X' and U using Eq. (2-4)
Reinitialize and 0
while not reach the maximum iteration do for c 1 to do
Calculate T(c) using Eq. (7)
[Determine the flexible threshold for class c.}
end for
10
+ to num nun_iters do
II
From X, draw a mini-batch
X = {1r, 36); b E (1.
From U, draw mini-batch
( {uu; b E (I....,B))
/* Learning from labeled
dataset X */
Z = Ox'( U (x // Get two augmented embedding views by dropout inference twice.
13
Calculate I,CS using Eq. (13). Mixup embedding using Eq. (10), and gain Z
14
15
Calculate L, using Eq. (11)
/* Learning from unlabeled dataset U */
17
Genarate pseudo labels using
Eq. (5).
18
Calculate unsupervised loss using
Eq. (6) with (cc).
19
Calculate the overall loss L using Eq. (14).
20
end end


Reviews:
[{"review_id": "SYdzcH8aYH", "reviewer": "Reviewer_MQ5c", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your comments and feedback. We address your concerns here:\n## 1) The comparision with the paper \"Robust neural relation extraction via multi-granularity noises reduction\", zhang et al. 2021.\nWe have not read the Zhang's paper before and we apologize for that. Zhang et al. 2021 indeed utilizes the unlabeled samples, and we will cite it and discuss it in the future version of our paper. However, our approach is completely different from zhang et al. 2021 except for the idea using unlabeled samples. \n\nThe biggest difference is that our paper is a **sentence-level training framework** aiming at solving the problem that MIL-based techniques are not proficient in precisely mapping each sentence in a bag with explicit sentence labels, while **Zhang et al. 2021 is a MIL-based method**.  Zhang et al. 2021 identifies the confident samples within each bag(by selecting samples closest to the maximum value instance in the bag), which is actually a variant of **ONE** aggregation strategy, the difference is that **ONE** selects the samples with the maximum value but their work selects more within each bag. Therefore, their approach will still be subject to the limitations of the MIL framework. Our method gets rid of the MIL techniques and utilizes weighted K-nn to identify the confident samples, and then we traing our model in a sentence-level way, which is totally different with theirs.\n\nIn terms of the semi-supervised learning, SSL framework can be  divided  into two parts, the supervised part and the unsupervised part. Our methods are completely different from their work in this two part:\n* **Supervised part**. Zhang et al. 2021 just calculates the CE loss for the select confident samples in a bag(the normal way of MIL-based techniques). **They ignore that the labeled set still remains noise**. We utilize mixupcontrastive  learning to untilize the labeled samples, which is noise-robust, which can be shown at Table 3. \n\n* **Unsupervised part**.  Zhang et al. 2021 keeps the consistency between the unlabeled instance and the perturbations(add random gaussian noise). We utilize the unlabeled instance by pseudo labeling. Besides, **our methods decouples the generation of utilization of pseudo lables by two parameter-independent head to avoid error amplifying**.  We do not update our classifier $\\phi$ with any unlabeled samples to ensure it to generate reliable pseudo labels. Their methods using just one classifier, which may cause error accumulation during the training stage.\n\nZhang et al. 2021 does not provide code link in their paper and I tried to contact with the first author but it's a non-exsit email address. We will try our best to reproduce their work and compare the results with it in the future version.\n\n## 2) The novelty of our work. \nWe argue that our methods are not all exsiting techniques. We propose \n* **Pseudo Classifier Head and corresponding Gradient Backpropagation Strategy**: The exsiting approaches(fixmatch, flexmatch) generate and utilize pseudo labels with the same head, which may cause training bias and amplifying the model's errors. To mitigate the problem, we propose utilizing a two-classifier model consisting of an encoder $\\theta$ with both a classifier head $\\phi$ and a pseudo classifier head $\\psi$. We optimize the classifier head $\\phi$ using only labeled samples and without any unreliable pseudo labels from unlabeled samples. Unlabeled samples are used solely for updating encoder $\\theta$ and pseudo classifier head $\\psi$. The details can be seen at Figure 2 and Algorithm 1. The ablation study in table 3 illustrates the superiority of our approach.\n* **MixupContrastiveLearning: we combine mixup with contrastive learning to make our supervised learning part noise-robust**. Note that our dataset is textual data, so directly mixup the instance will not applicable. We mixup the embeddings for different instances to get mixed instance. Then we contrast the mixed instance using mixup label(E.q. 11). The ablation study in Table 3 shows the advancement of MixupContrastiveLearning with the normal supervised contrastive learning.\n* **Data augmentation with Dropout and adjust agumentation strength by adjusting the dropout rate**.\nPseudo Labling usually needs a strong augmentation and a weak augmention in visual field. It is hard for textual data to adjust the agumentation strength, so we propose to adjust dropout rate to generate embeddings of difference agumentation strength. Figure 3 shows that proper-agumentation(0.4 dropout rate) helps the performance of DSRE.\n* **In addition to these components**, our framework also has its own novelty. **We are the first paper utilize semi-supervised learning for sentence-level training**. Besides, previous semi-supervised learning framework ignore the remaining noise of the labeled dataset and **we are the first paper to make the supervised part noise-robust** and our work achieves a significant enhancements on **both sentence-level and bag-level evaluation**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea is straightforward and the motivation is well explained.\n2. The ablation study is quite good, especially for the virtualization of the representations in Fig. 4.", "reasons_to_reject": "1. The idea of sufficiently using mislabeled sentences as un-labeled data is not new and a similar idea has been explored two years ago in [1].  \n2. The techniques proposed in this paper are all existing techniques. Therefore, it appears to be a technical report or a workshop paper rather than a research paper in the main conference.  I do not see a lot of technical novelties and insights in this paper.\n\n\n\n[1] Zhang, X., Liu, T., Li, P., Jia, W., & Zhao, H. (2020). Robust neural relation extraction via multi-granularity noises reduction. IEEE Transactions on Knowledge and Data Engineering, 33(9), 3297-3310.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a new method, SSLRE, for distantly supervised relation extraction to solve the problem of mislabeled instances. The method uses a few learning objectives including mix-up supervised contrastive learning and curriculum pseudo labeling. The results of three widely used datasets appear to be good.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "X2zBrSeWiW", "reviewer": "Reviewer_HoSe", "summary": "", "questions": "I’m wondering the performance of the weighted KNN graph to select confident samples. Maybe you can evaluate it with the NYT10m test set, which is manually annotated.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing valuable suggestions and acknowledging our work.  We deeply appreciate your thorough review.\n\n## 1) The performance of weighted KNN to select confident samples.\n\nWe conducted an evaluation of the performance of weighted k-nearest neighbors (kNN) in terms of its ability to select confident samples. To elaborate, we intentionally corrupted the labels of instances in the nyt10m test set with a random probability of 20%, 40%, and 60%. Our objective was to assess whether our weighted kNN method could effectively identify the uncorrupted (confident) instances. We trained our model on the corrupted nyt10m test set for 10 epochs, considering its relatively smaller size compared to the training set, which required more epochs to converge. The other hyper-parameters remains consistent with the paper. In order to evaluate the ability of the weighted kNN in identifying confident samples, we reported the recall and precision metrics. The results are as follows:\n\n|     |  Prec. |  Rec.  |\n|:---:|:------:|:------:|\n| 20% | 0.9851 | 0.8970 |\n| 30% | 0.9732 | 0.8758|\n| 40% | 0.9531 | 0.8723 |\n| 50%| 0.9254 | 0.8655 |\n| 60% | 0.8735 | 0.8172 |\n\nIt is worth noting that **precision is the more important metric because our goal is to make the labeled data as clean as possible after selecting confident samples**. Even with a noisy label rate of 60%, our weighted kNN method still achieves a precision of 87.35% for the identified confident samples. This indicates that only 12.65% of the labeled data is noisy, which is significantly smaller than the 60% noise rate. Additionally, our weighted kNN method demonstrates a recall of over 80% for confident samples. Although a few clean samples may not be selected, they can still be utilized through pseudo-labeling techniques.\n\n## 2）The novelty of our paper. \nIt's true that some components are already be proposed by previous papers. However, we also propose other original components for DSRE: such as Pseudo Classifier Head, dropout based pseudo labeling(adjust agumentation strength by adjusting the dropout rate), MixupContrastive Learning.  In addition to these components, our framework also has its own novelty. **We are the first paper utilize semi-supervised learning for sentence-level training**. Besides, previous semi-supervised learning frameworks ignore the remaining noise of the labeled dataset and we are the first paper to make the supervised part noise-robust and **our work achieves a significant enhancements on both sentence-level and bag-level evaluation**.\n\n## 3) Other issues\nWe sincerely apologize for the typos and grammatical errors. We will correct them in the future version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea of utilizing noisy samples instead of simply throwing them is novel, and the pseudo labeling method is interesting. Experiments on sentence-level datasets show significant improvements over various existing models.", "reasons_to_reject": "Several existing mechanisms are applied, such as CPL and Mixup, making the model a little ad hoc and complex.", "questions_to_authors": "I’m wondering the performance of the weighted KNN graph to select confident samples. Maybe you can evaluate it with the NYT10m test set, which is manually annotated.", "paper_topic_and_main_contributions": "This paper focuses on the sentence-level DSRE task and proposes SSLRE framework. SSLRE first selects confident samples with weighted K-NN graph to train the sentence encoder. Then, different from existing methods, SSLRE utilize unselected noisy samples to train the sentence encoder with generated pseudo labels and avoid information loss. Besides, several mechanisms are applied in order to improve performance, such as CPL and mixup.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "There appears several mistake of capital letters, for example Line\t14, 47, 222-224, etc."}, "extra_text": {}}, {"review_id": "GyATHMPKeD", "reviewer": "Reviewer_JnUq", "summary": "", "questions": "- Question A: In L257, you state that you build a “weighted” K-NN graph, but the weights/distance do not seem to be used in equation (2). Is that correct, or did I misunderstand this? Would it make a difference to compute q_c (s_i) as weighted by the cosine similarity (e.g. reduce the influence of outliers)?\n- Question B: For Table 1 - how did you arrive at sentence-level results for bag-level methods like RESIDE? In addition, these are explicitly not designed to perform well at sentence-level evaluation since they are trained on bag-level labels only, so I think the comparison is not entirely fair. (I.e. in Table 1 I would only compare to RL-DSRE, ARNOR and SENT). Also - in line 474 you claim that your approach ‘significantly’ outperforms all baselines - if so, please support this with stddev/conf interval info\n- Question C: In Table 1, micro-F1 scores for the BERT-XXX are around ~54%, but Gao et al. 2021 report around ~64 micro-F1 for sentence-level eval (Table 4 in their paper, except for BERT-ATT). What is the reason for this drop of 10 points?  Similarly for bag-level evaluation, results reported in Gao et al. for micro-F1 are at 60-62\n- Question D: In Tables 1-3, do you report results of a single run, or averaged over multiple runs? (I couldn’t find info on this in the paper, but maybe I overread something)\n- Question E: (Suggestion) If possible, could you include P/R curves in the appendix, as is common in DSRE?\n- Question F: Do you have an explanation why RESIDE performs so poorly on bag-level evaluation for NYT10m (and to a lesser degree, on Wiki20m)?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply grateful for you detailed suggestions and review. We address your concerns here:\n##  Q.A: weighted K-nn\n\nSorry for the equation mistake. **The weights are actually the cosin similarity, which means that the more similar(closer) instance in the K-nerghborhood will contribute more to the $q_c(s_i)$**. So the equation (2) should be: $$\\hat{q}\\_c(s\\_i) = \\frac{1}{\\sum\\_{s\\_k \\in \\mathcal{N}\\_i}d(z\\_i, z\\_k)} \\sum\\_{ \\boldsymbol{s}\\_k \\in \\mathcal{N}\\_i}d(z\\_i, z\\_k) \\cdot \\mathbb{1}(\\tilde{y}\\_k = c)$$\n\nThe DSRE performance comparison between weighted K-nn and vanilla K-nn on NYT10m is outlined below:\n\n|              | Prec. | Rec. | F1   |\n|:------------:|:-----:|:----:|------|\n| weighted Knn |  **57.4** | **72.0** | **63.8** |\n|  Vanilla Knn |  56.5 | 71.7 | 63.1 |\n\nWe also conducted an evaluation of the performance of weighted k-nearest neighbors (kNN) in terms of its ability to select confident samples. To elaborate, we intentionally corrupted the labels of instances in the nyt10m test set with a random probability of 20%, 40%, and 60%. Our objective was to assess whether our weighted kNN method could effectively identify the uncorrupted (confident) instances. We trained our model on the corrupted nyt10m test set for 10 epochs, considering its relatively smaller size compared to the training set, which required more epochs to converge. The other parameter remains consistent with the paper. In order to evaluate the ability of the weighted kNN in identifying confident samples, we report the recall and precision metrics. The results are as follows:\n\n|     |  Prec. |  Rec.  |\n|:---:|:------:|:------:|\n| 20% | 0.9851 | 0.8970 |\n| 30% | 0.9732 | 0.8758|\n| 40% | 0.9531 | 0.8723 |\n| 50%| 0.9254 | 0.8655 |\n| 60% | 0.8735 | 0.8172 |\n\nIt is worth noting that **precision is the more important metric because our goal is to make the labeled data as clean as possible after selecting confident samples**. Even with a noisy label rate of 60%, our weighted kNN method still achieves a precision of 87.35% for the identified confident samples. This indicates that only 12.65% of the labeled data is noisy, which is significantly smaller than the 60% noise rate. Additionally, our weighted kNN method demonstrates a recall of over 80% for confident samples. Although a few clean samples may not be selected, they can still be utilized through pseudo-labeling techniques.\n\n## Q.B \n### 1)How to arrive at sentence-level results for bag-level methods?\n\nFor all multi-instance learning methods(bag-level methods), there always be a sentence encoder to encode each sentence in the bag and a classifier scores $e_i$(or a query based score) representing how well the input sentence matches each relation. Then it aggrates all sentence representation with the guidance of $e$ to get the bag representation. We simply omit the last aggregation step during sentence-level evaluation, just using $e_i$ to get the sentence-level results for inference.\n\nFor RESIDE, it also encodes each sentence and aggregates all sentence representation using ATT aggregation method. We omit the ATT aggregation step and use the query score for each sentence to get the sentence-level results. \n\n### 2) Whether the comparision between bag level methods and sentence level methods using sentence-level evaluation is fair?\nIt is essential to emphasize that **the sentence-level labels do not have more information than bag-level labels(that specifies which bag relation label each sentence is corresponding to)**. They are the same actually.\n\nFor examples,  <London, United Kingdom> has two relation \"/location/location/contains\",  \"/location/country/capital\". Then the bag <London, United Kingdom> is assign with these two label. For sentence level training, each sentence in the bag will be assigned with these two labels(The sentence will occur several times with different label).There is an example from the NYT10m training set as follows. So bag-level labels and sentence-level labels are equivalent, **only the training methods are different**. \n* {\"text\": \"Her husband , Mr. Felgate , 45 , still lives in his native London , where he is a director and spokesman for the Carbon Trust , a publicly funded entity that is working to help the United Kingdom reduce its carbon emissions , as mandated by the Kyoto Protocol .\", \"relation\": **\"/location/location/contains\"**, \"t\": {\"pos\": [59, 65], \"id\": \"m.04jpl\", \"name\": \"London\"}, \"h\": {\"pos\": [181, 195], \"id\": \"m.07ssc\", \"name\": \"United Kingdom\"}}\n* {\"text\": \"Her husband , Mr. Felgate , 45 , still lives in his native London , where he is a director and spokesman for the Carbon Trust , a publicly funded entity that is working to help the United Kingdom reduce its carbon emissions , as mandated by the Kyoto Protocol .\", \"relation\": **\"/location/country/capital\"**, \"t\": {\"pos\": [59, 65], \"id\": \"m.04jpl\", \"name\": \"London\"}, \"h\": {\"pos\": [181, 195], \"id\": \"m.07ssc\", \"name\": \"United Kingdom\"}}\n\nIndeed, it is worth mentioning that several other studies, specifically [1] and [2], have also assessed sentence-level results for bag-level methods. Our evaluation methodology aligns with theirs.\n* [1]SENT: Sentence-level Distant Relation Extraction via Negative Training [ACL 2021]\n* [2]Manual Evaluation Matters: Reviewing Test Protocols of Distantly Supervised Relation Extraction[ACL 2021]\n\n### 3) stddev info \nSee Q.D\n## Q.C inconsistant performance with Gao et al. 2021\nWe reproduce Gao et al. 2021's results using their official repository **OpenNRE** with the same hyperparameters as section 5.1 of their paper using seed 42. However, we can not achieve the the sentence-level evaluation results their paper claims. We suspect that there are some  inconsistancy on evaluation code between the scripts they used and the official repository. We used to ask the first author of the Gao et al. 2021 of their original training script, but they lost that. There are also some github issues reporting that they can not reach the  results on the paper. To obtain more rigorous results, we report the average results(seed 1-9) and standard deviation using **OpenNRE**，the results are as follows:\n|                 | NYT10m       |              |              | Wiki20m      |              |              |\n|-----------------|--------------|--------------|--------------|--------------|--------------|--------------|\n|                 | Prec.        | Rec.         | Mircor_F1    | Prec.        | Rec.         | Micro_F1     |\n| Bert-ATT | 49.5 +- 0.46  |58.7 +- 0.55 |53.7 +- 0.52 |71.5 +- 0.61 |78.1 +- 0.45 |74.7 +-  0.51 |\n|Bert-ONE| 52.7 +- 0.95 |62.0 +- 1.12| 57.1  +- 0.96 |73.0 +- 0.49 |73.4 +- 0.28 |73.2 +- 0.33 |\n|Bert-AVG| 52.1 +- 0.31 |55.6 +- 0.32 |53.8 +- 0.31  |80.7 +- 0.28 |76.8 +- 0.25 |78.6 +-  0.27 |\n|                 | AUC          | Micro_F1     | Macro_F1     | AUC          | Micro_F1     | Macro_F1     |\n|Bert-ATT |49.7 +- 0.44 |52.9 +- 0.53 |24.6 +- 0.49| 88.1 +- 0.33 |80.9 +- 0.44|81.0 +- 0.40|\n|Bert-ONE| 57.2 +- 0.53 |54.4 +- 0.82 | 35.8+- 0.71 |89.7 +- 0.36 |81.2 +- 0.21 | 81.9 +- 0.24|\n|Bert-AVG |57.1 +- 0.66 |56.2 +- 0.47 |34.1 +- 0.56 |88.9 +- 0.23 |82.7 +- 0.26|81.2 +- 0.25|\n\n## Q.D  stddev\nSorry for not mentioning the relevant experiment protocol. We actually run all experiments of SSLRE using random seed 42. We did not run it 10 times and provide the stddev info because that the results is low-invariance and the seed has a small influence of the results. Due to this, most of the DSRE paper do not provide the seddev info either. However, to be more meticulous，we run SSLRE for ten times with random seed 1-9 and report the stddev info as follows:\n\n|                 | NYT10m       |              |              | Wiki20m      |              |              |\n|-----------------|--------------|--------------|--------------|--------------|--------------|--------------|\n|                 | Prec.        | Rec.         | Mircor_F1    | Prec.        | Rec.         | Micro_F1     |\n| SSLRE_SENT_EVAL | 57.6 +- 0.42 | 71.7 +- 0.51 | 63.8 +- 0.49 | 81.9 +- 0.30 | 81.2 +- 0.24 | 81.6 +- 0.28 |\n|                 | AUC          | Micro_F1     | Macro_F1     | AUC          | Micro_F1     | Macro_F1     |\n| SSLRE_BAG_EVAL  | 66.9 +- 0.43 | 71.6 +- 0.26 | 36.9 +- 0.45 | 91.6 +- 0.23 | 83.5 +- 0.19 | 84.0 +- 0.25 |\n\nAs can see, the invariance is small, which means our approach significantly outperforms all baselines is supportable.\n\n## Q.E P-R curve\nSure, p/r curve is common in other bag-level DSRE papers. We will report the results on the appendix. But the rebuttal can not provide pictures, we report some important p-r point (the precison when recall are 0.1,0.2...0.9) on nyt10m dataset as follows:\n\n|           |    0.1    |     0.2    |     0.3    |     0.4    |     0.5    |     0.6    |     0.7    |     0.8    |     0.9    |     AUC    |\n|:---------:|:---------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|\n| **SSLRE-ONE** |   0.8091  |   0.8506   | **0.8877** | **0.9028** | **0.9032** | **0.8877** | **0.8008** |   0.3797   |   0.1097   | **66.5** |\n|    PARE   |   0.9133  |   0.8307   |   0.7682   |   0.7273   |   0.6541   |   0.5933   |   0.5056   |    0.408   |   0.2547   |    61.1    |\n|    CLI    |   0.7895  |   0.7831   |   0.7708   |   0.7338   |   0.6563   |    0.584   |   0.5035   |   0.3436   |   0.1599   |    57.4    |\n|   HICLRE  |   0.8647  |   0.8202   |   0.7911   |   0.7632   |   0.7065   |   0.6181   |   0.5157   |   0.3939   |   0.2102   |    61.0    |\n|  RESIDE | 0.7023  |  0.6008 | 0.5560 | 0.5230 | 0.4236 | 0.3120 | 0. 2460 | 0.1350 | 0.0701 | 36.8 |\n|  Bert-ONE | **0.922** | **0.8686** |   0.7697   |    0.665   |   0.5675   |   0.5121   |   0.4659   |    0.408   |   0.2641   |    56.7    |\n|  Bert-ATT |   0.7753  |   0.7136   |   0.6779   |   0.6375   |   0.6171   |   0.5423   |   0.4521   |   0.3787   |   0.2501   |    49.5    |\n|  Bert-AVG |    0.78   |    0.74    |    0.704   |   0.6806   |   0.6375   |   0.5669   |   0.4997   | **0.4258** | **0.2957** |    57.1    |\n\n## Q.F The poorly performance of RESIDE\nCompaired to other methods, RESIDE performance relatively poorly, we suspect that it is due to the fact that they use entity types as extra information, which leads to overfitting biased heruistics of entities.\n\n## Other issues: \n* We sincerely apologize for the grammatical errors. We will correct them in the future version.\n* The **ONE** in Line 36 is explained in line 430, which first predicts relation scores for each sentence in the bag and then takes the highest score for each relation.\n* L79, The DSRE dataset are NYT10m and NYT10.\n* DISTRE is missing in the Table2: We apologize for our carelessness, and it should be included in table 2. The results is as follows:\n\n|                 | NYT10m       |              |              | Wiki20m      |              |              |\n|-----------------|--------------|--------------|--------------|--------------|--------------|--------------|\n|                 | AUC          | Micro_F1     | Macro_F1     | AUC          | Micro_F1     | Macro_F1     |\n| DISTRE| 52.1 +- 0.68 |53.1 +- 0.86 |26.6 +- 0.72| 87.1 +- 0.39 |80.9 +- 0.46|81.4 +- 0.44|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well organized and mostly easy to follow.\n- The approach addresses some interesting gaps in distantly supervised relation extraction    \n- Overall the study is an interesting contribution to the relation extraction task", "reasons_to_reject": "- Experimental protocol somewhat under-specified, especially if results are obtained across a single or multiple runs (hindering reproducibility & confidence in reported results)\n- Experimental results reported for some of the baselines are inconsistent with results report the original papers (e.g. BERT-XXX, Gao et al. 2021)", "questions_to_authors": "- Question A: In L257, you state that you build a “weighted” K-NN graph, but the weights/distance do not seem to be used in equation (2). Is that correct, or did I misunderstand this? Would it make a difference to compute q_c (s_i) as weighted by the cosine similarity (e.g. reduce the influence of outliers)?\n- Question B: For Table 1 - how did you arrive at sentence-level results for bag-level methods like RESIDE? In addition, these are explicitly not designed to perform well at sentence-level evaluation since they are trained on bag-level labels only, so I think the comparison is not entirely fair. (I.e. in Table 1 I would only compare to RL-DSRE, ARNOR and SENT). Also - in line 474 you claim that your approach ‘significantly’ outperforms all baselines - if so, please support this with stddev/conf interval info\n- Question C: In Table 1, micro-F1 scores for the BERT-XXX are around ~54%, but Gao et al. 2021 report around ~64 micro-F1 for sentence-level eval (Table 4 in their paper, except for BERT-ATT). What is the reason for this drop of 10 points?  Similarly for bag-level evaluation, results reported in Gao et al. for micro-F1 are at 60-62\n- Question D: In Tables 1-3, do you report results of a single run, or averaged over multiple runs? (I couldn’t find info on this in the paper, but maybe I overread something)\n- Question E: (Suggestion) If possible, could you include P/R curves in the appendix, as is common in DSRE?\n- Question F: Do you have an explanation why RESIDE performs so poorly on bag-level evaluation for NYT10m (and to a lesser degree, on Wiki20m)?", "paper_topic_and_main_contributions": "The paper addresses the task of distantly supervised relation extraction, and more specifically, assigning relation labels at the sentence level instead of the typically used bag-level. The authors present a semi-supervised learning approach that, in contrast to prior work, does not filter out low-confidence / noisy instances during training, but instead treats them as unlabeled instances that get assigned pseudo-labels during SSL. The approach incorporates a range of further modifications, such as data augmentation with droptout, curriculum learning and mixup supervised contrastive learning, to address the noise inherent in distantly supervised data. Experimental results on the NYT10m and wiki20m indicate that the proposed approach outperforms strong baseline models. The main contributions of this paper is hence an NLP engineering experiment.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- L014 ‘discarding’\n- L017 ‘Relation’\n- L36 ‘ONE’ is unclear/unexplained. Is it an abbreviation or an emphasis?\n- L79 ‘the DSRE dataset’ - which one - there are many datasets for DSRE, e.g. NYT10,  T-Rex, Dis-Rex\n- L192 ‘discussed’   \n- L193 ‘are not limited to’\n- L218 ‘weighted’ ?\n- L259 ‘we’\n- L291 ‘we’    \n- L425 - no sentence period, i.e. ‘… of evaluation, we …’ ?\n- L431 strategy\n- Table 2 - DISTRE seems to be missing - is there a reason for that?"}}]

Author rebuttal:

Meta-review: The paper presents an approach for making good use of the noisily labeled instances in distant supervision. The results show the effectiveness of the proposed approach. However, one major concern is that the approach appear to be presenting a collection of existing techniques, which makes the current work less attractive as an original research article. It is nevertheless worth sharing with the community.
Decision: Accept-Findings