HFMRE: Constructing Huffman Tree in Bags to Find Excellent Instances for Distantly Supervised Relation Extraction
Authors: Min Li
Cong Shao
Gang Li
Mingle Zhou
Venue: EMNLP 2023 Findings (2023)
Keywords: multi-instance learning; relation extraction; Huffman Tree; aggregation strategies
TL;DR: Constructing Huffman Tree in Bags to Find Excellent Instances for Distantly Supervised Relation Extraction

Abstract: Since the introduction of distantly supervised relation extraction methods, numerous approaches have been developed, the most representative of which is multi-instance learning (MIL). To find reliable features that are most representative of multi-instance bags, aggregation strategies such as AVG (average), ONE (at least one), and ATT (sentence-level attention) are commonly used. These strategies tend to train third-party vectors to select sentence-level features, leaving it to the third party to decide/identify what is noise, ignoring the intrinsic associations that naturally exist from sentence to sentence. In this paper, we propose the concept of circular cosine similarity, which is used to explicitly show the intrinsic associations between sentences within a bag. We also consider the previous methods to be a crude denoising process as they are interrupted and do not have a continuous noise detection procedure. Following this consideration, we implement a relation extraction framework (HFMRE) that relies on the Huffman tree, where sentences are considered as leaf nodes and circular cosine similarity are considered as node weights. HFMRE can continuously and iteratively discriminate noise and aggregated features during the construction of the Huffman tree, eventually finding an excellent instance that is representative of a bag-level feature. The experiments demonstrate the remarkable effectiveness of our method, outperforming previously advanced baselines on the popular DSRE datasets.

HFMRE: Constructing Huffman Tree in Bags to Find Excellent Instances for Distantly Supervised Relation Extraction
Min Li Cong Shao" Gang Li Mingle Zhou'
Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center (National Supercomputer Center in Jinan),
Qilu University of Technology (Shandong Academy of Sciences), Jinan, China 2 Faculty of Data Science, City University of Macau, Macau, China Limineqlu. wew.eelluuuuuuuueeeeemmmmm en, 10431 2205900stu. qlu. edu. ligêqlu. edu. cn, zhouml@qlu edu. cn
Abstract
bag lovel
Since the introduccion of distantly supervised relation extraction methods, numerous ap- proaches have been developed, the most rep- resentative of which is multi-instance learning (MIL). To find reliable features that are most representative of multi-instanne bags, aggrega- tion strategies such as AVG (average), ONE (at least one), and ATT (sentence-level atten- tion) commonly used. These strategies tend to train third-party vectors to select sentence- level features, leaving the third party to decide/identify what is noise, ignoring the in- trinsic associations that naturally exist from sentence sentence. In this paper, we pro- pose the concept of circular cosine similarity. which used explicitly show intrinsic associations between sentences within bag. We also consider the previous methods to be a crude denoising process as they are inter- rupted and do not have continuous noise de- tection procedure. Following this considera- tion, we implement relation extraction frame- work (HFMRE) that relies the Huffman tree, where sentences are considered as leaf nodes and circular cosine similarity considered as node weights. HFMRE can continuously and iteratively discriminate noise and aggregated features during the construction of the Huffman tree, eventually finding an excellent instance that is representative a bag-level feature. The experiments demonstrate the remarkable effec- tiveness of our method', outperforming previ- ously advanced baselines the popular DSRE datasets.
5
000
ATT
AVG
ONE
Figure I: Three aggregation strategies in the MIL. frame- work.
of a sentence or paragraph to identify the target relationships of entities in the text. Translating peo- ple's perceptions of the physical world into seman- tic information that a computer structured format. Traditional relation extraction methods currently rely on large-scale, high-quality datasets, which are often labor-intensive and time- consuming. To address the difficulty of obtain- ing high-quality annotated samples, Mintz et al. (2009) et al. proposed a distantly supervised rela- tion extraction algorithm. They make an important assumption: for a triad in an existing knowledge graph, is assumed that any sentence in external document repository (e.g. Wikipedia) that contains the pair of entities reflects this relation some ex- tent. Based on this assumption, the distantly super- vised algorithm can label sentences in an external document repository with relations based on an an- notated mini-knowledge graph. which is equivalent to doing automatic annotation of samples. How- ever, this assumption suffers from data annotation problems, since sentence may contain only this entity pair, but does not express the relationship in Freebase.
understand in
1 Introduction
In natural language understanding tasks, relation extraction is an important fundamental task. Its goal is to understand the contextual information
To alleviate the noise problem present in the dataset, Zeng al. (2015) al. first combined multi-instance learning with deep learning, which is now the dominant learning framework. Given a set of multi-inee instance bags with categorical labels,
Equal Contribution
Corresponding author
'The source code and data can be available at http:://gthhb..oom/unnnnng/g/gg

---

each bag contains several sentences without cate- gorical labels. The model builds multi-instance classifier by learning from multi-instance bags and applies this classifier to the prediction of unknown multi-instance bags. There are three main meth- ods used to reduce or eliminate noisy data from the bag during MIL: Average (AVG) (Gao et al., 2021), At-Last-One (ONE) (Zeng et al., 2015) and Sentence-Level Attention (ATT) (Lin al., 2016). As shown in Figure 1, the AVG is an average rep- resentation of all sentences in the bag as a feature representation of the bag. The ONE is to first select the highest confidence sentence in each bag, and then use only the highest confidence sentence as a feature of the hag for subsequent training. The ATT uses selective attention mechanism that enables the model to assign different weights sentences in a bag. using a weighted average of the sentence- level representations to produce a bag-level repre- sentation.
the relation vectors are trained, and are likely be influenced by other factors in the training pro- cess, and is largely impossible to train a perfect relation vector. We use more direct and effective way to identify the noise, called the circular cosine similarity method (see section 3.2). Simply put, it is loop that calculates the sum of the similar- ity of each sentence in the bag to other sentences. The greater the sum of the similarity of sentence, the more relevant it is to other sentences and the more likely is be a positive example, and vice versa. Then, to exploit this correlation information between sentences, we consider each sentence in the bag as a node needed to construct a Huffman tree based on the idea of Huffman trees, and use the circular cosine similarity node weights to it- eratively identify noise and aggregate sentence fea- tures by merging old nodes generate new nodes, and finally construct an excellent instance that can represent the features of the bag.
All of the above methods have some weaknesses. The AVG uses an averaging operation that treats each sentence in the bag equally and inevitably in- troduces noise into the bag-level feature representa- tion, causing a degradation of model performance. The ONE, based on the at-least-one assumption, selects the highest scoring sentence in the bag as the bag-level feature representation, avoiding the effect of noise as much as possible, but not mak- ing full use of the data in the bag, resulting a waste of data resources, The ATT uses learnable attention to selectively assign weights to sentences, and although it compensates for the shortcomings of AVG and ONE, however, we believe that this is not an optimal aggregation strategy, as it does not take into account information about the corre- lation between sentences, and each sentence exists in isolation within the bag.
As the paper uses the Huffman tree idea to solve the relation extraction problem, we name our base- line model HFMRE. By constructing Huffman trees to reduce noise, finding excellent instances within bags has reasonable interpretability: A Huff- man tree is the tree with the shortest path with weights, the higher the weight, the closer to the root node. Extending to the Huffman tree in the paper, the circular cosine similarity is used as the weight of the nodes, so that the positive instances in the bag are close to the root node and the neg- ative instances are far from it. When aggregat- ing sentence features, the negative instances are then fused one step ahead of the positive instances. continuous Since the aggregation of features updating process, as shown in Figure 2, the positive instances express more features in the root node, and the noise impact of the negative instances is significantly reduced. To validate the effective- ness of HFMRE, we evaluated our model on three mainstream DSRE datasets, including NYT1Od, NYT10m and wiki20m. The results showed that HFMRE significantly outperformed the current state-of-the-arr baseline, increasing the AUC by 3.2% and boosting the P@M to 86.4. In addition, the ablation study showed the individual contribu- tion of each module.
We hold one hypothesis the inherent asso- ciation between sentences within the same bag is more helpful in discriminating noise. Based on this opinion, a simple and effective baseline has been developed. First, the intrinsic association of the two sentences is visualized using cosine simi- larity. However, we are not the first to use cosine similarity to reduce the effect of noise, Yuan et al. (2019) et al. calculate the similarity between sen- tences and relation vectors capture the correla- tion between them, and thus reduce the number of sentence instances that are not correlated with the relation facts. This is an indirect approach, where
Overall, the main contributions of the paper are summarized as follows:
1. We propose a baseline model HEMRE for the DSRE task. It uses the idea of the Huffman

---

et al. (2021) al. propose contrastive instance learning framework that uses the initial MIL an encoder of relational triples and constraint positive pairs against negative pairs for each instance. So as to improve the performance of DSRE models in the MIL framework. (2) Reinforcement learn- ing framework. Qin et al. (2018) et al. propose deep reinforcement learning framework that be applied to any state-of-the-art relation extrac- tor. Feng et (2018) et al. define instance selec- tion as reinforcement learning problem, where the instance selector selects high-quality sentences by reinforcement learning. The relation classifier makes predictions about the sentences, providing rewards to the instance selector. (3) Hierarchical Contrastive Learning Framework. Li et al. (2022) et al. propose hierarchical contrastive learning framework for the DSRE task. It can make full use of semantic interactions within specific levels and cross levels to reduce the impact of noisy data.
FC
Scale
Figure 2: An example of nodes fusion during Huffman tree construction. The right child is the one with the
lower weight by default.
tree algorithm, which treats sentences as nodes to be selected, discovers noise and aggregates sen- tence information in the process of constructing Huffman tree, and ultimately constructs excellent instances that express bag-level features.
The concept of circular cosine similarity is proposed. Compared with the selective atten- tion mechanism, circular cosine similarity can di- rectly, quickly and effectively discriminate poten- tially noisy data in the bag, and thus effectively guide the model to selectively aggregate sentence features.
2.2 Huffmmnn Tree
The Huffman tree was first proposed by Huffman (1952), also known as an optimal binary tree, for given N leaf node with weighted values, the aim is construct minimum binary tree with weighted path lengths. The Huffman tree is a data structure that is commonly used for data compres- sion and coding. Later, Huffman trees were also used in other areas such as channel coding Yin et al.. 2021; Liu et al., 2018; Wu et al., 2012), text compression(Dath and Panicker, 2017; Bedruz and Quiros, 2015; Mantoro et al., 2017), image com- pression PresiionYYaan and Hu, 2019; Kasban and Hashima, 2019; Patel al., 2016), audio coding(Yi et al., 2019; Yan and Wang, 2011), etc.
3. The experiments demonstrate that our method outperforms previous baselines, achieving state-of- the-art results on three datasets and providing robust baseline model for the DSRE task.
2 Related Work
2.1 Distantly Supervised Relation Extraction
Distantly supervised relation extraction was pro- posed as a solution to the problem that large amounts of high-quality annotated data are diffi- cult and costly to obtain. The main learning frame- work currently available is multi-instance learn- ing, which was first applied relation extraction by Hoffmann al. (2011) al. The method was soon used in most neural models because of its effectiveness in alleviating the noise problem in the dataset. E.g., neural networks based on CNN(Zeng et al., 2015; Jiang et al., 2016), Trans- former(Al Alt et al., 2019), GNN(Vashishth et al., 2018), BERT(Christou and Tsoumakas, 2021), etc. Although MIL's pattern can greatly reduce DS noise, does represent many other useful sen- tence features in the dataset. Therefore many other Frameworks have recently been derived based on MIL. (1) Contrastive learning framework, Chen
In recent years, with the development of deep learning. the idea of Huffman trees has been in- troduced. Morin and Bengio (2005) al. was first proposed using Huffman trees for Hierarchical Softmax, and the method was subsequently widely used and developed by later gemera and Hinton, 2008; Chen et al., 2016; Mikolov et al.). With the rise of large deep neural network models, Gajjala et al. (2020) et al. train large deep neural network (DNN) models in distributed manner based on Huffman coding techniques. Due the computationally intensive and memory-intensive nature of neural networks, which makes them diffi- cult to deploy on embedded systems with limited hardware resources, Huffman coding was intro-

---

duced enable model compression. Han al. et al. apply a three-stage pipeline: pruning, trained quantization and Huffman coding. The network is first pruned by learning only the critical con- nections. Next, the weights are quantized to force weight sharing and finally, Huffman coding is used to compress the parameter files. Wang et al. (2020) et al. proposed a compression schema (HEPC) based Huffman coding, which can reduce the kernel schema number and index storage overhead of sparse neural networks with little loss of accu- racy. Today, Huffman trees have been used a variety of fields, for example, in natural language processing, Khan et al. (2020) et al. to develop fixed output length text codes using Huffman's 's al- gorithm and to construct new vocabularies based on codewords. In the field of biology, Jin et al. (2016) al. introduced Huffman coding calculate the similarity of DNA sequences.
and "#"(Wu and He, 2019). Thus, the sentence is processed before being fed the encoder as:
45 ((CLS), S.....,
We input the sentences to the encoder (BERT) to generate a context-rich semantic embedding h containing each token x, then followed the setting of Devlin et al. (2019) et al. using the final hidden state corresponding to the special token "[CLSJ" as the aggregated sequence representation of the relation extraction task.
3.2 Constructing Huffman Tree Nodes
In this section, we will prepare the leaf nodes needed for the Huffman tree algorithm. It is well known that each node to be selected Huffman tree has an information frequency (often recorded as a weight) and a name that belongs to it. We therefore make three important assumptions to make the conversion of sentences to leaf nodes:
3 Methodology
We treat each sentence in the bag as node to be selected that we need to construct a Huffman tree, so each bag eventually generates Huffman tree and then looks through the Huffman tree to find excellent instances. The Huffman tree constructed in the method has the following characteristics:
a. Assume that the order of the sentence in the bag the "name" attribute the node. b. Assume that the circular cosine similarity of the sentence is the "weight" attribute of the node. c. Further, we add new attribute "value" " to the node, and assume that the encoded the sentence is the "value" attribute of the node, which is used as a feature vector when aggregating excellent instances within the bag.
of
1) The parent nodes in a Huffman tree aggregate the features of the child nodes.
2) The root node of a Huffimnn tree aggregates the features of all sentences within the bag and is used as the bag feature for relation querying.
In conclusion, this is the conceptual definition of the nodes of the Huffman tree this study. The conceptual diagram of the nodes is shown Figure and the formula is expressed as follows:
3) A sentence with more reflected bag features (excellent instances) is closer to the root node and expresses more features in the root node.
weight/palue)
(2)
The main architecture of the HFMRE model is shown in Figure 3. There are four major steps: sen- tence encoding, constructing Huffman tree nodes, constructing Huffman trees and relation querying, The specific details described below.
The circular cosine similarity the sum the cosine similarity of a sentence to all other sentences in the bag. As shown in Figure 3, each sentence in the bag is calculated against the other sentences, just like in round-robin football game, where each take team plays each other once. Ultimately, the sum of the similarity of sentence other sen- tences as the circular cosine similarity of that sen- tence. Let bag Be(e,, e2) = {Ir, I.. II,. contains n sentences, then the circular cosine simi- larity of the jth sentence is defined as:
3.1 Sentence encoding
Specifically, we employ a bag e2( 11.1, Iin as a unit to encode sentence features, each containing sentences. For each sentence I, X x xxx in the bag, we add the special token "[CLS]" the beginning of the sentence, and the head and tail entities in the sentence are surrounded by the special symbols "S"
CCSC, Σ
(3)

---

Construct Huffmsnn
PLM Encoder
Chicaior Civiae Siuilarity Caleniation
Bie
4 (o oo ( h sl 6
ls 98.9
A 1n will
F. 12 161
Ao (eg mag
name: : order in bag (eg. 0.1....21
neight: ciretite cosine similariyy valer: f. 12.1g---22
Tree Noale
1
f
K
h
Classiferr
assumption :
Sgrosd
atzertion
noot_
relution
Figure Model architecture for HEMRE.
There is some conceptual advantage in using cir- cular cosine similarity as the weight of a Huffman tree node, which essentially reflects the overall rel- evance of sentence to other sentences. Assuming that sentence has a higher cireular cosine simi- larity than sentence B, then sentence A is a better representation of the whole bag than sentence B, because sentence is closer to the other sentences and jointly represents the bag features. Moreover, when a sentence has significantly lower circular similarity than other sentences, then there is a high probability that the sentence is noisy piece of data.
Nnow (name) (1.Ma..Na.......ooo (name)) +1
New (valuc) = (NN *W,) +bs
(5)
N₂ (vatuc) Nuew (value) No (vatue)]
Nnew (valuc)
Nnw (weight)
where a denotes the non-linear activation func- tion, ① denotes vector connection, W, denotes a trainable parameter matrix, and b, is the bias, both of which are shared globally, a is a hyper- parameter, and denotes the total number of nodes in the set.
3.3 Constructing Huffman Tree
Huffman tree, originally data structure for com- puter numerical storage, is used compress and encode data. Its construction is continuous itera- tive process, merging the two binary trees with the smallest weight each time, and the weight of the merged tree is the sum of the weight of the first two trees, so the structure of Huffman tree is bit like a pyramid, and the node at the top has the largest weight.
It is worth noting that due to the merging of nodes into a new node (subtree), the set of nodes is thus updated old nodes are deleted and new nodes are added. equation (6), the weights of the new node are the circular similarity between the sentences in the updated set of nodes, and likewise, the weights of the other nodes to be selected in the are updated the same time.
The above steps are repeated until there is only one node left in the set list, i.e. the root node of the Huffman tree, root hamelweiihtt valuu)) and the Huffman tree construction is complete. Based on the three traits of the Huffman tree in method, the root node continuously aggregates the features of the outstanding instances in the bag, the feature root root(ralude (valuc) of the root node for subse- quent relation querying. The code implementation of this process can found in the Appendix D.
In this paper, we follow the algorithmic idea of Huffman trees. The two sentences with the smallest weight (circular cosine simi- larity) are fused cach time in the set of nodes list (noder node2, nodea), where the sen- tence with the smaller weight is multiplied by reduction factor a, which is used to control the ef- feet of noise. As shown in Figure 2. The new node after fusion can be formulated as:
use

---

3.4 Relation Querying
We trained query vector 4i for each relation r; separately, as shown in Figure 3, and all the query vectors form a relation matrix, denoted as Q. The root node constructed in the above process is then denoted as K. The attention score of each query vector and root(value) is calculated based on the idea of attention, and then activated by sig- moid function, the probability pi of root(value) and each relation r; is returned. In inference, positive prediction is performed p; > threshold (0.5). The process is denoted as:
HEMRE HGC)
CIL
REDSandT
RESIDE
Peessoon
PCNN +AII
Rccal
Figure 4: PR Curve for Models on NYTIOd.
P(KIQ)= sigmoid(QK")
4.2 Evaluation Parameters and Hyper-parameters
3.5
Loss Function
We follow the previous setup and evaluation parameters that are widely used in the literature. These include P@N (P@ 100, P@200, P@ P@300): indicating the top most confident predictive pre- cisions computed by a model on a triad; P@M: indicating the average of the three P@N results above; AUC: indicating the area under the ROC curve, meaning the classifier's ability to classify positive and negative cases; M-F1: macro fl; μ-F1: micro fl, and we also plotted PR curves show the trade-off between model precision and recall.
We use a simple binary cross-entropy loss function without adding any loss term to evaluate the differ- ence between the predicted label and the gold label for each bag.
4 Experiments and Analysis
We conduct experiments on popular datasets from the DSRE task to validate the effectiveness of our method. The experimental results were analyzed in detail and ablation experiments were performed understand the key factors affecting the perfor- mance of the model.
We use bert-base-uncasee as the pre-training weights for our model HEMRE, and the detailed settings of the hyper-parameters are given in Ap- pendix A.
4.1 Datasets
4.3 Baseline Models
We evaluated our method HFMRE on three popu- lar DSRE benchmarks - NYT10d, NYT10m and Wiki20m, and the statistics for the datasets are pre- sented in Appendix C.
For the NYT1Od dataset, we compared some of the models that are representative of recent years. Including PCNN-ATT(Lin et al., 2016), (Vssissh et al., 2018), DISTRE(AI al., 2019), REDSandT(CCrrssoo and Tsoumakas, 2021), CChnC et al., 2021) and HiCLRE(Li et al., 2022). For the recently released datasets NYT10m and Wiki20m, we report the compari- son results with the bascline models BERT-ONE, BERT-AVG, and BERT-ATT from the original pa- per(Gao et al., 2021), and additionally add com- parisons with the more robust model CIL and the latest model HICLE.
NYT10d is the most popular dataset used distantly supervised relation extraction tasks from Riedel et al. (2010) et al. The dataset was distantly supervised by NYT corpus with Freebase.
NYTI0m is dataset recently published by Gao al. (2021) et al. It merges some inappropri- ate relation categories from NYTIO form a 25- category relation dataset and contains manually annotated test set.
Wiki20m is also published by Gao et al. (2021) al. Its test was derived from the supervised dataset Wiki80, and the training/validationnees set was repartitioned to ensure that no identical sen- tence instances existed between the three partitions to avoid any information leakage.
4.4 Evaluation on Distantly Supervised Set
Table 1 summarizes the experimental results of our model versus the baseline models on NYT1Od, where bold indicates the best score and underlining indicates the second best score. As can be seen

---

NYT10d
Models
AUC
P@ 100
P@ 200
P@ 300
P@ M
PCNN-ATT
34.1
73.0
68.0
67.3
69.4
RESIDE
41.5
81.8
75.4
74.3
77.2
DISTRE
42.2
68.0
67.0
65.3
66.8
REDSandT
42.4
78.8
75.0
73.0
75.3
CIL
50,8
90.1
86.1
81.8
86.0
HiCLRE
45.3
82.0
78.5
74.0
78.2
HFMRE
54.0
89.0
87.0
83.3
86.4
Table I: Experiment results distantly supervised datasets.
from the above results: (1) our model HFMRE significantly outperforms baseline models, im- proving by 3.2 AUC pts compared to the strongest
BERT-AVG achieves the best AUC result, which we suspect is caused by the different principles of "N/A" relation labeling in Wiki20m and NYT10 Wiki20m labels the relations of entity pairs out- side of its relation ontology as "N/A, and NYT10 labels entity pairs without relations as "N/A". . In other words, our model may predict the entity pair marked as "N/A" in Wiki20m as a relation other than the relation ontology. We achieved the best scores on the M-FI and μ-Fl metrics, demonstrat- ing that we are still the best performing model in terms of overall accuracy and recall.
baseline CIL, even when the CIL uses an additional loss term (MLM loss, CL loss). (2) Our model achieves the best performance all metrics except for the P @ 100 metric, which was slightly lower than CIL. (3) Our model achieves state-of-the-a new result on the NYT10d dataset.
Figure 4 reflects the overall precision-recall pro- file of our model HFMRE versus the other baseline models NYT1Od. It be observed from the curves that (1) our HEMRE model has a more con- vex curve compared to the other baseline models, enclosing almost all other curves, especially in the threshold interval of 0.2 to 0.8, showing higher pre- cision and recall results. (2) Although our curve is slightly lower than the CIL in the initial small interval, which may be the reason why our model is slightly lower than the CIL in the P@ 100 metric, it is clearly higher for us in the other intervals.
In summary, our model has been well gener- alized and is still able to guarantee performance in real scenarios with manual annotation, which means that our approach can be applied to real problems.
4.6 Ablation Study
The following ablation experiments are designed understand the factors affecting the performance the HFMRE model: (1) To verify the validity of the circular cosine similarity, instead of using the circu- lar cosine similarity to identify the in-bag noise, we identify the noise by the regular sentence-level at- tention and then construct Huffman tree based the attention score. (2) To verify the validity of the relation query, we use the conventional full connec- tion layer instead of the relation query to obtain the probability score of each relation. (3) To verify the validity of constructing Huffman trees, we perform subsequent relation querying by weighting the sen- tences in the bag according to the cosine similarity summation, such that HFMRE degenerates into common multi-instance learning framework. (4) We remove all three components mentioned above so that HFMRE degrades to plain BERT-Att, using BERT-AL Att as a base criterion for comparison. The results the experiments on the NYT1Om dataset
4.5 Evaluation on Manually Annotated Set
The test set implemented using the distant supervi- sion hypothesis suffers from data annotation prob- lems and does not accurately reflect model perfor- mance, so we further evaluated our method on test sets using manual annotation (NYT10m, Wiki20m). The results of the experiment are shown in Ta- ble (1) On NYT10m, our model achieves the best scores on the AUC (63.4) and jrF1 (62.4) met- rics, and the second best score on M-F1(35.7). We can also observe an interesting phenomenon that, in contrast the NYTI0 dataset, the HICLRE model outperforms the CIL model across the board, and we believe that the different levels of semantic inter- action and multi-granulan granularity recontextualization of the HICLRE model can play a greater role anno- tating accurate data. (2) On Wiki20m, surprisingly

---

NYT10m
Wiki20m
Models
AUC
M-F1
AUC
M-F1
JI-F1
BERT-ONE
58.1
33.9
61.9
88.9
81.1
81.6
BERT-AVG
56.7
35.7
60.4
89.9
82.0
82.7
BERT-ATT
51.2
25.8
54.1
70.9
64.3
66.8
CIL*
59.5
36.3
60.5
88.7
81.9
82.4
HiCLRE*
61.4
36.9
60.9
85.6
76.7
78.0
HFMRE
63.4
35.7
62.4
89.1
82.4
83.3
Table 2: Experiment results on human-annotated datasets. R means that we have run their model on datasets which not covered in the original paper and reported the results. We have obtained the original code of the model from the respective authors and have successfully replicated the results in their paper.
Models
AUC
Change in AUC
Encoder
AUC
P@
100
P@200
(I) CCSC
60.2
-3.2
PCNN
44.6
94.0
92.5
(2) Relation Query
57.7
-5.7
SPANBERT
61.0
93.0
91.5
Huffman Tree
58.2
-5.2
BERT
63.4
99.0
95.0
BERT-Att
51.2
-12.2
Table 4: Experimental results of different encoders on nyt10m dataset.
Table 3: Ablation experiments on the nyt10m dataset.
4.7 Case Study
4.7.1
The Influence of Sentence Encoders
are shown in Table 3.
To further understand the effectiveness of our method, conduct experiments using different encoders verify that our method is encoder- independent. Taking nyt10m as an illustration, the experimental results are shown Table 4.
From the experimental results, it noted that all four models after ablation show a significant per- formance degradation compared the complete model HFMRE. From model (1) HFMRE(-CCSC, confirm our previous speculation that the cir- cular cosine similarity method does focus more on noise than traditional selective attention, and that the intrinsic connections between sentences are of great benefit in identifying noise. Model (2) HFMRE (-Relation Query) performance dropped by 5.7 AUC pts, illustrating the effectiveness of our relation querying step, which indeed can query through the relation matrix to relation that best fits the entity pair. Model (3) HFMRE(-Huffman Tree) demonstrates that constructing Huffman tree is an essential process and that the bag-level features constructed by this process can effectively improve the performance of the model. When we ablate the model in its entirety, the performance of the model (4) BERT-Att shows huge reduc- tion (63.4 + 51.2), which again illustrates the effectiveness of the modules in approach. These modules are complementary to each other and are indispensable. We conducted the same ex- periments on the remaining datasets and the results are presented in Appendix B.
We report experimental results of the model on three different encoders and find interesting phe- nomena. Firstly, our model performs well pre- trained models (BERT, SPANBERT), enough to match even surpass existing methods. Second, on PCNN, the AUC metric of the model gets signif- icantly decreased due to the structural flaws of the PCNN model, however against the common sense the model does not decrease on the P@M (100, 200) metrics, but instead outperforms HFMRE (SPANBERT). Overall, HFMRE achieves good re- sults using different encoding methods, proving that approach is encoder-indeppendnt.
4.7.2
The Influence of Diffferent Model Combinations
We have demonstrated the effectiveness of the in- dividual components of the HFMRE model in our ablation study, and in order to gain a deeper un- derstanding of the positive effects of the individual components, we conducted experiments with dif- ferent combinations Sentence Encoder = (BERT, SPANBERT, PCNN), Weight = (Circular Cosine

---

Models
AUC
HFMRE architecture. For example, HFMRE can construct different Huffman trees for different re- lations, i.e. 53(Number of relations NYT10d dataset) Huffman trees can be constructed within a bag, and then relation querying can be performed on the 53 outstanding instances. Alternatively, pos- itive pair Huffman trees and negative pair Huff- man trees can be constructed separately to further improve model performance through contrastive learning.
(APCNN+CCSCC+uuffan Tree
44.6
(BPCIN-+ATT-Weighhed Average
56.8
(CBERT+AATT-Wagheee Average
51.2
(D)BERT+ATT+ Huffman Tree
60.2
[E]BER++CCSC+ Weighted Average
58.2
(D)BERT+CCCSC+Huffmnn Tree (OSTAANBERT-CCSC-HUUlm))) Tree
63.4
61.0
Table 5: Experimental results of different model com- binations on the nyt10m dataset.
Limitations
Similarity (CCSC), Attention Score (ATT)), and Aggregator (Weighted Average, Huffman Tree).
Although our HFMRE is able to find excellent bag- level features, there are still some limitations. (1) The complexity of our method is determined by the number of nodes needed construct a Huffman tree, so when there are too many sentences in a multi-instance bag, the complexity of the model skyrockets, which means that more GPU memory is needed. In practice, therefore, we limit the num- ber of nodes and we seleet the required Huffman nodes from multi- bag by random sam- pling. Through experimentation, we recommend that selecting 10 and less number of sentences from bag nodes is the optimal choice. Our HFMRE has only been experimented on monolin- gual datasets, and its performance on multilingual datasets has yet to be verified.
Taking nyt10m as an illustration, the experimental results are shown in Table 5.
The analysis reveals that (1) In the case of BERT encoder, both proposed components CCSC and Huffman Tree substantially improve the model performance, and using combination (C) as a baseline comparison, can be observed that the performance of combinations (D), (E), and (F) are all significantly improved. (2) Our model architecture is suitable for BERT or its variant mod- els (SPANBERT, etc.), but not suitable for PCNN. We suspect that this is due to the convolutional kernel, which messes up the overall semantics of sentence. The features encoded by PCNN are composed of many local semantics, rather than BERT encoding features from the perspective of the global semantics of the whole sentence. Be- cause of this coding characteristic of PCNN, the original position of the sentence in the vector space changes, so that the circular cosine similarity can not distinguish the noise effectively.
Acknowledgements
This work was supported by the National Key R&D Plan of China (2022YFF060800--0))
References
5 Conclusion and Outlook
Christoph Alt, Marc Hübner, and Leonhard Hennig. 2019. Fine-tuning pre-trained transformer language models distantly supervised relation extraction. Proceedings of the 57th Annual Meeting of the Asxo- ciation for Computational Linguistics, pages 1388- 1398, Florence, Italy. Association for Computational Linguistics.
In this work, we propose a relation extraction model HFMRE for distantly supervised tasks to address the shortcomings of existing aggregation strategies. Moving away from the traditional mind- set, HFMRE innovatively treats sentences as leaf nodes and continuously identifies noise and aggre- gates positive example features within the bag as the sentences are used to construct Huffman trees. Experiments have demonstrated the effectiveness of our method, as compared to existing baseline models, our HFMRE can significantly identify in- bag noise and able find an excellent instance that represents a bag-level feature. In future work, we see more possibilities for the
Rhen Anjerome Bedruz and Ana Riza Quiros. 2015. Comparison of huffman algorithm and lempel-ziv algorithm for audio, image and text com- pression. In 2015 International Conference on Humanoid, Nunotechnologs, Information Technol- ogy, Communication and Control, Environment and Management (HNICEM), pages 1-6.
Tao Chen, Haizhou Shi, Siliang Tang, Zhigang Chen, Fei Wu, and Yueting Zhuang. 2021. CIL: Contrastive instance learning framework for distantly supervised

---

relation extraction. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11 11th International Joint Confer- ence on Natural Language Processing (Volume I: Long Papers), pages 6191-6200, Online. Association for Computational L Linguistics.
Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S. Weld. 2011. Knowledge- based weak supervision for information extraction of overlapping relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguisties: Human Language Technologies, pages 541-550, Portland, Oregon, USA. Association for Computational Linguistics.
Wenlin Chen, David Grangier, and Michael Auli. 2016. Strategies for training large vocabulary neural lan- guage models. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 1975-1985, Berlin, Germany, Association for Computational Lin- guistics.
David A. Huffman. 1952. A method for the construction of minimum- reduudaann codes. Proceedings of the IRE, 40(())1098-11101
Xiaotian Jiang, Quan Wang, Peng Li, and Bin Wang. 2016. Relation extraction with multi-instance multi- label convolutional neural networks. In Proceed- ings of COLING 2016, the 26th International Con- ference on Computational Linguistics: Technical Pa- pers. pages 1471-1480, Osaka, Japan. The COLING 2016 Organizing Committee.
Despina Christou and Grigorios Tsoumakas. 2021. Improving sestantly-uuerviise relation extraction through bert-based label and instance embeddings. IEEE Access, 9:62574-62582.
Devi Dath and J. Vinitha Panicker. 2017. Enhancing adaptive huffman coding through word by word com- pression for textual data. In 2017 International Con- ference on Communication and Signal Processing (ICCSP), pages (048-105)
Xin Jin, Renean Nie, Dongming Zhou, Shaowen Yao, Yanyan Chen, Jiefu Yu, and Quan Wang. 2016. A novel dna sequence similarity calculation based on simplified pulse-coupled neural network and huffman coding. Physica A: Statistical Mechanics and its Applications, 661:325-338.
Jacob Devlin, Ming- Min Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers). pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Hany Kasban and Sherief Hashima. 2019. Adaptive radiographic image compression technique using hi- erarchical vector quantization and huffman encoding. Journal of Ambient Intelligence and Humanized Com- puting, 10:2855-2867.
Abdul Rafae Khan, Jia Xu, and Weiwei Sun. 2020. Cod- ing textual inputs boosts the accuracy of neural net- works. In Proceedings of the 2020 Conference on Empirical Methods in Natural Languuge Processing (EMNLP), pages 1350-1360, Online. Association for Computational Linguistics.
Jun Feng, Minlie Huang, Li Zhao, Yang Yang, and Xiaoyan Zhu. 2018. Reinforcement learning for re- lation classification from noisy data. In Proceedings of the aaai conference on artificial intelligence, vol- ume 32.
Dongyang Li, Taolin Zhang, Nan Hu, Chengyu Wang, and Xiaofeng He. 2022. HiCLRE: A hierarchical contrastive learning framework for distantly super- vised relation extraction. In Findings of the Asso- ciation for Computational Linguistics: ACL 2022, pages 2567-2578, Dublin, Ireland. Association for Computational Linguistics.
Rishikesh R. Gajjala, Shashwat Banchhor, Ahmed M. Abdelmoniem, Aritra Dutta, Marco Canini, and Panos Kalnis. 2020. Huffman coding based encoding techniques for fast distributed deep learning. Pro- ceedings of the Ist Workshop on Distributed Machine Learning, DistributedML'200 page 21-27, New York. NY, USA. Association for Computing Machinery.
Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016. Neural relation extraction with selective attention over instances. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2124-2133, Berlin, Germany. Association for Computational Linguistics.
Tianyu Gao, Xu Han, Yuzhuo Bai, Keyue Qiu, Zhiyu Xie, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2021. Manual evaluation mat- ters: Reviewing test protocols of distantly supervised relation extraction. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1300-1318 Online. Association for Computa- tional Linguistics.
Jianhua Liu, Wei Yang. Liusheng Huang, and Wuji Chen. 2018. A detection rrsecttinnresistanttt covert timing chan- nel based on geometric huffiman coding. In Wireless Algorithmss, Systems, and Applications, pages 308- 320, Cham. Springer International Publishing.
Song Han, Huizi Mao, and William J Dally. Deep com- pression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprini 1510.00494
Teddy Mantoro, Media A. Ayu, and Yayuk Anggraini. 2017. The performance of text file compression using

---

Saannon-fano and huffman on small mobile devices. In 2017 International Conference on Computing, En- gineering, and Design (ICCED), pages 1-5.
Jingzheng Wu, Yongji Wang. Liping Ding, and Xi- aofeng Liao. 2012. Improving performance of net- work covert timing channel through huffman coding. Mathematical and Computer Modelling, 6591999 Advanced Theory and Practice for Cryptography and Future Security.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey
Dean. Efficient estimation of word representations in vector space. arXiv preprint
Shanchan Wu and Yifan He. 2019. Enriching pre- trained language model with entity information for relation classification. In Proceedings of the 28th ACM international conference information and knowledge management, pages 2361-2364.
Mike Mintz, Steven Bills, Rion Snow, and Daniel Ju- rafsky. 2009. Distant supervision for relation ex- traction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACI. and the 4th International Joint Conference on Natural Language Processing of the AFNLP. pages 1003-1011, 1, Suntee, Singapore. Association for Com- putational Linguisties.
Diqun Yan and Rangding Wang. 2011. Huffman table swapping- -hased steganograpy for mp3 audio. Multi- media Tools and Applications, 52:291-305.
Andriy Mnih and Geoffrey E Hinton. 2008. A scalable hierarchical distributed language model. Advances neural information processing systems, 21.
Xiaowei Yi, Kun Yang. Xianfeng Zhao. Yuntao Wang. and Haibo Yu. 2019. Ahem: Adaptive huffman code mapping for audio steganography based on psychoa- coustic model. IEEE Transactions Information Forensics and Security, 14(8):2217-22311
Frederic Morin and Yoshua Bengio. 2005. Hierarchi- cal probabilistic neural network language model. In International workshop on artificial intelligence and statistics, pages 246-252.
Hoover H. E. Yin, Xishi Wang, Ka Hei Ng. Russell W. F. Lai, Lucien K. L. Ng, and Jack P. K. Ma. 2021. On multi-channel huffman codes for asymmetric- alphabet channels. 2021 IEEE International Sym- posium on Information Theory (ISIT), pages 2024- 2029.
Rachit Patel, Virendra Kumar, Vaibhav Tyagi, and Vishal Asthana. 2016. A fast and improved image compression technique using huffman coding. In 2016 International Conference on Wireless Commu- nications, Signal Processing and Networking (WISP. NET), pages 2283-2286.
Shuyun Yuan and Jianbo Hu. 2019. Research image compression technology based on huffman coding. Journal of Visual Communication and Image Repre- sentation, 59:33-38.
Pengda Qin, Weiran Xu, and William Yang Wang. 2018. Robust distant supervision relation extraction via deep reinforcement learning. Proceedings of the 56th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 2137-2147, Melbourne, Australia. Association for Computational Linguistics.
Yujin Yuan, Liyuan Liu, Siliang Tang, Zhongfei Zhang, Yueting Zhuang, Shiliang Pu, Fei Wu, and Xiang Ren. 2019. Cross-relation cross-bag attention for distantly- supervised relation extraction. In Proceedings of the AAAI conference on artificial intelligence, volume 33,
pages
Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In Proceedings of the 2010th Euro- Machine Learning and Knowl-
Daojian Zeng, Kang Liu, Yuho Chen, and Jun Zhao 2015. Distant supervision for relation extraction via piecewise convolutional neural networks. In Proceed- ings of the 2015 Conference on Empiricul Methods in Natural Language Processing, pages 1753-1762, Lisbon, Portugal. Association for Computational Lin- guistics.
pean Conference edge Discovery in Databases Volume Part III, ECMLPKDD' 10. page 148-163, 163, Berlin, Heidelberg. Beringer-Verlag,
Shikhar Vashishth, Rishabh Joshi, Sai Suman Prayaga, Chiranjib Bhattacharyya, and Partha Talukdar. 2018 RESIDE; Improving distantly-supervia
neural re- lation extraction using side information. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1257-1266. Brussels, Belgium. Association for Computational Linguisties.
Jingyu Wang, Songming Yu, Jinshan Yue, Zhe Yuan, Zhuqing Yuan. Huazhong Yang, Xueqing Li, and Yongpan Liu. 2020. High pe utilization enn ac- celerator with channel fusion supporting pattern- compressed sparse neural networks. In 2020 57th ACM/IEEE Design Automation Conference (DAC), pages 1-6.

---

A Hyper-parameter Setting and Implementation Details
Hyper-parameters
Value
hidden size
786
max_seq__en
258
epochs
3/3/5
learning rate
2e-5
weight_decay decay
le-5
train batchsize
16
test batchsize
16
Activation function
tanh
optimizer
adamw
num hum_nodes
1/6/6
Table 6:
Detailed settings for pprammerrmeters. num_ londdeeessseeeeeeeeee represents the maximum number of nodes needed to construct a Huffman tree.
Implementation details: The IDE used for the experiments in this paper is Pycharm2021 Pro- fessional Edition. the PyTorch version is 1.9.1; CUDA version is 11.6; CUDNN version 10.2.The model training and inference were performed on NVIDIA A100-SMX with 40GB of GPU memory and 16GB of CPU memory.
HEMRE takes approximately 40 minutes to train each epoch for the NYTIOd dataset, 30 minutes for the NYT1Om dataset, and 50 minutes for the Wiki20m dataset.
For parameter tuning, we take manual tun- ing approach, with tuning intervals of {8, 16, 32) for batch size, (1, 2, 15) for num_ _odes, and {le 2e 5} for learning rate.
B Ablation Study
Models(AUC)
NYT10d
NYT10m
Wiki20m
(I) CCCC
52.9(-1.1)
60.2(-3.2)
82.8(-6.3)
(2) Relation Query
50.6(-3.4)
57.7(-5.7)
80.8(-8.3)
(3) -Huffman Tree
52.8(-1.2)
58.2(-5.2)
86.7(-2.8)
(4) BERT-Att
27.8(-26.2)
51.2(-12.2)
70.9(-18.2)
Table 7: Performance of different ablation models on
AUC metrics.

---

C
Datasets Statistics
Train
Validation
Test
Datasets
#Rel
#facts
#sents
N/A
#facts
#sents
N/A
#facts
#sents
N/A
NYTIOd
18,409
522,611
74%
1,950
172,448
96%
53
NYT10m
17.137
417,893
80%
4,062
46,422
80%
3,899
9,744
32%
25
Wiki20m
157,740
698,721
59%
17,485
64,607
73%
56,000
137,986
25%
81
Table Statistics of three datasets.
D
Huffman tree Algorithm
Algorithm 1 Construct Huffman tree inside the bag
Require: HulfmanNode (name, weight, value}
Creating leaf each leaf node corresponds to a name. weight and value.
Ensure: The Huffman Tree
1: createl codoodsonds,ss, name, weight, value); nodes,
Le.. sentences within a bag-
2: PriorityQucar queue; Creating Huffman tree node queus.
3: for each node in nodes do:
4: enqueue(queue, node):
5: end for
6:
Constructing Huffman trees.
7: while size(queue) do:
8:
Remove the two nodes with the lowest weight from the queue.
9:
nodel qequeue(quenee);
10:
node2 dequeue(queuu));
11:
Create node their parents, default right
child is the smaller weight.
12:
parent createNodeenodell. node2);
13:
parent.left nodel;
14:
parent.right node2:
15:
Putting parent nodes in the queue.
16:
enqueue(queue, parent);
17: end while
18: return dequeue(queue); The node left the queue
the root node.
19: void createNode(leffrrihtt)::
node name Max(queue(i).aame I:me))
node, weight = CCSC(queue);
neatrfe leli.featureDrrihh..feuuu.


Reviews:
[{"review_id": "7Yidqi8DSd", "reviewer": "Reviewer_sGH7", "summary": "", "questions": "1) how would the conclusion look if a different base embedding (other than BERT) is used?\n\n2) In line 502, there was some discussion around Area under PRC. can the paper shed some light on the AUPRC numbers of this method?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your careful review of our paper, and any deficiencies, comments, or confusion you point out will be answered below.\n\n$\\textbf{ 1.\tTime complexity analysis }$\n\nAssuming that there are n sentences in the bag, the complexity of merging leaf nodes is o(1), then constructing a Huffman tree has to be looped n times, and the time complexity of each loop is no more than log2n, so the time complexity of constructing a Huffman tree is $nlog_{2}^{n}$.\n\nBecause the circular cosine similarity has to be calculated once after each node is merged, the first time there are n nodes, each node has to be calculated once, so the first time is n, the second time there are n-1 nodes, so it is n-1, since the total number of loops is n times, the time complexity of the circular cosine similarity is n+(n-1)+(n-2)+... ...+1=n(n+1)/2.\n\nSo the time complexity of HFMRE is $nlog_{2}^{n}+ n(n+1)/2$.\n\n$\\textbf{ 2.\tResponse to case studies on different approaches to selecting excellent examples }$\n\nWe additionally conducted experiments (using nyt10m as an example) to form a case study with the aim of comparing the quality of instance selection for different methods. The experiment is divided into two parts: one for the quality of the selected sentence-level features and one for the quality of the selected bag-level features.\n\n|  Sentence-level | AUC | μ-F1 |  Bag-level | AUC | μ-F1 |\n| :---- | ----: | :----: | :---- | ----: | :----: |\n| BERT-One | 58.1  | 61.9 | BERT-Att | 51.2 | 54.1 |\n|OUR | **60.9** | **62.1** | BERT-Avg | 56.7 | 60.4|\n| - | - | - | OUR | **63.4**  | **62.4** |\n\nAccording to the experimental results, our method outperforms previous methods both in terms of the quality of instances selected for individual sentences and the quality of bag-level features selected.\n\n$\\textbf{3.\tResponse on using different encoders (except BERT) in Q1}$\n\n|  Encoder | AUC | P@100 |  P@200 | P@300 |\n| :---- | ----: | :----: | :---- | ----: |\n| HFMRE(PCNN) | 44.6| 94.0| 92.5| 91.0|\n|HFMRE(SPANBERT)| 61.0 | 93.0 | 91.5 | 88.7 |\n| HFMRE(BERT) | 63.4| 99.0 | 95.0| 93.7  |\n\nWe report experimental results of the model on three different encoders and discover some interesting phenomena. Firstly, our model performs well on pre-trained models (BERT, SPANBERT), enough to match or even surpass existing methods. Second, on PCNN, the AUC metric of the model gets significantly decreased due to the structural flaws of the PCNN model, however, against common sense the model does not decrease on the P@M (100, 200, 300) metric, but instead outperforms HFMRE (SPANBERT).\n\n$\\textbf{4.\tResponse to the question about the PR curve graph at line 502 of the paper, mentioned in Q2}$\n\nThis paper's discussion of the PR curve at line 502 actually revolves around Figure 4, not Figure 5. We apologize that our error caused you to be confused when reading the manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) constructing the Huffman tree using circular cosine similarity seems novel; it also shows potential in the ablation study\n\n2) provides state-of-the-art results in most of the datasets.", "reasons_to_reject": "1) No time complexity results are presented for the paper. A thorough analysis of those aspects will help the community.\n\n2) As this paper is about selecting good instances from a bag. But, there were no case study presented to compare side by side different methods on their quality of selections.\n\n", "questions_to_authors": "1) how would the conclusion look if a different base embedding (other than BERT) is used?\n\n2) In line 502, there was some discussion around Area under PRC. can the paper shed some light on the AUPRC numbers of this method?", "paper_topic_and_main_contributions": "This paper presents a methodology for selecting instances for training models in relation extraction tasks.  It assumes a multi-instance learning setup and proposed weighting strategies of samples in a bag. The proposed method shows improvement over other state-of-the-art methodologies.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "\n1)  Section 3.4 The process is denoted as → incomplete sentences \n\n2) Figure 5 reflects the overall precision-recall pro503 file of our model HFMRE versus the other baseline models on NYT10d. (Figure in Appendix)\n\n3)  In Page 2,  line 170 uses the idea of the Harman tree algorithm. should it be Huffman tree?\n\n4) Table 3 caption says:\n\n\" Citation commands supported by the style file. The style is based on the natbib package and supports all natbib citation commands. It also supports commands defined in previous ACL-style files for compatibility.\"\n\n5) Table 5 is in the appendix\n\n6) citing spacing issues \n"}}, {"review_id": "E3dBJyX2Tm", "reviewer": "Reviewer_LrYw", "summary": "", "questions": "1 The author has been emphasizing the inherent association, what is the inherent association?\n2 Besides the validity of the method in the overall experimental results, are there more detailed indicators or examples to demonstrate the validity of the proposed method?\n3 Why is there no comparison with the large language model?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your careful review of our paper, and any deficiencies, comments, or confusion you point out will be answered below.\n\n$\\textbf{1.\tResponse on the inherent association mentioned in the paper}$\n\nIntrinsic association actually refers to the distribution of the positions of the sentences within the bag in the vector space. Simply put, when two sentences are more semantically similar, their positions in the vector space are closer. Conversely, the more distant the positions in the vector space. We summarise the proximity of sentences in vector space as the strength of the intrinsic connection between sentences.\nIn our paper, we further upgraded the notion of intrinsic association to the notion of circular cosine similarity by taking the sum of the distances of a sentence from all other sentences in the bag as its own circular cosine similarity. We explain its conceptual strengths and why it works in lines 352-364 of the paper.\n\n$\\textbf{2.\tResponse on the use of other indicators or experiments to demonstrate the validity of the methodology}$\n\n(1) We conducted ablation experiments in our paper to demonstrate the effectiveness of the individual modules in the methodology.\n\n(2) We performed additional experiments to compare the results of experiments using different encoders. The nyt10m dataset is used as an example.\n\n|  Encoder | AUC | P@100 |  P@200 | P@300 |\n| :---- | ----: | :----: | :---- | ----: |\n| HFMRE(PCNN) | 44.6| 94.0| 92.5| 91.0|\n|HFMRE(SPANBERT)| 61.0 | 93.0 | 91.5 | 88.7 |\n| HFMRE(BERT) | 63.4| 99.0 | 95.0| 93.7  |\n\nWe report experimental results of the model on three different encoders and discover some interesting phenomena. Firstly, our model performs well on pre-trained models (BERT, SPANBERT), enough to match or even surpass existing methods. Second, on PCNN, the AUC metric of the model gets significantly decreased due to the structural flaws of the PCNN model, however, against common sense the model does not decrease on the P@M (100, 200, 300) metric, but instead outperforms HFMRE (SPANBERT).\n\n(3) We additionally conducted experiments (using nyt10m as an example) to form a case study with the aim of comparing the quality of instance selection for different methods. The experiment is divided into two parts: one for the quality of the selected sentence-level features and one for the quality of the selected bag-level features.\n\n|  Sentence-level | AUC | μ-F1 |  Bag-level | AUC | μ-F1 |\n| :---- | ----: | :----: | :---- | ----: | :----: |\n| BERT-One | 58.1  | 61.9 | BERT-Att | 51.2 | 54.1 |\n|OUR | **60.9** | **62.1** | BERT-Avg | 56.7 | 60.4|\n| - | - | - | OUR | **63.4**  | **62.4** |\n\nAccording to the experimental results, our method outperforms previous methods both in terms of the quality of instances selected for individual sentences and the quality of bag-level features selected.\n\n(4) We conducted additional experiments to compare the impact of different model combinations on the experimental results (using the nyt10m dataset as an example).\n\n|  Encoder | AUC |\n| :---- | ----: |\n|  PCNN+CCSC+Huffman tree| 44.6 |\n|  PCNN+ATT+weighted average| 56.8|\n|  BERT+ATT+weighted average| 51.2|\n|  BERT+ATT+ Huffman tree| 60.2|\n|  BERT+CCSC+ weighted average| 58.2|\n|  BERT+CCSC+Huffman tree| **63.4**|\n|  SPANBERT+CCSC+Huffman tree| 61.0 |\n\nWe conducted experiments with different combinations of Sentence Encoder = {BERT, SPANBERT, PCNN}, Weight = {Circular Cosine（CCSC）, Attention Score (ATT)}, and Aggregator = {Weighted Average, Hoffman Tree}, and the results of the experiments are shown in the table above.\nThrough analysis, it is found that our model architecture is suitable for BERT or its variant models (SPANBERT, etc.), but not suitable for PCNN. We suspect that this is due to the convolutional kernel, which messes up the overall semantics of a sentence. The features encoded by PCNN are composed of many local semantics, rather than BERT encoding features from the perspective of the global semantics of the whole sentence. Because of this coding characteristic of PCNN, the original position of the sentence in the vector space changes, so that the Circular cosine similarity can not distinguish the noise effectively.\n\n$\\textbf{3.\tResponse on why comparisons with larger models were not made}$\n\nLarge models are a hot topic in current research, and we have two reasons for not comparing with large models:\n\n(1) Firstly, we think it is unfair to compare with large models, the methods we compare in our paper are based on models such as BERT-base, CNN, etc., which have much smaller number of parameters than large models.\n\n(2) Secondly, to the best of our knowledge, we are not aware of any large model studies on distantly supervised relation extraction. Therefore we have no baseline model to compare.\n\n$\\textbf{4.\tResponse on Soundness}$\n\nIn the Soundness metrics, it was mentioned that some of the claims in the paper not being fully supported, and since you didn't specify what experiments were missing, we added three additional experiments (shown in the response above,), and combined them with the experiments in the paper, we hope that these experiments will meet your expectations.\n\nThere is also a huge problem mentioned in the Soundness about the technique and methodology in the paper. We describe our methodology in detail in the paper with reasonable explanations, and in terms of technical implementation, we provide the source code in the supporting material.\n\n\n$\\textbf{5.\tResponse on Reproducibility}$\n\nWe state that all data used in the paper are publicly available.\n\nIn the Reproducibility indicator, you pointed out that the experimental results are difficult to reproduce, that the experimental data are not available at institutions or consortia other than the authors, and that insufficient detail was provided.\n\nThis view may be due to the fact that you have ignored the appendices and supporting materials of our paper. In the appendix of the paper, the parameter settings as well as the implementation details are explained in detail, and the download links to the experimental data and the environmental requirements are provided in the source code.\n\n\n$\\textbf{6.\tResponse on the novelty of the methodology and motivation in the paper}$\n\nThe first is the novelty of the motivation --- in the field of distantly supervised relation extraction, a large amount of mislabelled data is unavoidable, as datasets are labeled distantly against a corpus and a knowledge base. As a result, getting models to learn to recognize noise has been an unavoidable problem since the direction was proposed to the present day, and even beyond. In other words, getting the model to identify noise and removing it is the fundamental task to be addressed by distantly supervised relational extraction. Existing state-of-the-art methods, whether using reinforcement learning, comparative learning, or various multi-level, multi-granularity coding modules, ultimately aim at reducing the noise problem. Therefore, we do not feel that the claim of novelty of motivation is valid.\nThe next is the novelty of the method --- the innovation of our method lies in two points, one is that we abandon the use of third-party vectors to identify in-bag noise, and instead use the intrinsic associations, so that we do not need to train additional model parameters, and to the best of our knowledge we are the first to not use the trained vectors/matrices to guide the model in identifying the noise. Second, our method is a continuous iterative process in identifying noise and aggregating bag features, and we re-identify the data that is most likely to be noisy with every node we merge. Existing methods all perform noise identification only once and then directly aggregate bag features based on the weights assigned.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Experimental results are solid.", "reasons_to_reject": "Poor written, methods are not novel, and motivation is not very novel.", "questions_to_authors": "1 The author has been emphasizing the inherent association, what is the inherent association?\n2 Besides the validity of the method in the overall experimental results, are there more detailed indicators or examples to demonstrate the validity of the proposed method?\n3 Why is there no comparison with the large language model?", "paper_topic_and_main_contributions": "The authors propose a new architecture, HFMRE, for Distantly Supervised Relation Extraction. The authors focus on how to select the important information from the bags. Firstly, HFMRE uses the idea of the Harman tree algorithm, which treats sentences as nodes to be selected, discovers noise and aggregates sentence information. Then, using the concept of circular cosine similarity to quickly and effectively discriminate potentially noisy data in the bag. The experimental result shows that proposed method is effective.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In L170, 'it' should be changed for 'It'."}, "extra_text": {}}, {"review_id": "efH8BBRmqj", "reviewer": "Reviewer_kjoj", "summary": "", "questions": "Q1: The experiments in the paper are all conducted with BERT sentence encoder. However, the proposed method is encoder-independent. So, I’m wondering how does the HFMRE model perform if the sentence encoder is switched to PCNN. This result may help proving your hypothesis mentioned in Line 107.\n\nQ2: The two contributions of this paper focus on weight computation and bag feature aggregation in MIL. I’m hoping to see more experimental results on more complete model combinations. For example, sentence encoder={BERT, PCNN}, weight={cosine, circular cosine, attention score}, aggregator={weighted average, Huffman tree}. These results may lead to more solid contributions.\n\nQ3: In abstract you claimed that existing strategies rely on third-party vectors to determine what is noise, but when merging tree nodes HFMRE still uses a thirty-party matrix W_s, which decides how much of each input component (either noise or not) contributes to the output representation. Is this noise-deciding method still relying on third-party parameters? Would you please explain the key difference between HFMRE and existing methods from this perspective?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your careful review of our paper, and any deficiencies, comments, or confusion you point out will be answered below.\n\n$\\textbf{1.\tResponse on Q1}$\n\n|  Encoder | AUC | P@100 |  P@200 | P@300 |\n| :---- | ----: | :----: | :---- | ----: |\n| HFMRE(PCNN) | 44.6| 94.0| 92.5| 91.0|\n|HFMRE(SPANBERT)| 61.0 | 93.0 | 91.5 | 88.7 |\n| HFMRE(BERT) | 63.4| 99.0 | 95.0| 93.7  |\n\nWe report experimental results of the model on three different encoders and discover some interesting phenomena. Firstly, our model performs well on pre-trained models (BERT, SPANBERT), enough to match or even surpass existing methods. Second, on PCNN, the AUC metric of the model gets significantly decreased due to the structural flaws of the PCNN model, however, against common sense the model does not decrease on the P@M (100, 200, 300) metric, but instead outperforms HFMRE (SPANBERT).\n\n$\\textbf{2.\tResponse on Q2}$\n\n|  Combination | AUC | \n| :---- | ----: |\n| PCNN+CCSC+Huffman tree | 44.6|\n|PCNN+ATT+weighted average| 56.8 |\n| BERT+ATT+weighted average | 51.2|\n| BERT+ATT+ Huffman tree | 60.2|\n| BERT+CCSC+ weighted average | 58.2|\n| BERT+CCSC+Huffman tree | **63.4**|\n| SPANBERT+CCSC+Huffman tree | 61.0|\n\nWe conducted experiments with different combinations of Sentence Encoder = {BERT, SPANBERT, PCNN}, Weight = {Circular Cosine（CCSC）, Attention Score (ATT)}, and Aggregator = {Weighted Average, Hoffman Tree}, and the results of the experiments are shown in the table above.\n\nThrough analysis, it is found that our model architecture is suitable for BERT or its variant models (SPANBERT, etc.), but not suitable for PCNN. We suspect that this is due to the convolutional kernel, which messes up the overall semantics of a sentence. The features encoded by PCNN are composed of many local semantics, rather than BERT encoding features from the perspective of the global semantics of the whole sentence. Because of this coding characteristic of PCNN, the original position of the sentence in the vector space changes, so that the Circular cosine similarity can not distinguish the noise effectively.\n\n$\\textbf{3.\tResponse on Q3}$\n\nWe consider that existing methods rely on third-party vectors to determine what is noise, so the HFMRE model uses circular cosine similarity to determine noise, and the lower the circular cosine similarity, the more likely it is to be recognized as noise (this is explained in the manuscript at lines 352-364). The circular cosine similarity does not train any parameters and does not depend on third-party vectors.\n\nFor merging tree nodes we use a parameter matrix WS, which serves only to reduce the dimensionality of the vectors (1536 to 768). In this paper the hidden dimension of the nodes is 768, so we are going to use the matrix WS to reduce the dimension of the newly merged nodes to 768.\n\nFor determining the contribution of each input component to the output representation, we are using a hyper-parameter α to control the effect of noise.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposes the circular cosine similarity to measure the inherent associations between sentences and the Huffman tree-based method to capture bag representations. The proposed model is novel and outperforms baseline models on several benchmark datasets.", "reasons_to_reject": "Some of the important experiments are missing, such as comparison of different sentence encoder, weight computing method, and bag feature aggregator. See details in Q1 and Q2 of “Questions For The Authors”.", "questions_to_authors": "Q1: The experiments in the paper are all conducted with BERT sentence encoder. However, the proposed method is encoder-independent. So, I’m wondering how does the HFMRE model perform if the sentence encoder is switched to PCNN. This result may help proving your hypothesis mentioned in Line 107.\n\nQ2: The two contributions of this paper focus on weight computation and bag feature aggregation in MIL. I’m hoping to see more experimental results on more complete model combinations. For example, sentence encoder={BERT, PCNN}, weight={cosine, circular cosine, attention score}, aggregator={weighted average, Huffman tree}. These results may lead to more solid contributions.\n\nQ3: In abstract you claimed that existing strategies rely on third-party vectors to determine what is noise, but when merging tree nodes HFMRE still uses a thirty-party matrix W_s, which decides how much of each input component (either noise or not) contributes to the output representation. Is this noise-deciding method still relying on third-party parameters? Would you please explain the key difference between HFMRE and existing methods from this perspective?\n", "paper_topic_and_main_contributions": "This paper focuses on the Distant Supervised Relation Extraction (DSRE) task and propose a Huffman tree-based Multi-Instance Learning (MIL) method. Experiments show the remarkable effectiveness of the proposed method on several popular DSRE datasets.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 502: Figure 5 -> Figure 4\n\nLine 521: Table 5 -> Table 2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for performing distantly supervised relation extraction based on Huffman tree multi-instance learning and circular cosine similarity, which considers intrinsic associations between sentences using BERT as an encoding, and demonstrates good experimental results. The reviewers agree that the proposed method is novel and the presented results are solid. They are mainly concerned with a need for further analysis and experimentation in order to strengthen the paper. Some further results and details are presented by the authors in their response, including the use of different sentence encoders (SPANBERT and PCNN), a description of the time complexity of their method, and another baseline comparison. The paper appears to be mostly well executed as the results are robust, which the provided new experiments will supplement.
Decision: Accept-Findings