Reasoning Makes Good Annotators : An Automatic Task-specific Rules Distilling Framework for Low-resource Relation Extraction
Authors: Yilin Lu
Juncheng Li
Xiaoqiang Wang
Haochen Shi
Tao Chen
Siliang Tang
Venue: EMNLP 2023 Findings (2023)
Keywords: relation extraction
language model
rule mining and pattern mining
TL;DR: Automatic task-specific rules distilling framework based on the language model for relation extraction.

Abstract: Relation extraction is often challenged by insufficient labeled data. Previous methods exploit knowledge from unlabeled data by generating pseudo labels in a self-training pipeline, which suffers a gradual drift problem. Logic rules, a transferable and explainable form of expert knowledge, have achieved promising success by improving the model with weak labels. But manually writing comprehensive rules set is challenging and tedious. To alleviate the human labor of writing high-quality rules, in this work, we propose ARIA, an Automatic task-specific Rules distilling framework. Specifically, we guide the pre-trained language model to reason rules as experts and compose them into robust compound rules for data labeling. Besides, ARIA could continuously enrich the rules set to power the labeling ability by discovering reliable model-labeled data for distinguishable rules generation. Experiments on two public datasets demonstrate the effectiveness of ARIA in a low-resource scenario.

Reasoning Makes Good Annotators: An Automatic Task-specific Rules Distilling Framework for Low-resource Relation Extraction
Yilin Lu Zhejiang University
Juncheng Li
Xiaogiang Wang Zhejiang University xq. xq.wang @ zju.edu.cn
Zhejiang University
22121281 @zju.edu.cn
Haochen Shi Université de Montréal haochen.shi @ umontreal.ca
Tao Chen Zhejiang University ttc @ zju.eddu.cn
Siliang Tang Zhejiang University siliang @ zju.edu.cn
Abstract
Seed Data
Reliable Model-
Data Filtering
Labeled Data
Generalize
Relation extraction often challenged by insuf- ficient labeled data. Previous methods exploit knowledge from unlabeled data by generating pseudo labels in a self-traiming pipeline, which suffers gradual drift problem. Logic rules, a transferable and explainable form expert knowledge, have achieved promising success by improving the model with weak labels. But manually writing comprehensive rules set is challenging and tedious. To alleviate the human labor of writing high-quality rules, in this work, we propose ARIA, an Automatic specific Rules dlstillnng frAmework. Specifically, we guide the pre-trained language model to rea- son rules as experts and compose them into robust compound rules for data labeling. Be- sides, ARIA could continuously enrich the rules set power the labeling ability by discovering reliable podel-labeled data for distinguishable rules generation. Experiments on two public datasets demonstrate the effectiveness of ARIA in a low-resource scenario.
Input
RE
Comprehensive
Reason
Rules
Improve
Figure I: An illustration of how ARIA continuously explore PLM MLM--rasoned rules for model improvement.
human written rules (Zhou et al., 2020) are time- consuming and difficult to be completed enough for emerging domains, some work attempts to generate logic rules automatically. For example, the distant supervised methods (Riedel et al., 2010) extract the knowledge base (KB)'s facts rules for data labeling. These methods label the sentences con- taining the specific entity pair with KB's s relation label regardless of the context, which generates noise labels easily. Thus, how to handle the data's s context for accurate labeling deserves studying.
Recently, the pre-trained language model (PLM) shows broad cognitive capabilities that could be distilled into downstream tasks and work well even without any training data (Kojima et al., 2022). Specifically. some work (Wei et al., 2022; Saparov and He, 2022) propose the chain of thought prompts exploit the PLM's reason- ing ability by guiding to generate the intermedi- ate natural language reasoning process like human. The explainable reasoning process infers the asso- ciation between the input and output and could be deemed as the prerequisite for the output answer. Motivated by this, given labeled instances input. guide the PLM to imitate human relation rea- soning manner and summarize the key information supporting relation inference from the reasoning process into transferred rules automatically.
1 Introduction
Relation extraction is fundamental task in natural language processing. Training supervised models with manually annotated data is labor-intensive. This motivates methods for model learning under a low-resource setting with limited annotations.
Semi-supervised methods (French et al., 2018; Sun and Grishman, 2012) aim to explore knowl- edge from the unlabeled data for better model gen- eralization. Self-training pipeline (Rosenberg et al., 2005; Lin et al., 2019) iteratively adds the model's high-confidence predictions over unlabeled set to the training set and re- traini the model. However, the noise in the lodel-labeled data may accumulate during the training process (gradual drift problem).
We propose an automatic task-specific logic rules distilling framework, ARIA, which leverages PLM replace human for continuously high- quality labeling rules discovery. As shown in Fig- ure 1, starting from limited seed data, ARIA alter-
The logic rule is explainable and transferred form summarize knowledge, which could re- place human for weak labels generation. Since the
Juncheng Li is the corresponding author.

---

nates between Data-to-Rule stage and Rule-to-Data stage. The former guides the PLM to reason spe- cific rules from labeled data by following human reasoning paths and asks the PLM to merge them into compound rules. The latter adopts the rules over the unlabeled set to improve the RE model, which is leveraged to generalize beyond the exist- ing rules. Then we filter reliable model-labeled data for further high-quality rules generation. Dif- ferent from the previous work, which summarizes rules based on the restricted knowledge from the ex- perts or knowledge bases, ARIA could continuously explore comprehensive rules by guiding PLM to imitate human reasoning manner.
previous work, our method leverages PLM's broad knowledge rather than human knowledge or the restricted knowledge of jointly trained modules to discover data for high-quality rules generation. In summary, our contributions are four-fold:
We develop framework, ARIA, to guide the PLM to continuously summarize comprehen- sive labeling rules following different reason- ing paths as human.
2. We propose a Graph-based Data Filter that leverages PLM's knowledge to discover re- liable model-labeled data to generate distin- guishable rules and enrich the rules set for accurate labeling.
There are two major challenges to automatically generate task-specific rules for accurate labeling: 1) how to guide the PLM follow human's 's reason- ing manner and summarize crucial information of relation inference into comprehensive rules; 2) how to diversify the reasoned rules set with high-quality rules to improve the labeling ability.
ARIA achieves a great performance on two public datasets TACRED and SemEval under the low resource setting, which outperforms all the baselines and on average achieves 2.65% higher F1 than the best one.
4. We discuss ARIA's potential under ChatGPT. Embedding Roberta as a safe and efficient annotator, ARIA gets competitive labeling pre- cision as ChatGPT. We also show the small- scaled PLM's potential to assist ChatGPT for reliable reasoning: the in-context learning en- hanced by our rules' representative informa- tion could reduce ChatGPT's hallucination and improve at most 23.04% in precision.
To solve the first issue, we guide PLM to im- itate typical reasoning chains of human (e.g.. in- duction, abduction) by defining several types of meta-rule templates, which derives the key infor- mation related to the relation inference that can be used for the rules construction. For each data, to summarize rules from it, we guide the PLM's reasoning by prompts, which are built by following the ceasoning-specifee meta-rule templates, and the output reasoning words are used to build different types of rules. Since some work (Wang et al., 2022) shows reasoning in different ways helps an- swer correctly, we compose the reasoning rules into compound rules and ask PLM to pick out the most comprehensive compositions for robust labeling.
2 Related Work
Semi-Supervised Methods. With a low require- ment of labeled data, semi-supervised methods (French et al., 2018; Sun and Grishman, 2012; Li et al., 2020, 2023; Yu et al., 2023) exploit knowl- edge from the unlabeled set. Since self-training methods (Rosenberg et al., 2005) accumulate noise in pseudo labels (Hu et al., 2021b), some methods (Han et al., 2018; Lin et al., 2019) lower the noise by a joint training dual retrieval module, whose classification ability is limited by the training data' scale. Our work, instead, denoises by a Data Filter, which leverages PLM's broad cognitive capabilities to pick reliable model-labeled data for distinguish- able rules generation.
To solve the second issue, the RE model im- proved by the rule-labeled data, is used to general- ize beyond the existing rules and alleviate the grad- ual drift problem. Since the PLM's 's reasoning rules conclude the information crucial for relation infer- ence, for each relation, we pick the model-labeled data that could generate rules consistent with this relation's 's existing rules and distinguishable from other relations' by modeling their rules' relevance. Specifically, propose Graph-based Data Fil- ter, which builds graph of both model-labeled data and seed data propagate their rules' features. For each relation, the model-labeled data with fea- tures close to its seed data and far from others' are picked for further rules generation. Compared with
Weakly Supervised Methods. Logic rules are proposed to improve the model with weak labels. Since the manually written rules are expensive and difficult to be completed enough for emerging do- mains (Zhou et al., 2020), many works discover rules automatically from KB (Riedel et al., 2010;

---

Model-Labeled Data
Thaining
Rule-Labeled Data
Compound Rules Set
Add
RE model
Graph-based Data Filter
Unlabeled Data Set
GAT
Componnd Rules Compiler
Reasoning Rules Geucrator
Enumeratic
M
Filtered Data
Reasoming
Ressoning
Evaleating
Seed Data
Prompts
Componnd Rules
Prompts
Figure 2: The overall framework of ARIA. In each iteration, ARIA 1) leverages the Reasoning Rules Generator to imitate human's reasoning manner and summarize rules from the labeled data; 2) asks the Compound Rules Compiler to compose reasoning rules into robust compound rules; 3) utilizes compound rules label data for training set enrichment; 4) learns a RE model to predict on the unlabeled data and uses Graph-based Data Filter to select reliable model-labeled data for further rules generation in the next iteration.
Ye and Ling, 2019). Recently, language models has shown their ability on various tasks based on prompting (Li et al., 2022a,b, 2021; Lu et al., 2023). Since the scale of KB is limited, PRBOOST (Zhang et al., 2022) asks PLM by prompt to pre- dict the relation directly and take the predictions as rules. This rule construction manner lacks cap- turing the task-specific reasoning process, mak- ing the rules less transferred and explainable. In- stead, ARIA builds rules in fine-grained manner by guiding PLM with different reasoning paths. Besides, PRBOOST requires human for rules se- lection, while ARIA could automatically pick the data that could generate high-quality rules by mod- eling their reasoning rules' dependency.
Filter then selects the reliable model-labeled data for further rules generation. Finally, the filtered data are taken as the input for the next iteration.
Notice that step 4) we pick the model-labeled data rather than rule-labeled data for further rules discovery, which are less likely to accumulate repeating patterns in the rules set based on the model's generalization ability. For initialization. the seed data are taken as the input in the first itera- tion. Then for the iteration t 1, the input labeled data is the filtered data output from the iteration
3.1 Reasoning Rules Generator
In this section, we introduce the design of our meta- rule templates, which are instantiated by building cloze-style prompts to guide the PLM to reason specific rules under different reasoning paths.
3 Method
We introduce ARIA in this section. Given limited seed data, ARIA continuously distills specific rules by PLM for labeling over the unlabeled set and improves the task model with rule-labeled data. Overview. As shown in Figure 2, our framework iterates among the four steps: 1) From Data to Rules: Given set of labeled data, we propose Reasoning Rules Generator to guide the PLM generate specific reasoning rules of different reasoning path from each labeled data. 2) From Simple Rules to Compound Rules: A Compound Rules Compiler is proposed to compose each data's reasoning rules and ask the language model to pick the comprehensive composition as compound rules. 3) From Rules to Data: The compound rules used generate labels over the unlabeled set for further RE model improvement. 4) From Data Data: The rule-labeled data are merged with seed data to train a RE model, which is adopted to predict the unlabeled data. The Graph-based Data
Meta-Rule Templates Designing. We pro- pose four meta-rule templates build reasoning prompts from each instance, as shown in Table 1, where [input] indicates the sentence, [A]. [B] de- notes the mention of head and tail entities and [M] is the mask token. The design of the meta-rule tem- plates follows four types of reasoning in Causality, which aims to guide the reasoning of the induction, abduction, intervention and counterfactual between entities expressed in the sentence.
To be more specific, 1) Induction Meta-Rule treats one entity as premise and puts mask tokens for reasoning the conclusion related to the other entity. 2) Abduction Meta-Rule treats one entity as conclusion and aims at reasoning the premise related to the other. 3) Intervention Meta-Rule aims at reasoning the intervention brought by one entity to the other. 4) Counterfactual Meta-Rule guides to reason what will happen to one entity if the other entity does not exist.

---

Type
Meta-Rule Template
Induction
[Input]. We can infer that, [A] leads to [B]'s [M][M] while [B] leads to [A]'s [M][M]
Abduction
[Input]. We can infer that, [A] is [M] of [M] for [B], while [B] is [M] of [M] for [A].
Intervention
[Input]. We can infer that, [A] brought [M] to [B], while [B] brought [M] to [A].
Counterfactual
[Input]. We can infer that, without [A], [B] will [M][M], while without [B], [A] will [M][M].
Table I: Meta-rule templates to build reasoning prompts. [input], [A] and [B] refer to the sentence, head entity and tail entity. [M] the mask token.
Reasoning Rules Instantiation. As shown in Fig- ure 3, given an instance, to generate specific rules from it, we put its sentence and entity mentions to the slots in the meta-rulc templates to get four prompts. Then each prompt is fed to PLM respec- tively and the most likely predicted words for mask tokens are taken reasoning words. Instantiated rules are obtained by filling all the slots (including mask tokens) in meta-rule templates with specific information. Thus, each data could get four reason- ing rules, denoted as {p:)_=1- with the correspond- ing reasoning words {w)}--)
reasoning rules from an instance, contains the following components: a set of reasoning rules {pr Jics a label (same as the instance's), s), a thresh- old TH, and a similarity function g(--)) Given an unlabeled sample all similarity of the reasoning words between and ps exceeds the corresponding threshold. Formally,
is matched by Ps if the over-
1(u matched ps) = (g(ps, u) 2 TH)
(1)
g(ps, u) Σ s(ws, wi")
(2)
iE
where my denotes p.'s reasoning words. Notice we extract u's reasoning words w by following the same manner w, introduced in Sec. 3.1. Similarity Measurement. Since the reasoning words are predicted by the PLM from large vo- cabulary, the hard matching manner may lead to a low rule coverage. Thus, we design a soft matching manner, given the reasoning words of a rule and unlabeled data, we compute their embedding and take the cosine similarity as the matching score:
In Sec. 4.6, case studies are conducted show more examples of the reasoning rules.
3.2 Compound Rules Compiler
Some work (Wang et al., 2022) has shown reason- ing in different ways beneficial for finding the correct answer. Thus, for each data, we compose its four reasoning rules into robust compound rules, Composition Evaluation. As shown in Figure 3, we ask the PLM by prompt to evaluate if each composition has enough information to figure out the corresponding relation. Given a set of reason- ing rules (es) the template to build evaluating prompt for this composition is:
s s(wi, w Cos(e(w:), e(wx")
(3)
Reasoning Words Embedding. For rules or unla- beled data, we embed each word in wy by GloVe (Pennington et al., 2014) and concatenate the em- bedding as the reasoning words' embedding e(w:;) Data Labeling & Training Set Enrichment. Given unlabeled data u and compound rules set R. to create the uuleggenerated label, we go through R and measure if u matches each compound rule. When matched by multiple compound rules with confficiigg labels, the label with the largest similarity is chosen to label u and this largest similarity is taken as the labeling confidence.
[Input]. {pt Jies Question: Can we infer [A]
[Relation] [B]? Answer: [M] (Yes or No).
where [Input], [A], [B] are the sentence and two entities' mentions in the data, [M] refers to the mask token and [Relation] denotes the data's rela- tion label. P; is the mention of reasoning rules pt, with the sentence and the statement "We can infer that" being removed to ensure the prompt's 's fluency.
After feeding the evaluating prompt to PLM, the probability of the mask token being predicted as 'Yes" is taken as the composition's 's evaluation score. For each data, we enumerate all the possible compositions of its four reasoning rules and rank them by the evaluation score descendingly. Finally, in each data, the top N₂ compositions are picked and each composition will form a compound rule. Compound Rule Definition. Once a compound rule denoted as ps is built by a composition of
In each iteration, the top Nu rule-labeled data with the highest labeling confidence selected to enrich the training set.
3.3 Graph-based Data Filter
We utilize the RE model learned on the training set to generalize over the unlabeled set and provide model-labeled data for new rules discovery. For each relation, to select the model-labeles data having inference features consistent with its

---

Ieput: [Smith] faces first degree [assault] and other charges related the shooting.
Relation: is_ j__chorge__of
Rule
Rules Generation Example for Sec. 3.1 (In example. [A]- [Smirh], [R]- [assault])
Prompt
Reasoning Rules [P:]i-1
Ressoning Words
Meta Rule
[Inpor]. We can infer that. [Smith]] to [assuuht's fins sentence, [assanlt) leads [Smith]'s consiction.
Templates
M
[Inpur]. We can inder that. [Smith] is penson of probation for [sssault), while [assault] is crime of felony for [Smich].
Input
Prompts
[[npa]] We infor thot. [Smith]] brought puilty to [svsuult] while [sessult] brought injury [Smiihh).
[Input]. We cun infer that. without (Smich). [sssallt] will not charges, while without [assault). [Smith] will face murder.
Compusition Evaluntion Example Sec. 3.2 our example, we evaluate composition of rules py and pal Evalunting Prompt
Innuri [Smith] brought guilry (sossult). While [sssault] brought injery |Smical.
Evaluation Score
Input
M
Rules Composition
Without (Smith), ,. (assault) will not charges, while without (usssult), [Smich) will face munder.
P[MM =/ Yes')
Question: Can infer [Smith] charged [aessult]] Answer: (Yes Not.
Examples for generating reasoning rules and evaluating rules composition given the input sentence "Smith faces first-degree assault and other charges related to the shooting".
Figure 3:
seed rules and far away from other relations', we build a graph for this relation by connecting the model-labeled data with seed data based their reasoning words' relevance. Then a classifier is trained to figure out if data belongs to this relation (Positive) or not (Negative).
nary label of the node, P, is the predicted possibility of node i being Positive, and Lnei = -jev EKEN, 1hg Mhg-hyll 2 is encourage nodes with similar inference features to be close each other in the representation space.
High-quality Data Filtering. To pick the model- labeled data containing inference features consis- tent with the existing rules for relation and dis- tinguishable from other relations, we 1) compute the centroids' representation of Positive and Nega- tive seed nodes as the average of corresponding nodes' representation; 2) measure each model- labeled node's 's cosine similarity with the two cen- troids' representation as Spos and Sney: 3) rank all model-labeled nodes by S = Spoa Sneg descend- ingly to find distinguishable nodes. Then we select the top Np model-labeled nodes (data) for further rules generation of relation
Initial Node Embedding. Since node represents a data, for each node, we concatenate the data's ' four types of reasoning words' embedding (introduced in Sec. 3.2) its initial node embedding.
Graph Construction. We build a k-nearest neigh- bor graph G, = {V = v Vs, "/,,A) A} for relation r, which connects nodes based on their initial em- bedding's ' cosine similarity and aims to model the data's inference features relevance. Vpp. V, are nodes for model-labeled data and seed data, A is the adjacency matrix. To get the binary supervision signals, each node in V, will get a Positive binary label if its data belongs to r, otherwise, Negative. Inference Features Propagation. For rela- tion r and its corresponding graph G,, we ap- ply an independent 2-layer graph attention net- work (Veličković et al., 2018) to propagate nodes' inference features. Given node i's representation h and its neighbors N₁, in each layer the attention weight between node and j can be computed as:
4 Experiments
4.1 Datasets
By following (Hu et al., 2021a), we conduct exper- iments on two public datasets widely used for rela- tion extraction, SemEval (Hendrickx al., 2010) and TACRED (Zhang et al., 2017). The datasets statistic is shown in Appendix A. TACRED is more complicated than SemEval, since it contains more relation types and larger distribution bias between positive and negative instances.
exp(f(A ( Why])) aij = EECN, (AT( [Wh;, Whi))
(4) where W is parameter matrix and ʃ is LeakyReLu function. Then the 7th node represen- tation of the next layer is updated as:
4.2 Baselines
Supervised Baselines. We build classifiers with three common encoders and train them with only la- beled data: LSTM (Hochreiter and Schmidhuber, 1997), PCNN (Zeng al., 2015) and BERT (Ken- ton and Toutanova, 2019). Since BERT performs best, we take it as the encoder of all the following baselines and ARIA'S RE model for fair evaluations.
h = a; Wh; + EJEN, Wiy (5)
Objective. The classifier's s objective is defined as:
L = Luu + Laci
(6)
where Lsup= E (ynlogP; + (1 1o)log(1- - P.))
(i E Vi) is the supervised loss, yi is the bi-

---

SemEval
TACRED
Methods / % Labeled Data
5%
10%
30%
3%
10%
15%
LSTM (Hochreiter Schmidhuber, 1997)
22.65
32.87
63.87
28.68
46.79
49.42
PCNN (Zeng al., 2015)
41.82
51.34
63.72
40.02
50.35
52.50
BERT (Kenton and Toutanova, 2019)
70.71
71.93
78.55
40.11
53.17
55.55
Self-Trrininggeeerr (Rosenberg et al., 2005)
71.34
74.25
81.71
42.11
54.17
56.52
Mcan-Teacherggerr (Tarvainen and Valpola, 2017)
70.05
73.37
80.61
44.34
53.08
53.79
RE-Ennsembleesser (Lin et al., 2019)
72.35
75.71
81.34
42.78
54.83
55.68
DualEE-Paiviiseeeer (Lin al., 2019)
74.35
77.13
82.88
43.06
56.03
57.99
DualRE-PPoinwiieenrrer (Lin et al., 2019)
74.02
77.11
82.91
43.73
56.28
57.72
MRefGBERr (Li and Qian, 2020)
75.48
77.96
83.24
43.81
55.42
58.21
MetaSREgeRr (Hu et al., 2021a)
78.33
80.09
84.81
46.16
56.95
58.94
GradLREBERT (Hu el al., 2021b)
79.65
81.69
85.52
47.37
58.20
59.93
ARIABER (Ours)
80.24
82.40
86.07
49.59
60.86
62.57
BERT w. gold labels
86.66
87.25
87.87
62.56
64.15
64.51
Table Micro F1 (%) of methods on SemEval and TACRED datasets with different scales of seed data and 50% unlabeled data. For each method we conduct five runs with random seeds and report the average performance.
Low-Resource Learning Baselines. We compare ARIA with baselines studied under the low-resource setting. 1) Self-Training (Rosenberg et al., 2005) generalizes the learned model on unlabeled set and updates the model with pscudo labels. 2) Mean- Teacher (Tarvainen and Valpola, 2017) forms the teacher model from students by encouraging consis- tent predictions for similar inputs. 3) DualRE (Lin et al., 2019) jointly trained retrieval module to pro- vide psseudo-labeled data for the prediction module. Pointwise or Pairwise the way to measure the re- trieved data's quality. 4) RE-Ensemble (Lin et al., 2019) replaces DualRE's retrieval module with prediction module. 5) MRefG (Li and Qian, 2020) connects the unlabeled data with labeled ones se- mantically to build reference graphs. MetaSRE (Hu et al., 2021a) eval pseudo-labeled data's 's quality by meta-learning from the model's successful and failed attempts. 7) GradLRE (Hu et al., 202 lb) encourages the pseudo-labeled data to imitate the labeled data's gradient descent direction.
data are sampled as unlabeled data, whose gold labels are unavailable during the learning process. Hyperparameter Setting. Roberta-base (Liu et al., 2019) is taken as our PLM for rules reasoning. For the Compound Rules Compiler, in each data the top N, 8 high-quality combinations are picked to build compound rules and the threshold TH for data labeling set to 0.8. In each iteration, the top Na = 100 aulelld data with the highest label- ing confidence are added to the training data. We take DualRE-Pointwise as our RE model, trained with our training set (containing both seed data and rule-labeled data) and the unlabeled set. For each relation, the Data Filter picks the top Np 15 high-quality model-labeled data for further rules generation the next iteration.
4.4 Overall Performance
Table 2 shows the performance on two datasets. Overview: We could observe that ARIA outper- forms all the baselines both datasets with differ- scales of seed data, which demonstrates ARIA's effectiveness and stableness. Specifically, com- pared with the previous SOTA work GradLRE, ARIA average achieves an Fl improvement 0.62% in SemEval and 2.5 2.51% % in TACRED.
RE-Gold. We train our RE model with both seed data and the unlabeled data assigned with the gold labels indicate the upper bound performance (denoted as BERT w. gold labels).
4.3 Implementation Details
Compared with the DualRE-PPintwise that only trained with the seed data and unlabeled data, the RE model (with the same structure) embedded in ARIA average gets Fl score 4.89% higher in Se- mEval and 5.10% higher in TACRED. This result shows significant improvement from the compre- hensive rules distilled from the PLM.
Data Preparation. To imitate the low-resource scenario, we follow (Hu et al., 2021a) and utilize stratified sampling divide the original training set into seed data and unlabeled data. Specifically, for SemEval, we sample 5%, 10%, 30% training data as seed data, while for TACRED, 3%. 10%, 15% are sampled. For both datasets, 50% training
Among the two datasets, ARIA gets greater im-

---

Ong: Founded_hy by
Org: Memher of
Causc-Fffict (cl,
[Lee, the third son of the late Samsung group founder Lee Byung. became vice chairman of the Sumsung Corporation.]
[Shiite List groups lots of Shiite and other parties, including the Badr Orgunization former Badr Corps and independents.]
[Asthma causes swelling and narrowing of the airways.]
We can infer that, [Asthma] leads to [swelling]'s 's had symptoms, while [swelling] leads to [Asthma]'s worsening death.
We can infer that, [Samsung] leads to [Lee Byung]'s business success. while [Lee Byung] leads to
We can infer that, [Badr Organizaaioon] leads to IShitell's
large list. while [Shiit]] leads to [Badr Organizationl's Shiite party.
Org: Subsidiaries
[Samsung]'s I business compony.
[Cariial Cruise Lines president Robert Dickinson, will retire at the end of the year, parent Camival Corp said.]
Whole- Component (el,e2)
Cause-Effeee (el, c2)
[The kitchen holds u cooker, fridge, short: overything you need to prepure a light meal.]
[The burned fuels create air pollution that contributes to global warming and causes respiratory disease.]
microwave oven,
We can infer that, [Camival Corpl [Camival Cruise Lines]. while [Camival Cruise Lines] a subsidiary of operation for [Camival Corpl.
We can infer that, [pollution] is a source of risk [discase], while [discusc] is a source blame for [pollution].
parent of operations
We can infer that, [kitchen] a kind room for [cooker]. while [cooker] is a kind storage for [kitchen].
Per: Cause of Death
Enntit-Origin (cl,c2)
Per: Parents
[Beverly Sills, obituary Opera
[When the princess left her native land and traveled west her bridegroom, carried silkworm cocoons
[Catherine was born in Brooklyn in 1920, one three children of Albert and Gertrude Dittmars Roraback.] We can infer that, [Catherine] brought married to [Albert]. while [Albert] brought brith to [Cutherine].
legend Beverly Sills succumbed to lung cancer tonight at age 78.]
We can infer that, [Beverly Sills] brought succumbe to [lung cancer]. while [lung cancer] brought death [Beverly Sills].
headdress.
We can infer that, [princess] brought her [land], while [land] brought happiness pprincess].
Org: Subsidiaries
Per: Date_ot Birth
Product- Producer c2)
["Many People see the kibbutz corpse," " says Shlomo, head of the Kibbutz Studies Centre at Haifa University.]
[Bom on Aug. 15, 1925, a poor neighhorhood of Montreal, Peterson got his passion for music from his father.]
[Thee steering committee that drufted the document also passed it through two Palestinian GFM organizers.] We can infer that, without [committee]. [document] will not exist. while without |document). [committee] will he exist.
We can infer that, without [Haifa University]. [Kibbutz Studies Centre] will not exist, while without [Kibbutz Studies Centre), [Haifa University] will be exist.
We can infer that, without [Peterson], [Aug. 15, 1925] will be meaningless. while without [Aug. 15. 1925]. [Peterson] will be void.
Induction
Abduction
Intervention
Counterfactual
Figure 4: Examples of reasoning rules belonging to different relations summarized by the PLM in Section 3.1. The four types of color indicate four types of reasoning paths. The reasoning words for each rule are highlighted.
provements on TACRED compared with all base- lines, which shows ARIA could better leverage the PLM to exploit high-quality rules to improve the model's learning of more complex relations with skewed data distribution.
When extracting rules directly from all the model- labeled data without Data Filter, the Fi scores lower 3.47% ~ 1.86%. The decrease gets larger when running with fewer seed data. Thus when the seed data is quite limited, the Data Filter is neces- sary to pick reliable data from the unfitted model's predictions for high-quality rules generation.
4.5 Ablation Study
We present ablation studies on SemEval to show the components' effectiveness (Table 3).
Removing Compound Rules Compiler (w/o CRC). When labeling with single reasoning rules rather than compound ones, the performance gets lower, especially when the seeds' scale turns from 5% to 10%. This is because some relations can not be discriminated by single reasoning paths and more seed rules lead to noise labeling. Thus, Com- pound Rules Compiler is needed for fine-grained data labeling. The effect of labeling noise gets lighter the seed data is abundant enough (30%).
Methods / Labeled Data
5%
10%
30%
ARIA
80.24
82.40
86.07
ARIA w/o GDF
76.77
80.48
84.21
ARIA w/o CRC
77.99
78.09
84.31
ARIA w/o Indu.
75.65
80.87
84.65
ARIA w/o Abdu.
74.53
79.85
84.99
ARIA w/o Inte.
75.82
80.16
83.28
ARIA w/o Cont.
76.72
80.61
85.47
Table 3: Ablation results of ARIA and its variants on SemEval with different scales of seed data.
Removing one type of meta-rule each time (w/o Indu, w/o Abdu, w/o Inte and w/o Cont). The re- sult shows that each meta-rule contributes to ARIA.
Removing Graph-based Data Filter (w/ GDF).

---

The lack of each meta-rule causes a sharp perfor- mance degradation on small-scaled seed data (5%). This is because the labeling noise caused by the incomplete reasoning paths (as shown in Sec. 4.6)
Labeling Pricision Unhibeled Dutn
90
AIIA ARUAC
BL-K
Precision (8)
70-
can easily lead to the model's 's misunderstanding of relations when the correct seed data is limited.
50-
4.6 Case Study
40-
Figure 4 shows examples of four types of reason- ing rules. We can observe that the rules' reasoning words capture the information crucial for relation inference. For example, given an instance with en- tity pair (X, Y) and meeting "X leads Y's business success, while leads to X's business company", we can infer that X may be a company greatly con- tributed by Y and Y is likely to be X's founder. Thus our reasoning words are beneficial to sentence un- derstanding and rules construction.
10
20
30
40
50
60
Figure 5: Labeling precision of ARIA and different pipelines based ChatGPT.
predict the reasoning words quickly and is more time-saving to handle large-scale data in real sce- narios. Furthermore, Roberta could be deployed privately and avoid the data breach risk. This ex- ploration also shows our framework could embed different types of language models as the annotator. Achieve Concentrate In-Context Learning with Rules Knowledge. We apply in-context learning to annotate the unlabeled data directly by ChatGPT. Taking this as a baseline (denoted as ICL), we de- sign ICL-R, each example data in the context is followed by its reasoning rules mentions instanti- ated in ARIA. We can see ICL-R is at most 23.04% higher than ICL. Since in ICL-R, ChatGPT focuses more the representative knowledge points for lation reasoning summarized by rules, it is less likely be distracted by other information or hal- lucination caused by the example data. This shows the small-scaled PLM's potential to highlight the key points and assist the reasoning of ChatGPT. Future Works: Cooperation for Complex Sce- narios. ChatGPT shows powerful comprehension ability but sometimes generate hallucination. ARIA provides an efficient rules-distilingg manner that could reduce hallucination, but simple rules match- ing is not enough for complex semantics. To lever- age the advantages of both sides, we could first ask Roberta simple questions and summarize the key points into rules, which are collected to ChatGPT to get reliable reasoning with less hallucination. We could further ask ChatGPT if its conclusion obeys the rules or if the rules are reasonable, and fix the answer or the rules automatically.
Necessity of Compound Rules. Besides, the rea- soning rules provided by single reasoning path is not enough to discriminate different relations. For example, under the Counterfactuals reasoning path, the rules of relation Product-Prodde and Org: Subsidiaries have similar reasoning words. Thus, to label the relation Org: Subsidiaries accurately, the Abduction rule is necessary, which indicates the effectiveness of compound rules construction.
4.7 ARIA'S Potential under ChatGPT
To discuss ChatGPT's influence ARIA, taking 5% training data seed data and 50% unlabeled data in Semlival, we design different labeling man- ners based on ChatGPT and compare their labeling precision on the unlabeled data (as Figure shows). Notice that we sort the annotated results by label- ing confidence and compute the precision for the top K results (more details are in Appendix A.2). ARIA: Competitive Performance with Light and Privacy-Protected Annotator. We replace Roberta with ChatGPT for rules generation and composition in our framework, and label data with compound rules (denoted as ARIA-Cha PTLAACChaGPTT). When Top K is 5-45, ARIA's ' labeling precision is 1. 1.67%~7.72%% higher than ARIA-ChatGPT. This is because the hallucination of ChatGPT may make rules unreliable and lead to fake high-confident la- beling results. For example, when the ground-truth relation is "NA" (there is relation between the two entities), ChatGPT may generate unreasonable rules that seem brilliant, while Roberta, the naive PLM, may answer meaningless words. Compared with ChatGPT, Roberta embedded in ARIA could
Conclusion
We propose ARIA, which guides the PLM sum- marize comprehensive rules as human. Specifically, we build a Reasoning Rules Generator to replace human for high-quality rules generation and a Com-

---

pound Rules Compiler to compose rules into robust compound rules. Besides, ARIA could discover reli- able model-labeled data for further rules generation based on the rules relevance. Experiments on Se- mEval and TACRED show ARIA's effectiveness.
Xuming Hu, Chenwei Zhang, Fukum Ma, Chenyao Liu. Lijie Wen, and S Yu Philip. 2021a. Semi cemi-supervised relation extraction via incremental meta self-training. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 487-496.
Xuming Hu, Chenwei Zhang, Yawen Yang, Xiaohe Li,
Li Lin, Lijie Wen, and S Yu Philip. 2021b. Gradi- ent imitation reinforcement learning for low resource relation extraction. In Proceedings of the 2021 Con- ference on Empiricul Methods in Natural Language Processing, pages 2737-2746.
6 Acknowledgments
This work has been supported in part by the Zhejiang NSF (LR21F020004), Key Research and Development Projects in Zhejiang Province (No. 2023C01030, 2023C01032), NSFC (No. 62272411), 1), National Key Research and Develop- ment Program of China (201 80AAA00110),,)) Ant Group and Alibaba- Zhejjiang University Joint Re- search Institute of Frontier Technologies.
Jacob Devlin Ming- Meig-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171-4186.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. arXiv e-prints, pages arXiv-
Limitations
Juncheng Li, Minghe Gao, Longhui Wei, Siliang Tang, Wenqiao Zhang, Mengze Li, Wei Ji, Qi Tian, Tat- Seng Chua, and Yueting Zhuang. 2023. Gradient- regulated mmta--proppt learning for generalizable vision-language models.
our Reasoning Rules Generator, we take Roberta's most likely predicted words for mask tokens as reasoning words. This may ignore much important information contained in Roberta's 's pre- dictions. An approach to making better use of Roberta's 's knowledge is to take Roberta's ' hidden vectors for the masked tokens as virtual reasoning words. Besides, to better leverage our rules to han- dle the complex semantics, we could collect the rules with representative key points from Roberta and input them with context to ChatGPT for a reli- able answer.
Juncheng Li. Xin He, Longhui Wei, Long Qian, Linchao Zhu, Lingxi Xie, Yueting Zhuang, Qi Tian, and Sil- iang Tang. 2022a. Fine-grained semantically aligned vision-language pre- pre-training. Advances neural information processing systems, 35:7290-7303.
Juncheng Li, Siliang Tang, Linchao Zhu, Haochen Shi, Xuanwen Huang, Fei Wu, Yi Yang, and Yueting Zhuang. 2021. Adaptive hierarchical graph reason- ing with semantic coherence for video-and langlauaaee inference. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 1867- 1877.
References
Geoff French, Michal Mackiewicz, and Mark Fisher. 2018. Self-ensembling for visual domain adaptation. In International Conference on Learning Representa- lions.
Juncheng Li, Xin Wang, Siliang Tang. Haizhou Shi, Fei Wu, Yueting Zhuang, and William Yang Wang. 2020. Unsupervised reinforcement learning of transferable meta-skills for embodied navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12123-12132.
Bo Han, Quamming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W Tsang, and Masashi Sugiyama. 2018, Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurIPS.
Juncheng Li, Junlin Xie, Long Qian, Linchao Zhu, Siliang Tang, Fei Wu, Yi Yang. Yueting Zhuang, and Xin Eric Wang. 2022b. Compositional tem- poral grounding with structured variational cross- graph correspondence learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3032-3041.
Iris Hendrickx. Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010. Semeval-2011 task 8: Multi- way classification of semantic relations between pairs of nominals. In Proceedings of the 5th International Workshop Semantic Evaluation, pages 33-38.
Wanli Li and Tieyun Qian. 2020. Exploit multiple refer- ence graphs for semi-supervised relation extraction. arXiv e-prints, pages arXiv-2010.
Hongtao Lin, Jun Yan, Meng Qu, and Xiang Ren. 2019. Learning dual retrieval module for supervisd relation extraction. The World Wide Web Confer- ence, pages 1073-1083.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 18):1735- 1780.

---

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach.
Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 2810-2819.
Qifan Yu, Juncheng Li, Yu Wu, Siliang Tang, Wei Ji, and Yueting Zhuang. 2023, risually-promphr lan- guage model for fine-grained scene graph generation in an open world. Proceedings the IEEE/CVF International Conference on Computer Vision
Yilin Lu, Xiaoqiang Wang, Haofeng Yang. and Siliang Tang. 2023. Kice: A knowledge consolidation and expansion framework for relation extraction. Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, 37(1 13336-133433
Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao. 2015. Distant supervision for relation extraction via piecewise convolutional neural networks. In Proceed- ings of the 2015 conference on empirical methods in natural language processing, pages 1753-1762.
Jeffrey Pennington, Richard Socher, and Christopher D Manning, 2014. Glove: Global vectors for word rep- resentation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Rongzhi Zhang, Yue Yu, Pranav Shetty, Le Song, and Chao Zhang. 2022. Prompt-based rule discovery and boosting for interactive
Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In Proceedings of the 2010 European conference on Machine learning and knowledge dis- covery in databases: Part III. pages 148-163.
-superssdee learning. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume I: Long Papers), pages 745-758.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D Manning. 2017. Position-aware aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Lunguage Processing, pages 35- 45.
Chuck Rosenberg, Martial Hebert, and Henry Schnei- derman. 2005. Semi-supervised self-training of ob- ject detection models. In Proceedings of the Seventh IEEE Workshops on Application of Computer Vision (WACVMOTION Voluue 1- I-Volume 01, pages 29-36.
Wenxuan Zhou, Hongtao Lin, Bill Yuchen Lin, Ziqi Wang, Junyi Du, Leonardo Neves, and Xiang Ren. 2020. Nero: A neural rule grounding framework for label-efficient relation extraction. In Proceedings of The Web Conference 2020, pages 2166-2176.
Abulhair Saparov and He He. 2022. Language models are greedy reasoners: A systematic formal analysis of Mhain-of-thoughh arXiv e-prints, pages arXiv-22 10.
Ang Sun and Ralph Grishman. 2012. Active learning
for relation type extension with local and global data
A Appendix
views. In Proceedings of the st ACM international conference on Information and knowledge manage- ment, pages 1112.
A.1 Datasets Statistics
In this section we show the statistics of the two public datasets used in our experiments.
Antti Tarvainen and Harri Valpola. 2017. Mean teachers are better role models: Weight- consistency targets improve semi-supervised deep learning re- sults. Advances in neural information processing systems, 30.
Datasets SemFval TACRED
Relations
Train
Dev
Test
%Neg.
19
7199
800
1864
17,40
75049
25763
18659
78.68
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph attention networks. In International Conference on Learning Representations.
Table 4: Statistics the datasets in our experiments. Notice that Neg. refers to the percent of negative in- stances, which are labeled as NA.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2022. Self-consistency im- proves chain of thought reasoning in language mod- els. arXiv e-prints. pages arXiv-2203.
A.2 Implementation Details of ICL, ICL-R and ARIA-ChatGPT
For ARIA-ChatGPT, we redesign cloze templates to prefix templates to exploit ChatGPT's potential. Notice that for both ARIA and ARIA-ChatGPT in Figure 5, we report the labeling precision in the first iteration. For ICL and ICL-R, we randomly select one seed data per relation example data in context. For each unlabeled data, the ChatGPT predicts 5 times under the contexts with random example data. Then majority voting strategy is
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le. and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv e-prints, pages arXiv-2201.
Zhi-Xiu Ye and Zhe I. Ling. 2019. Distant supervi- sion relation extraction with intra-bag and inter-bag attentions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for

---

Inference Type
Meta-Rulc Template
[Input]. We infer that. leads [B]'s MASK. Question: MASK (Picase
words).
Induction
[Inpu]]. We infer that, leads [A]'s MASK. Question: MASK (Pleuse
words).
Input]. We can infer that. [A] MASK MASK for Question: MASK I= (Picase answer within word).
Input]. We can infer that.
MASK of MASK lor [B]. Questio: MASK 2= (Please answer within word).
Abduction
[nput]. We can infer that. [8] is MASK of MASK 2 for [A]. Question MASK == (Picase answer within word).
[Input]. We can infer that. [B] is MASK of MASK 2 for [A]. Question: MASK 2 (Please answer within word).
[Inutt]. We can infer that.
brought MASK to [B]. Question: MASK (Picase
within word).
Intervention
[mput]] We can infer that, [R] brought MASK to [A]. Questtion: MASK (Pleuse unswer within one word)
Input].. We can infer that, without [A]. [B] will MASK. Question: MASK (Please answer within two words).
Conterfactual
[Input]. We can infer that, without [B). [A] will MASK.
Question: MASK (Please answer within two words).
Table 5: Prefix Meta-Rule Template applied in ARIA-ChatGPT. [input], [A] and [B] refer the sentence, head entity and tail entity.
taken pick the final prediction and the max votes number is taken as the labeling confidence. Prefix Template for ARIA-ChatGPT. To generate and compose the rules by ChatGPT, we transfer the cloze template into prefix template by asking ChatGPT what the "MASK" refers to in natural language. The prefix Meta-Rule Template is shown in Table The prefix template for building the evaluating prompt is shown as follows:
where [Input], [A], [B] refers the sentence and the two entities mentions in the instance and [Rela- tion] donates the instance's s relation label mention. P; is the mention of reasoning rules p;, with the sen- tence and the statement "We can infer that" being removed. S represents the set of reasoning rules to be evaluated for composition.
Context Building in ICL, ICL-R. The template to build the context is shown in Figure 6. [Inpud]i [A]; [B; and [Relation]; are the sentence, head entity, tail entity and the label 7th example data. n the number of labels and [Relation Set is the set of all label mentions. [P']: = [(/))= is the set of reasoning rules' mentions of 7th example.
[Input]. {pties Question: Can we infer [A] [Relation] [B]? Answer: (Yes or No).
[Ipptt]
Q: What is the relation label [A], and [8], A:[Relation],
[Imput], hn
Q: What is the relation label of [4], and [B],"
A: [Relation],
[mpput]
What is the relation label [4] and [B] You
should answer one of following relations: [Relation Set].
A:
ICI.
[nput]] [P]]
Q: What is the relation label of [4]; and [B],
[Relation]
[mput]]
[P'],
Q: What is the relation label of [A], and [B),"
A: [Relation],
[[nput] [P']
What is the relation label of [A] and [B] You
should answer one the following relations:
[Relation Set].
A:
ICL-R
Figure 6: Template of contexts in ICL and ICL-R.


Reviews:
[{"review_id": "dfjE4E76bq", "reviewer": "Reviewer_g2bw", "summary": "", "questions": "1. In the soft matching part, why not use NLI to calculate the score? Since the rule is reasoned from data, NLI may be better than embedding similarity.\n2. There are many no-relation instances in RE datasets. How do you label them using rules?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the valuable comments. We are encouraged to see that our work is recognized as well-written, well-organized, and novel. We will explain your concerns point by point.  \n#### \n#### **R1: Comparison with prompt-based methods and NLI/summarization for RE**\nThank you for your constructive advice. To show our framework’s effectiveness, we compare ARIA with KnowPrompt[1] and NLI for RE[2] on TACRED with 10% training data. Notice that Roberta is taken as the pre-trained language model for all the above baselines, and stratified sampling is utilized to sample the seed data (Line 423-426). The results are shown as follows:\n\n| Methods       | Micro F1 |\n|---------------|----------|\n| KnowPrompt    | 58.24    |\n| NLI (Roberta) | 59.89    |\n| ARIA (ours)   | 60.86    |\n\nOur work achieves 2.62% F1 scores greater than KnowPrompt and 0.97% greater than NLI for RE. One reason is that these frameworks could not utilize the knowledge from unlabeled data, while our framework continuously discovers high-quality rules from the unlabeled data. What’s more, our work doesn’t require finetuning or prompt-tuning for PLM, which means the unopen-source PLM (like ChatGPT) could be embedded in our framework easily. We will add more comparisons in the future version.\n\n#### **Q1: Whether to use NLI for matching score calculation**\nThanks for your nice suggestion. With quick computation, cosine similarity is commonly used for soft sentence-rule matching and serves as the maximum inner-product searching in the large-scale retrieval system. Thus, we take cosine similarity for matching score calculation. We also conducted a brief experiment to evaluate the rules’ labeling performance on the same set of 1000 unlabeled data from SemEval with a pre-trained NLI model (Roberta-Large-MNLI) or cosine similarity as the matching manner.\n\n| Methods            | Precision (%) | Recall (%) | Micro F1 (%) |\n|--------------------|---------------|------------|--------------|\n| Cosine Similarity  | 65.17         | 62.29      | 63.69        |\n| Roberta-Large-MNLI | 75.38         | 58.19      | 65.67        |\n\nThe NLI model pre-trained on MNLI dataset achieves Micro F1 1.98% higher than cosine similarity in data labeling. This is because the NLI model could focus on not only word-level information, but also the context representation between rules and unlabeled data. We will discuss the NLI model’s potential in our framework in the next version.\n\n#### **Q2：Rules labeling for no-relation instance**\nThe data that fails to match with any rules is labeled as \"no-relation\". In our rules-matching manner, a threshold is set to filter out those data having inconsistent features with rules (As illustrated in Line 292-294). To deal with \"no-relation\", for unlabeled data, if it is not matched with any compound rules, which means it does not belong to any of the valid relation types, it will be labeled as\" no-relation\". Besides, for those data that are erroneously labeled as\" no-relation\" in the original dataset, our rules have the potential to label them with their actual relations.\n\n#### **Typos Grammar Style And Presentation Improvements**\nThanks for your suggestions. We will move ChatGPT's corresponding results to Table 2 in the next version.\n\n&nbsp;\n\n[1] KnowPrompt Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction\n\n[2] Label Verbalization and Entailment for Effective Zero and Few-Shot Relation Extraction", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and well organized.\n2. The proposed framework is interesting incremental work based on existing rule-based weak supervision methods.", "reasons_to_reject": "1. In experiments, it is necessary to compare with other key methods for low-resource RE, including (1) prompt-based methods (e.g., Knowprompt), and (2) indirect supervision based methods (e.g., NLI for RE, summarization for RE).\n2. Some design choices are not well explained (see questions below).\n\nAfter rebuttal: the authors have added necessary baselines, so I raised the soundness score.", "questions_to_authors": "1. In the soft matching part, why not use NLI to calculate the score? Since the rule is reasoned from data, NLI may be better than embedding similarity.\n2. There are many no-relation instances in RE datasets. How do you label them using rules?", "paper_topic_and_main_contributions": "This paper proposes a rule distillation framework for relation extraction. The framework is composed of (1) a reasoning rule generator based on predefined templates; (2) a compound rule compiler that combines multiple rules; (3) a relation extractor that is trained on both supervised and rule-labeled data. Experiments on two sentence-level RE datasets in the low-resource setting show the effectiveness of the proposed framework.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Better to add ChatGPT's ICL results to table 2."}, "extra_text": {}}, {"review_id": "GyhYcceMbq", "reviewer": "Reviewer_WGwB", "summary": "", "questions": "The  key motivation seems to enforce reasoning, however, I cannot understand the motivation for their chosen methodology. For instance, the necessity of creating compound rules. ", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s helpful feedback. We are encouraged that the reviewer finds our idea interesting and novel. We will explain your concerns point by point.\n\n&nbsp;\n\n#### **R1：Difficult to follow and read**\nThank you for your suggestion. We will carefully polish our paper in the next version.\n\n#### **R2, R3: Missing baselines for weak-supervision and semi-supervision**\nThanks for the nice suggestions. Among the baselines mentioned in Missing Reference, the work a) and d) require human-written labeling functions as rules for weak labels, while our rules are generated by PLM from data automatically, which needs less expert knowledge.\nBesides, for the rest of the baselines that could generate rules automatically, we also compare our framework with one of the best baselines, ARI, in our low-resource setting. With 5% training data in SemEval, our framework achieves F1 score **12.37%** higher than ARI, which demonstrates that the comprehensive compound rules distilled in our framework could utilize PLM’s broad knowledge to achieve a more fine-grained data labeling.\n\n| Methods            | Micro F1 (%) |\n|--------------------|--------------|\n| ARI (Ngram+linear) | 67.87        |\n| ARIA (Ours)        | 80.24        |\n\n#### **Q1: Necessity of creating compound rules**\nThanks for your question. As introduced in Line 105-109, motivated by the work[1], which reported that combining the reasoning process in different ways could lead to a more correct answer, we compose the reasoning rules into compound rules. Besides, in Line 509-517 in Case Study, we also illustrate by examples that a single type of reasoning rule is not enough for relation discrimination. What’s more, in Line 479-488 in the Ablation Study, removing the Compound Rules Compiler may lead to a noise rule labeling and a lower RE model performance. We will further clarify this more clearly in the revision. \n\n[1] Bert: Pre-training of deep bidirectional transformers for language understanding.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposes an interesting approach for enforcing reasoning using PLMs. ", "reasons_to_reject": "1. The paper is very difficult to follow and understand. \n2. Paper misses several relevant weak-supervision and relation extraction baselines . Refer missing citations section.\n3. No semi-supervised rule-based baselines are considered.\n", "questions_to_authors": "The  key motivation seems to enforce reasoning, however, I cannot understand the motivation for their chosen methodology. For instance, the necessity of creating compound rules. ", "paper_topic_and_main_contributions": "The paper proposes an approach for relation extraction using pre-trained language models (PLM) for creating rules. It uses an iterative approach in which small labeled set and meta-rule templates for creating subsequent rules automatically. These rules are converted into compound rules which are then fed to the rule aggregation algorithm.", "excitement": 2, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "   a) Robust Data Programming with Precision-guided Labeling Functions. AAAI 2020: 3397-3404\n\n   b) Learning to Robustly Aggregate Labeling Functions for Semi-supervised Data Programming. ACL (Findings) 2022: 1188-1202\n\n   c) Semi-Supervised Data Programming with Subset Selection. ACL/IJCNLP (Findings) 2021: 4640-4651\n\n   d) Learning from Rules Generalizing Labeled Exemplars. ICLR 2020\n\n   e) Snuba: Automating Weak Supervision to Label Training Data, VLDB 2018\n\n   f) Automatic Rule Induction for Efficient Semi-Supervised Learning, EMNLP 2022"}}, {"review_id": "C0ipIBYgnN", "reviewer": "Reviewer_Hf2b", "summary": "", "questions": "1. In Sec 3.2, L262-263, the paper mentions \"... We ask the PLM by prompt to evaluate if each composition has enough information ....\" followed by the definition of the Composition prompt; Does this mean that the composition prompt is evaluated for all possible compositions of rules in Table 1? What kind of additional run-time complexity does ARIA incur on this step?\n\n2. The need for Sec 3.2, i.e., the need to instantiate explicit compound rules is unclear. From Equations (1), (2), and (3), we observe that, given an unlabeled sample u, whether u matched p_S (the compound rule) does not depend upon p_S but depends on p_i, i.e., the rules obtained from the meta-rule template and the filled in prompt (from the LM). So, ideally, we donot require an instantiation of the compound rule p_S, correct?\n\n3. A low-resource setting for Relation Extraction task typically occurs in domain specific datasets such as Chemprot [1], SCIERC, etc. and not on Open domain datasets such as TACRED or SemEval. How does ARIA perform on either one of them in comparison to existing SOTA approaches? Also, referring to the evaluation section, using 3% of TACRED (or 5% of SemEval) over five random runs can still have a large variance - depending upon which 3% of TACRED got sampled for training.\n\n4. What is the difference between BERT (Row 3) and BERT w/ gold labels (final row) in Table 2? Any insights into why the performance improvement on TACRED is higher than that of SemEval? Also [2] shows that a significant chunk of the most challenging sentences from the dev/test sets of TACRED are incorrectly labelled, and therefore introduce a newer annotated variant called Re-TACRED with all the fixes. How does ARIA perform on Re-TACRED?\n\n[1] Peng, Yifan, Shankai Yan, and Zhiyong Lu. \"Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets.\" BioNLP 2019 (2019): 58.\n[2] Stoica, George, Emmanouil Antonios Platanios, and Barnabás Póczos. \"Re-tacred: Addressing shortcomings of the tacred dataset.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 15. 2021.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer for the constructive and insightful feedback. We are encouraged that the reviewer agrees that our paper is well-motivated, the framework description is ingenious, and the qualitative and quantitative analysis is meaningful. We will explain your concerns point by point.\n\n&nbsp;\n\n#### **R1: One word or several words for reasoning words**\nThanks for raising this concern. As introduced in Line 306-309 and Figure 3, each type of reasoning words (which is denoted as $w_i$) contains 2-4 single words （depending on the number of mask tokens in the corresponding template). And the embedding of $w_i$ (denoted as $e(w_i)$) is the concatenation of those single words embedding. Then, as illustrated in Line 333-336, the initial node embedding for a data point is $\\lbrace w_1||w_2||w_3||w_4\\rbrace$, where \"$||$\" denotes concatenation. We will further clarify this in the revision.\n\n#### **Q1: Time complexity for rules composition evaluation**\nThanks for your question. Let N be the number of data for rules generation. The time complexity for rules composition evaluation is $O(N)$. Since different data's reasoning rules are generated under different contexts, composing them into compound rules may lead to ambiguity and bring noise to data labeling. Thus, as introduced in Line 259-260, we only evaluate the composition of reasoning rules that are generated from the same data. As each data has four reasoning rules, one data requires $4^2 = 16$ times of composition evaluation and a total $16N$ times for all the data.\n\n#### **R2,Q2: The usage of compound rules instantiation**\nThanks for your question. As introduced in Equations (2), the matching score \"$g(.,.)$\" is the sum of the similarity scores between the unlabeled data and rules in $p_S$. Since one data could generate four reasoning rules, $p_S$ is denoted as a subset of them to form a compound rule (As introduced in Line 264-266, 280-282). Without the Compound Rules Compiler, we could not know which composition of the four reasoning rules $\\lbrace p_1, p_2, p_3, p_4 \\rbrace$ contains enough information to infer the specific relation and avoid confusion with other relations. Thus, the instantiation of compound rules is a necessary step for data labeling. We will clarify this more carefully in the revision.\n\n#### **R2: The motivation of choosing GAT to select model labeled data**\nGraph attention network (GAT) could aggregate neighbor nodes' information and pay less attention to the unimportant nodes. Since the prediction from PLM is not always reliable, and thus there are some noisy seed rules whose reasoning words may contain information irrelevant to relation inference, GAT could alleviate the noise's influence and focus on the nodes with inference features that are close to the real important features pointed by seed rules to select reliable model labeled data for new rules generation. We will further clarify this in the revision.\n\n#### **R2: How to ensure the rules diversity**\nThanks for raising this concern. First, new rules are generated from model-labeled data. Since RE model could generalize beyond the knowledge provided by the rule-labeled training data, diverse data with representation that is different from original seed rules could be utilized for new rules generation (As introduced in Line 110-113). \n\nSecond, we apply GAT to model the relevance between model-labeled data and seed rules. For each relation r, only the data with inference features close to r’s seed rules’ centroid and far from other relations’ could be chosen for further new rules generation, which means the newly added rules not only focus on the consistency with the consensus of previous seed rules, but also focus on being distinguishable from other relations’ seed rules (As introduced in Line 122-124).\n\n#### **Q3, Q4: ARIA performance on domain specific datasets and Re-TACRED**\nThanks for this useful advice, we will add ARIA’s performance under more domain specific datasets in the further version. To show the ARIA’s effectiveness, we compare AIRA with two best baselines in our paper under Chemprot with 10% training data. ARIA achieves Micro F1 1.58% higher than GradLRE and 3.46% higher than MetaSRE, which shows ARIA could transfer to specific domains easily with well performance. Besides, we also evaluate their performance on Re-TACRED: ARIA gets 2.25% greater than GradLRE and 3.97% greater than MetaSRE.\n\n| Methods     | Chemprot | Re-TACRED |\n|-------------|----------|-----------|\n| MetaSRE     | 60.16    | 66.57     |\n| GradLRE     | 62.04    | 68.29     |\n| ARIA (Ours) | 63.62    | 70.54     |\n\n#### **Q3: Five runs with random sampled training data**\nIn a semi-supervised setting, since many works report the mean performance under several runs of training and test using different random seeds [1][2][3], we follow these works’ setting for a fair and reliable comparison (As introduced in Line 423-426). What’s more, as reported in the above works and ours, stratified sampling is utilized for sampling seed data to ensure the label distribution is unchanged in different runs.\n\n#### **Q4: Difference between BERT (Row 3) and BERT w/gold in Table 2**\nThanks for your question. BERT in the 3rd row of Table 2 refers to the supervised baseline that takes BERT as the encoder and trains on only seed data (as introduced in Line 391-397), while BERT w/gold trains our RE model with both seed data and unlabeled data with gold labels (As introduced in Line 419-421). We will declare their difference more clearly in the future version.\n\n#### **Q4: Greater improvement on TACRED than SemEval**\nThanks for raising an important point. ARIA could show its advantages more obviously on complex datasets. As introduced in Line 385-389, there are more types of relations with larger label distribution bias in TACRED than in SemEval, making it a dataset that is close to real scenario and require better relation comprehension ability. \n\nFirst, our framework utilizes PLM's broad knowledge to reason rules that derive the key information related to relation inference and compose them into compound rules for fine-grained labeling among various relations. Especially in the original TACRED dataset there are many incorrectly labeled data, and they are very likely to be labeled with their actual labels by our comprehensive rules.\n\nSecond, for each relation, our Graph-based Data Filter could continuously discover data with distinguishable features to generate new rules and enrich the rules set. This could improve the rules' coverage and lead to comprehensive learning for those relation with few seed data.\n\n&nbsp;\n\n[1] Gradient Imitation Reinforcement Learning for Low Resource Relation Extraction\n\n[2] Learning Dual Retrieval Module for Semi-supervised Relation Extraction\n\n[3] Semi-supervised Relation Extraction via Incremental Meta Self-Training", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The problem statement is well-motivated and the overall description of the proposed ARIA framework is really ingenious.\n2. Improved state-of-the-art results on SemEval and TACRED datasets, followed by a meaningful qualitative and quantitative analysis.", "reasons_to_reject": "1. The paper needs to be clearer to read and follow. It misses several pieces of information that could make it much more straightforward. For example, L333-336 mentions that the initial node embedding for a data point is built by concatenating the embedding of the data's four types of reasoning words. However, Figure 3 shows that there could be multiple reasoning words for each of the four reasoning templates (Table 1). How does the aggregation take place, then?\n\n2. Specific modules of the proposed ARIA framework are introduced without motivating why they are needed. In addition, the current descriptions of these sections are difficult to follow. For example, why are the instantiation of explicit compound rules relevant for this task - especially when they are not even used in Equations (1), (2), and (3)? Why is a GAT required to select model-labeled data having inference features consistent with the seed rules? How does such a step ensure diversity, as the Introduction section claims?", "questions_to_authors": "1. In Sec 3.2, L262-263, the paper mentions \"... We ask the PLM by prompt to evaluate if each composition has enough information ....\" followed by the definition of the Composition prompt; Does this mean that the composition prompt is evaluated for all possible compositions of rules in Table 1? What kind of additional run-time complexity does ARIA incur on this step?\n\n2. The need for Sec 3.2, i.e., the need to instantiate explicit compound rules is unclear. From Equations (1), (2), and (3), we observe that, given an unlabeled sample u, whether u matched p_S (the compound rule) does not depend upon p_S but depends on p_i, i.e., the rules obtained from the meta-rule template and the filled in prompt (from the LM). So, ideally, we donot require an instantiation of the compound rule p_S, correct?\n\n3. A low-resource setting for Relation Extraction task typically occurs in domain specific datasets such as Chemprot [1], SCIERC, etc. and not on Open domain datasets such as TACRED or SemEval. How does ARIA perform on either one of them in comparison to existing SOTA approaches? Also, referring to the evaluation section, using 3% of TACRED (or 5% of SemEval) over five random runs can still have a large variance - depending upon which 3% of TACRED got sampled for training.\n\n4. What is the difference between BERT (Row 3) and BERT w/ gold labels (final row) in Table 2? Any insights into why the performance improvement on TACRED is higher than that of SemEval? Also [2] shows that a significant chunk of the most challenging sentences from the dev/test sets of TACRED are incorrectly labelled, and therefore introduce a newer annotated variant called Re-TACRED with all the fixes. How does ARIA perform on Re-TACRED?\n\n[1] Peng, Yifan, Shankai Yan, and Zhiyong Lu. \"Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets.\" BioNLP 2019 (2019): 58.\n[2] Stoica, George, Emmanouil Antonios Platanios, and Barnabás Póczos. \"Re-tacred: Addressing shortcomings of the tacred dataset.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 15. 2021.", "paper_topic_and_main_contributions": "In this paper, the authors introduce ARIA, an automatic task-specific logic rules-distilling framework that uses a large language model (LM) to continuously build high-quality rules for labeling relations. In the context of the relation extraction task, the authors argue that neither supervised methods (which require labor-intensive annotations) nor distant supervision approaches (which ignore the data context while assigning relation labels) are suitable for building relation extraction systems under a low-resource setting. Instead, a logic rule, which is an explainable way to summarize knowledge, should be used to generate labels. Humans typically create logic rules; however, ARIA suggests using LMs to build high-quality rules for discovering relations.\n\nARIA works in the following way: it learns to find high-quality rules for relations from the seed data and annotates a collection of unlabeled data, which is then fed as input to a Relation extraction classifier. The model-predicted relation extraction instances are then used to improve ARIA's generalization capability beyond the rules that it currently knows. For the first step, given a seed-labeled example, ARIA uses a collection of meta-rule templates and an LM to construct a chain of thought reasoning rules. In the next step, one or more previously generated rules are composed to yield a compound rule, which is applied to the unlabeled dataset to identify matching instances. An unlabeled example, u, is said to match a rule r if the similarity score between u and r exceeds some predefined threshold. The top N_d matchings selected from this step are fed to the Relation extraction model to build an updated RE model.\n\nThe updated RE model is then applied over the remaining unlabeled set and provides model-labeled data for discovering new rules in the final Graph-based data filtering step. To select model-labeled data consistent with the seed rules, a GAT is trained over the graph built using the seed data and the model-labeled data as nodes and cosine similarity between the nodes as edges. Finally, the filtered data from this iteration is combined with the data from the previous iteration, and the process continues.\n\nThe authors use SemEval and TACRED datasets to demonstrate the effectiveness of their approach. Moreover, they illustrate comparisons with existing state-of-the-art approaches and ablation results for the various components of ARIA.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "1. The word \"donated\" used throughout the paper is a typo. It should be \"denoted\"."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a rule distillation framework for relation extraction and comprises three components: (a) a reasoning rule generator, (b) a compound rule compiler to combine multiple rules, and (c) a relation extractor trained on both supervised and rule-labeled data. The proposed method is evaluated on sentence-level relation extraction in a low-resource setting.

The paper is moderately well-written and represents an incremental advancement in existing rule-based weak supervised methods. However, it lacks some references and semi-supervised baselines. Following discussions, the model was subsequently evaluated on two new datasets, Chemprot and Re-TACRED, as the original dataset TACRED was found to contain incorrect labels. The results indicate a slight performance improvement compared to the current baselines, and this could be further strengthened through statistical significance testing.
Decision: Accept-Findings