Generating Commonsense Counterfactuals for Stable Relation Extraction
Authors: Xin Miao
Yongqi Li
Tieyun Qian
Venue: EMNLP 2023 Main (2023)
Keywords: Counterfactual Data Augmentation
Commonsense-constrained Generation
Relation Extraction
TL;DR: We propose a commonsense-constrained counterfactual data augmentation method for relation extraction tasks.

Abstract: Recent studies on counterfactual augmented data have achieved great success in the coarse-grained natural language processing tasks. However, existing methods encounter two major problems when dealing with the fine-grained relation extraction tasks. One is that they struggle to accurately identify causal terms under the invariant entity constraint. The other is that they ignore the commonsense constraint. To solve these problems, we propose a novel framework to generate commonsense counterfactuals for stable relation extraction. Specifically, to identify causal terms accurately, we introduce an intervention-based strategy and leverage a constituency parser for correction. To satisfy the commonsense constraint, we introduce the concept knowledge base WordNet and design a bottom-up relation expansion algorithm on it to uncover commonsense relations between entities. We conduct a series of comprehensive evaluations, including the low-resource, out-of-domain, and adversarial-attack settings. The results demonstrate that our framework significantly enhances the stability of base relation extraction models.

Generating Commonsense Counterfactuals for Stable Relation Extraction
Xin Miao', Yonggi Li', Tieyun Qian 1.2*
School of Computer Science, Wuhan University, China Intellectual Computing Laboratory for Cultural Heritage, Wuhan University, China {miaoxin, liyongqi, qty edu.cn
Abstract
Original: this movie so bad.
Label: negotive
Recent studies on counterfactual augmented data have achieved great success in the coarse- grained natural language processing tasks. However, existing methods encounter two major problems when dealing with the fine- grained relation extraction tasks. One is that they struggle to accurately identify causal terms under the invariant entity constraint. The other is that they ignore the commonsense constraint. To solve these problems, we propose novel framework generate commonsense counter- factuals for stable relation extraction. Specif- ically, identify causal terms accurately, introduce an intervention-basee leverage constituency parser for correction. To satisfy the commonsense constraint, in- troduce the concept knowledge base WordNet and design a bottom-up up relation expansion al- gorithm on it to uncover commonsense rela- tions between entities. We conduct a series of comprehensive evaluations, including the low-resource. out-of-domain, and adversarial- attack settings. The results demonstrate our framework significantly enhances the stability of base relation extraction models'.
Counterfactual: this movie good.
Label: positive
Original: eggs moved into buckets.
Label: entity destination
Counterfactual: eggs were caused by buckets.
Label: effect-couse (without commonsense constraint)
Counterfactual: eggs were contained in buckets.
Lobel: content-contoiner (with commonsense constraint)
Figure I: Counterfactuals in sentiment amalysis (A) and relation extraction (B) tasks. The words blue denote causal terms, and those in orange denote entities.
strategy and
Instability has always been associated with neu- ral networks. For example, Li et al. (2021) finds that BERT performs poorly in the de-biased set con- structed by replacing high- fegh-frequency words with low-frequency ones. The instability is primarily caused by spurious correlations between the high- frequency words and labels. Specifically, this can be attributed to neural networks tending to learn shortcuts between instances and labels (Geirhos et al., 2020). Such property may hurt the model's stability some challenging scenarios, e.g., low- resource or cross-domain scenarios.
1 Introduction
The relation extraction (RE) task aims to extract the semantic relation between entities in the text. RE can facilitate wide range of downstream applica- tions such as knowledge graph construction (Gajen- dran et al., 2023) and question answering (Li et al., 2022b), having aroused much attention in recent years. Typical RE methods based on pre-trained language models like BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019). While getting im- pressive performance, they exhibit certain degree of instability in RE (Li al., 2021; Chen al., 2023), even ChatGPT (Wei et al., 2023).
In attempts to mitigate spurious correlations, counterfactual augmented data (CAD) is rising trend. CAD can be defined flipping the label of instance with minimal editing (Kaushik al., 2019). which explicitly provides localized views of decision boundaries (Treviso et al., 2023). Cur- rent CAD methods consist of the following three steps: (1) identifying causal terms, that are causally related to the label; (2) editing the causal term, to flip the label to other prepended ones; and (3) filter- ing out inconsistent instances, where the predicted label is different from the prepended one.
Existing methods have made great success coarse -ourse-grained natural language processing tasks like sentiment analysis (Yang et al., 2021; Chen
"Corresponding author.
'The code and data used in our experiments are available at: https: /g thub. con/NL WWWPMMMMHHUUE WHU/CCG

---

al., 2021; Howard et al., 2022) and natural lan- guage inference (Wu et al., 2021; Ross al., 2021; Wen et al., 2022). However, the fine-grained RE task is ender-explored (Zhang et al., 2023). Due to the intrinsic property of RF, i.e. invariant entity constraint and commonsense constraint, existing methods encounter the following two challenges.
structure knowledge i.e. the common hypernyms. to interconnect entities, thereby expanding their re- lations. We design bottom-up relation expansion algorithm for implementation, which can uncover commonsense relations between entities.
We conduct comprehensive evaluations on a se- ries of challenging scenarios, including the low- resource, out-of-domain, and adversarial- set- tings. The results demonstrate that our CCG gen- erates more reasonable counterfactuals, which can consistently enhance RE models' stability in var- ious scenarios. Furthermore, we also conduct a series of in-depth analysis studies. The results con- firm the effectiveness of various CCG strategies and demonstrate that our generated counterfactu- als not only semantically readable but, more importantly, consistent with commonsense.
Firstly, existing methods struggle accurately identify causal terms, due to the constraint of invari- ant entities (Zhang al., 2023). That is, keeping the entities unaltered. The state-of-the- art methods primarily employ the model's gradient attributions to identify the causal term (Ross et al., 2021; Wen et al., 2022), i.e. selecting certain words with the highest gradient. However, entities convey more information (Zhang al., 2023), which implies that the gradient of causal terms can be confounded by the related entities. Zhang al. (2023) propose a syntactic-tailored strategy, i.e. taking the words along the shortest dependency path (SDP) between entities as causal terms. The drawback is its heavy reliance on syntactic parsing quality.
Related Work
2.1 Relation Extraction (RE)
Early deep learning RE models get promising per- formance by either extracting better semantic fea- tures from sentences (Zhou et al., 2016; Zhang et al., 2017) or incorporating syntactic features over dependency graph (Guo et al., 2019; Mandya al., 2020). More recently, pre-trained language models (PLMs) (Devlin et al., 2018) based RE methods can achieve superior performance (Yamada et al., 2020; Qin et al., 2021; Chen et al., 2022).
Secondly, and more importantly, all existing methods ignore the commonsense constraint. They assume that all labels can be treated as prepended ones. This is feasible coarse-grained tasks. For example, in sentiment analysis, simply replacing orientation, the causal terms can flip the sentiment
as shown in Fig. 1 (A). However, for RE, the nature of entities imposes commonsense constraints on causal terms, as shown in Fig. (B). In brief, the relation between entities should conform the cog- nitive understanding of the real world. Although existing methods pre-train RE base model on the existing dataset for consistency filtering (Ross et al., 2021; Wen et al., 2022: Zhang et al., 2023), there remains an issue of entanglement with the base model. For example, in low-resource scenar- ios, the base model is severely affected by spurious correlations and propagates the errors backward.
Despite the remarkable progress, the stability of RE methods has been largely neglected from the perspective of CAD. The only research towards this direction (Zhang al., 2023) ignores the common- sense constraint. Moreover, its performance is en- tangled with the base model due to the consistency filtering. In contrast, our CCG takes commonsense into account and is independent of the base model during the counterfactual generation process. This ensures that our counterfactuals are reasonable and not constrained by the base model.
To address these two challenges, we present a Commonsense Counterfactual Generation (CCG) framework for stable relation extraction. Firstly, to identify the causal terms accurately, we present a novel intervention- based strategy, which can ac- curately identify the editable words by potential interventions. After obtaining preliminary results, we further utilize the constituency parser for cor- rection. Secondly, to satisfy the commonsense con- straint, we leverage the concept knowledge base WordNet (Miller, 1995) and utilize its hierarchical
2.2 Counterfactual Data Augmentation
Inspired by causal inference (Pearl, 2009; Pearl and Mackenzie, 2018), studies to improve model stability by eliminating spurious correlation fea- tures have received increasing attention (Kaushik et al., 2019; Niu et al., 2021). Among them, coun- terfactual data augmentation methods have become one of the mainstreams, especially in natural lan- guage understanding tasks (Kaushik et al., 2019; Garg and Ramakrishnan, 2020; Wang and Culotta,

---

2021; Ross et al., 2021; Robeer et al., 2021; Chen al., 2021; Yang et al., 2021; Wen al., 2022; Howard et al., 2022: Zhang et al., 2023). These methods generate augmented data by replacing the causal terms in original samples and flipping the corresponding labels. In this way, the impact of causal items on the output is emphasized while the importance of spurious correlation features is weak- ened (Kaushik al., 2019; Joshi and He, 2021). However, most of these approaches have been performed on coarse-grained natural language un- derstanding tasks, such as sentence-level sentiment analysis (Kaushik et al., 2019; Chen et al., 2021; Yang et al., 2021; Dixit et al., 2022; Howard et al., 2022), text classification (Garg and Ramakrish- nan, 2020; Wang and Culotta, 2021), and natural language inference (Wen et al., 2022; Dixit al., 2022). For fine-gras fine-grained tasks like relation extrac- tion, one key property the commonsense asso- ciated with the entity pair. If the commonsense constraint is violated, the obtained counterfactuals may negatively affect the model's s performance and stability. Based on this observation, this work ex- plicitly introduces commonsense knowledge into the counterfactual generation for the first time.
3.2 Causal Terms Identification
To meet the requirement of minimal perturbations. the causal terms should he identified precisely. Oth- erwise, the emrors will affect the later counterfactual generation. To this end, we propose an intervention- based strategy combined with constituency parser. The details are shown in Figure 2 (1).
MLM Intervenor We first employ a masked lan- guage model (MLM) as the intervenor. It inter- venes on each contextual word separately. Specifi- cally, it masks each word individually using <mask> symbol and then performs cloze-style filling. To ensure the rationality of each intervention, we only select the top A words predicted by the intervenor.
Trained Indicator The indicator is a trained RE- based model based on the existing dataset. It pre- diets each intervention instance separately. If a predicted label changes, we regard the intervened word as the preliminary causal term. It is worth noting that even in low-resource scenarios, the base model can still work because the decision space for determining the change of predictions (binary clas- sification) is much smaller than the dccision space for predicting the actual outcomes (multi-class clas- sification, directly proportional to the number of relations), which greatly reduces the complexity of the problem. The process can be formalized as:
3 Methodology
3.1 Task Definition
Let X = {(2 = (E), (.,, be the dataset, where E X is sentence contains known entity pair 6 and their context Ç₁ Yi E Y is the corresponding relation of the entity pair. Given the sentence Ti, RE aims extract the relation y/-
1, b(x,)) + b(a (xAuu(r((!))
o(w()
(I)
0, otherwise,
where r r(w:) denotes the intervention word, 2; w represents the sentence 2; without the word w/. and 6 is the trained indicator.
For a given instance (x; = (e,, (i), vi). we de- fine the commonsense counterfactual augmented data (CCAD) as: generating a counterfactual in- stance (& = (e;, 4), (k) that meets the following requirements. (1) Minimal perturbation: only the causal term in is edited. (2) Commonsense con- straint: di needs satisfy the commonsense con- straint regarding e and d F yi- (3) Label flipping: d; needs to be consistent with ÿi- Note that re- quirements (1) and (3) are inherited from previous works (Kaushik et al., 2019; Garg and Ramakrish- nan, 2020; Wang and Culotta, 2021). and require- ment (2) is proposed by this work.
Constituency Parsing based Causal Item Iden- tification After obtaining the preliminary causal term, We observed that words belonging to the same phrase as the preliminary causal term were not identified, although they should be treated as a whole. To alleviate this issue, we utilize a con- stituency parse tree to find the overlooked causal term. To achieve automatic operation, we design the processing workflow: (1) traverse upward from the node where the preliminary causal term are located to find the nearest verb phrase (NVP) prepositional phrase (NPP). (2) If NVP is found, traverse its child nodes and label the verb (VP) or preposition (IN) nodes as causal items, then finish. (3) If NPP is found, traverse its child nodes and label the preposition (IN) nodes as causal items.
Our framework CCG consists of three compo- nents: causal terms identification, relation expan- sion, and controllable editing. Each module satis- fies one of the aforementioned requirements. The overview CCG is shown in Figure 2.

---

Causal Terms Identification
Relation Expansion
Original SEES were removed into buckets.
Another: the pizza dough
bin.
A
Relation:
destinaitnn
A
Rolation: content-contalner
being
dough
concoction
foodstuff
Counting
Fomovad
thrown
MIM Intervenor
bin
Htperrymm
container
artifact
Intervention
B
eggs were removed from buckets.
ea2t are removed into buckets,
Prediction
OR Indicator
Trained
B
Bottom-- Retrieval
foodstuff-vene vencittff-ueceil
concoctian-cottanee
RelationNet
CRSS removed buckets.
foodstuff container
food-vessel
contalner [content contaiiien 3)
J Constitency Parser
food
foodstufff- (contant container:
SM1 were removed into buckets.
UP
Mininal Verb Phrase
VBD
%
foodstuff
food
Potential Relations
Hore
Wordheet Hypernymo
ramooved
bucket
container
Controlled Editing
egg: were <masko <mask> buckets. intity-destination
Egge were removed into buckets. (ineit
epps cmasko <masko buckets. content-contoleer
Editor Model
Counterfoctuaf: eges were contained in buckets. (Inference)
Figure 2: An overview of CCG framework, which consists of three modules, i.e., causal term identification, relation expansion, and controlled editing. The numbers 1, 2, and 3 in the colored circle denote the module id.
3.3 Commonsense Constrained Relation Expansion
its occurrence count are accumulatively recorded the corresponding values. The entire process be formalized in lines 1-9 of Algorithm 1.
To meet the requirement of commonsense con- straint, we incorporate a concept knowledge base WordNet to expand the relations among entities. Specifically, to achieve relation transitivity, it is necessary establish connections between entities. Hypernymy (super-name)
Bottom-up Retrieval We also obtain the hyper- nyms of entities through WordNet. Then, pair- wise match the hyperyms between entities in bottom-up strategy. That is, priority is given the lowest-level hypernym pair, and then proceed- ing using the similar strategy iteratively. According to the matching order, each hypernym pair is used to retrieve records from RelationNet. If record exists and contains a new relation, we add it to the potential relations list. Due the lower common hypernym in the hierarchical structure, the closer their semantics, our bottom-up retrieval strategy can proactively identify semantically closer entity pairs with different relations. As a result, the rela- tions discovered earlier come from the entity pairs with more similar semantics, hence should be given higher priority. We select the top K. rela- tions for experimentation. A more detailed process is described in lines 10-25 in Algorithm 1. Note that the ratio threshold H can be adjusted control the scope of upward retrieval. yield more accurate relations but less data, while a larger H. will have the opposite effect. We will discuss the impacts of these parameters in $5.8.
transitive relations between concepts (Miller, 1995), e-g., "primate" is a hypernym for "human". This semantic relation organizes the meanings of nouns into a hicrarchi- cal structure. Moreover, the concepts gradually become broader in the hierarchy of hypernyms. Based on this property, entities can be connected through common hypernyms. Furthermore, sume that for any entity pair, the lower common hypernym in the hierarchical structure, the closer their semantics are, and the more likely their rela- tions can expanded. We implemented this idea through Module 2, as shown in Figure (2).
RelationNet Construction Given known in- stance, first obtain the hypernyms of entities through WordNet. Then, we pairwise match the hy- peryyms between entities to form hypernym pairs. Finally, we respectively use the hypernym pairs as keys in RelationNet, while the current relation and
smaller H will

---

Algorithm 1: Bottom-up Relation Expansion.
experiments on the low-resource SemEval (Hen- drickx al., 2019) under two scenarios i.c. 1%, 3%, 5%, 10% (Li et al., 2022a) and 2-shot, 4-shot, 8-shot, 16-shot, 32-shot (Chen et al., 2022).
Require:
Thain set {(2, = (f), (.).)) WordNet
W, Input instance (x = e2), v). Ratio threshold
H. Top K n;
Output: Potential relations y {1h -ind.
RelationNet R.
4.2 Out-of-domain Setting
1: for (el. <f) in X
H. = W(e
Spurious correlations also exist in out-of-domail settings, due to domain-specific tokens (Calderon et al., 2022). To validate CCG's performance in this scenario, following previous work (Zhang et al., 2023), employ the ACE 2005 dataset (Gr- ishman et al., 2005) for evaluation. We select four sub-datasets from different domains, includ- ing weblogs (WL., conversation (BC), broadcast news (BN), and newswire (NW). Specifiiaa we conduct experiments with WL-BB,, WL-BN and WL-NNW settings, respectively. Taking the WL-BBC setting as example, we use WL as the training set, and BC as the test set.
H² W(e
for h! in H. do
for hē in II do
R= [(h), h2) (u 1)}:
end for end for end for
10: Hi W(e)):
11: H2 W(ez):
12: for hop rlene H²) H) do
13: H {[hops. hop2) (hops + hop2
hop) (hop: 0) (hops 2 0)}:
14:
for (hopi, hop2) in
15:
h1 Hi [
16:
h2 = Ha hhopal:
17:
Dnop = Dnop (R]/
18:
end for
19:
sort Yhopi
4.3 Adversarial-attan Settings
20:
for Dhop in Ynop do
21:
if Whop * y and Dhop in y then
Adversarial attacks are common way for evalu- ating model robustness (Lin et al., 2021). We con- struct an adversarial-attack test dataset based the original test set of SemEval, namely RE-Attack?. In general, we employ a semi-automatic approach to intervene on causal terms and entities separately, generating perturbed instances with flipped and in- variant labels. Table 7 shows the detailed statistics of the datasets used in our experiments.
22:
- lshop]:
23:
end if
24:
end for
25:
end for
3.4 Controlled Editing
To meet the requirement of label flipping, need to edit the causal term conditioned the uncovered new relation. The process can be re- garded as controlled editing. We employ a prompt learning (Carlsson et al., 2022: Yang et al., 2022) method to accomplish this goal. Our proposed method consists of two stages, including fine- tuning and inference, as shown in Figure 2 (3).
4.4 Baselines
The comparative methods include the following three types. The first type, conventional methods: (I) Synonym Replacement (Zhang et al., 2015) replaces words with synonyms; (2) Back Trans- lation (Sennrich et al., 2015) translates text into another language, then translates it back; (3) BERT- MLM (Jiao et al., 2019) replaces words by MLM.
At the fine-tuning stage, put the available data into the pre-defined template and use them to fine-tune a generative model. Then, at the inference stage, we input the prompt that contains the new relation to the fine-tuned generative model and let it generate the content. Note the new relation is just the prompt, and the goal is to generate the masked tokens for satisfying this prompt. Finally, we treat the generated content the counterfactual.
The second type, counterfactual methods: (1) MICE (Ross et al., 2021) identifies causal terms by gradients and trains editor to edit them; (2) AutoCAD (Wen et al., 2022) is similar with MICE, but it introduces unlikelihood strategy for the ed- itor; (3) CoCo (Zhang al., 2023) exploits syn- tactic and semantic dependency graphs to discover substitution. They all require consistency filtering- The third type is LLM, we use ChatGPT, the most powerful LLM. We carefully designed a prompt to guide its generation of counterfactuals.
4
Evaluation Protocol
4.1 Low-resource Settings
Spurious correlations are particularly prevalent in low resource settings (Nan et al., 2021). To validate that CCG can mitigate such impact, we conduct
2To the best of our knowledge, RF-Attack is the first datasct for adversarial- -ttack testing in RE.

---

1%
3%
5%
10%
2-shot
4-hhot
S-sbot
16-shot
32-shot
Method
R-BERT
Original
33,26,
59.311,0
68.66,
76.47
23,48,
34.65y,39
50.26,,1 1s
63.19
75.58u
Synonym Rep.
29.78261
60.06
69.57
23.48).m
35.78
51.171.1
64.37aas
76.03
Back Trans.
27. 26
56.08
64. 22.02
75.5 15.53
2..5
28.362.0
43.30pus
61 61.80
14.381.4
BERT-MIM
36,46g.19
62.00g.s
67. 67.97
77.246.00
26.09%.90
33.460,00
48. 48.542,38
60.661.9
75.066.37
MICE
62. 64o.
72.51o.
77 2..22..s
50.04,22
69:92nat
75.78n.51
AutoCAD
18.
71,81
77. 77.10s.4n
50.412.09
70.23
69..
CoCo
62.24
69.97
77 77.40p.0r
49.05
65 65.65
26,38jur
CharGPT
38.78am
21.844,23
67. 67.90
75.15
26.29,.a m2
36.73
54.31
63. 63.19
72.691.10
CCG
42.66
66.30
73.01c.0
78.088,
35.1
47.52
61.87
69.12p
76.8%u
R-RoBERTa
Original
35.77
64,27
69.99,
78.27
30.26,
40.73
55.061
66.22
76.680.s
Synonym Rep.
31
62, 32
71.244.73
78.51y.7
27.18
37.174.19
55.0%
67.71
76. 76.14
Back Trans.
29.79
k1.10
68.00
78.23
26.81
30.233.47
50.93,m0
76.23u.1
BERT MLM
38.24gm
63.90
70,28..0
77.65
30.03
35.24k.6
53.37%.0
64.50u
75.78u
MICE
68.421.s
74.76,
78.21o.m
$4,03
70.77
76.20nar
AutoCAD
67,5
74.746.79
78.276,7
$4.116.52
71 71.62n.5
76.81nss
CoCo
65.572m
74.116..1
78.40p.s8
$4.08g.06
67.78a.s
77.201aa
38.71a
64.44,,a4
70.14..11
76.256.52 02
27.00
39. 39.874.14
56.34b
65.25189
73.950.0
70.16
75.32
79.29 m1
49.53
61.64
72.35
77.63 c0
Table 1: Results for low-resource settings SemEval in terms of Fi-scores. Scores in bold indicate the best result, and the underlined ones are the second best. The subscript denotes the standard deviation. "-" indicates there is no augmented data due the base model being undertrained and not performing proper consistency filtering.
WL + BC
WL
WL NW
BC
WL BN
WL - NW
Method
R-BERT
R-RoBEKTa
Original
70.432.44
70.552.51
69.421.41
74.170.70
70.54
74.93a.74
Synonym Rep.
71.77
72.76n.72
70.38
74.48p.80
21.75
74.43
Back Trans.
71.762.18
72.821.74
70.25
74.951.23
71.641.9
75,280.00
BER
71.881.66
71.680.81
70.49.
73.541.14
69.75a.a
73.42a.48
MICE
70.421.60
70.72 .32
69.51
73.531.33
70.35
73.86
AutoCAD
70.761.14
71.971.34
69.970.77
74.321.04
71.230.72
73.80u.52
CoCo
69.951.14
71.171.25
69.03
73.79
70.00
73.561.10
ChatGPT
52.70k.99
55.941.21
54.51p.3
59.55p.50
60.11a.m
61.741.13
CCG
73.160.80
73.11
70.69p:91
75.55p,75
72.110.89
76.000.53
Table Results for out-of-domain settings on ACE 2005 in terms of Fl-scores.
4.5
Implementation Details
All reported results are the mean and the standard deviation of micro-F1 value over random seeds. For more implementation details, please check $A.
For our method, we use BERT-base as the inter- venor and set N to 100. The constituency parser is from CoreNLP², To ensure that at least one po- tential relation can be found for each instance, we set 7 to 0.8 in all experiments. To ensure quality, we set K. to 1. For fair comparison, all methods employ GPT-2 base (Radford al., 2019) editor. Following (Zhang et al., 2023), we use two rep- resentative PLMs i.c. BERT-base and RoBERTa- base as the base model for RE. To better fit the task, the specific implementation details follow R- BERT (Wu and He, 2019). We use the development set to select the optimal epoch for the base model.
Results and Analysis
5.1 Results for Low-resource Settings
The results for low-resource settings are shown in Table 1. We make the following observations. 1) CCG achieves the best performance with relatively small standard deviations in almost all cases. The noteworthy point is that the improvement becomes more significant when the data size decreases. For example. the available data decreases from 10% to 1 %,, CCG gains about 1 to 9% enhancements. 2) The conventional methods exhibit unstable per- formance, which may lead to negative impacts. 3)
github.io//oreNLL. Lo/CoreLLP/ "https://huggiingface
co/bertt basterttbbse-ueee base-bsseeee uncased baseuuaaaeed 'https://hugging: co/roberta- baase

---

Method
R-BERT
R-RoBERTa
R-BERT
Method
Original
53.341.78
64.161.19
10%
32-shot
WL - BC
Adv-attack
Synonym Rep.
55.762.07
65.77p.89
Back Trans.
1..61
60.241.20
CCG
78.080.co
76,88p.71
73.16n.c
70.48127
BERT MLM
37.85p.87
51.38p.70
w/ MIl
76.33a.3
76.14 76.141.08
70.23p.88
68.01g.56
MICE
62.569.68
72.770.62
who Parser
77.63074
76.87n.44
72.480 01
69.36 as
AutoCAD
63.79g.s8
73.45m.72
w/o LCA
76.52u.55
75.05u.83
71.17..1
68.79
CoCo
59.971.35
71.29p.96
WBR
76.55
73.83p.38
71.641.0
68.701.10
ChaiGPT 56.15
65, 65.781.31
CCG
70.481.27
77.19
who Editor
77.04a.32
72.89p.7a
66.501.40
65.921,22
Table 3: Results for adversarial-attan settings on RE- attack in terms of Fl-scores. Note that for computational consumption reasons, the augmented data for ChatGPT is obtained by merging data from previous experiments.
Table 4: Results in terms of Fi-scores for ablation study, including three settings above mentioned. The best and second best results in bold and underlined, respec- tively. The subscript denotes the standard deviation.
The counterfactual methods encounter a big chal- lenge under extremely low-resource settings, e.g.. 1% and 2-shot scenarios. This is because limited available data prevents the base model from work- ing properly in consisteney filtering. 4) Although ChatGPT possesses certain counterfactual reason- ing capabilities in low-resource settings, it achieves suboptimal results. After observation, we find that it lacks proficiency in exccuting relation expansion.
5.4 Ablation Study
To validate the effectiveness of the main compo- nents in CCG, we introduce the following variant baselines for the ablation study. 1) CCG w/o MII re- moves the MLM Intervenor based Indicator (MII) and replaces it with et al., 2022). 2) CCG w/o Parser removes the con- stituency parser that is used for alleviating the prob- lem of missing causal words. 3) CCG w/o LCA re- moves the Lowest Common Ancestor (LCA) strat- egy when discovering potential counterfactual re- lations. 4) CCG w/o WBR removes the WordNet- based Bottom-up Retrieval (WBR) and replaces it with random selection strategy. 5) CCG w/o Editor removes the controlled Editor and directly fills the blank with the most frequent causal term in the dataset corresponding to the target relation. The experimental results are shown Table 4. From Table 4, we can observe that, 1) Remov- ing any of the modules or strategies causes perfor- mance degradation, which shows that every mod- ule in our approach plays an important role. 2) Removing MII or Parser results in up to 2.9% per- formance degradation, which further emphasizes the importance of identifying causal terms precisely and demonstrates the effectiveness of our approach to improve causal term identification. 3) Removing LCA or WBR results in 2%-3% decrease, which shows the importance of the hierarchical structure of WordNet for finding target relations that con- form to commonsense. 4) Removing Editor results in the most performance degradation, up to 6.3%, as using fixed causal terms introduces new biases.
basde method (Wen
5.2 Results for Out-of-domain Settings
Table 2 reports the results for out-of-domain set- tings. We make the following observations. 1) CCG outperforms all comparative methods in all target domains while maintaining relatively small standard deviations. 2) The conventional meth- ods perform well when data abundant, e-g., Back Trans. can achieve suboptimal results in WL-BC and WL-NNW settings. 3) The enhance- ment brought by the counterfactual method is quite limited, the reason is that the base model is influ- enced by in-domain spurious correlations, making it once again unable to perform proper consistency filtering. 4) It is interesting that ChatGPT is always the worst, showing that it is unable to adapt itself in the settings for the counterfactual generation task.
5.3 Results for Adversarial-attack Settings
The results for the adversarial-attack settings are shown in Table 3. We make the following obser- vations. 1) CCG achieves the best performance among all methods. The enhancement is quite significant, approximatcly 13-17%. 2) Generally, the counterfactual methods are superior to the con- ventional ones, which indicates that counterfactual augmented data provide more reliable assurance in complex and extreme scenarios, driving the base model to lean towards learning causal features.
5.5 Human Study
Since it is not possible to automatically evaluate the model's ability of causal terms identification and relation expansion, we conduct a human study.

---

Case
Moby Doll was first captive orca displayed public aquarium exhibit
Component Whole
AutoCAD
Moby Doll the first captive caused in public exhibit x
Effect-Couse Couse
CoCo
Moby Doll was the primary primary orea generated by The primary electron exhibit
Effect-Coue Cause
ChatGPT
Moby Doll the first captive
kept in a public aquarium container... X
Product-Container
Container
CCG
Moby Doll was the first captive orca to public aquarium exhibit o
Entiry Desiiy-Desiinatinn
Case
Colonial families the United States descended from the immigrants
Oriii-
AutoCAD
Colonial families discuss United States from the immigrants x
Message-Topic
CoCo
Colonial families of the United States enclosed in the immigrants. x
CharGPT
Colonial families the United States belonged to the immigrants.
Collection Member
CCG
Colonial families of the United States of the immigrants
Collection-Member
Table 5: Abbreviated instances for case study. Entities are in orange causal terms are in blue Red denotes missed causal terms, V denotes compliance with the defined requirements, while the X is the opposite.
CCG
CoCo
AJACCAD
NICE
AutoCAD
CoCo
CG6
ChanGPT
90
#1-3core (%)
80
Average e Score
120
114
75.59
76
90
Quantity
Scote
53.77
50
Causal Term Validity
Commonsense Rating
60
Figure 3: Results of buman study. We apply the macro- Fl score for evaluating causal terms identification. To evaluate whether the generated counterfactual conforms to commonsense, we divide the instances into three categories and assign corresponding scores: aligned (2), marginally aligned (1), and not aligned (0).
41
35
Grammatical Suggestions (1)
Overall Quality (1)
Figure 4: Results of readability study. Grammatical sug- gestions denote the number of grammatical suggestions by Grammarly, and overall quality denotes the overall quality of writing in this document including readability. indicates a lower value is better, and indicates a higher value better. For example, ChatGPT achieves score of 76, indicating that the text by ChatGPT is better than 76% of all texts checked by Grammarly.
Specifically, we randomly select 100 samples from SemEval and evaluate counterfactuals generated by CCG, CoCo, and AutoCAD by using manual annotation. The results presented in Figure We can observe that, 1) CCG substantially out- performs existing methods in finding causal terms, indicating that our proposed causal term identifi- cation module aligns best with the human experi- ence. 2) Regarding the evaluation of whether the generated counterfactuals consistent with com- monsense, only CCG scores above 1, i.e., above the level of "marginally aligned".. This shows that our proposed caaamonsens--onstrriinne relation expunsion module effectively improves the consis- tency of counterfactuals with human commonsense, resulting in more reasonable counterfactuals.
sense constraint. For example, Case 1, there a low likelihood of a Effect-Cause relation between entities "orca" and "exhibit". Furthermore, Auto- CAD overlooks partial causal words in Casc and Case 2. CoCo is influenced by syntactic parsing er- rors in Case 1, which reduces the readability of the sentence. ChatGPT either generates illusory rela- tion i.e. non-existent relation in Case or reverses the relation in Case
5.7 Readability Study
To thoroughly evaluate the quality of generated counterfactuals, we conduct readability study to verify grammatical correctness and semantic read- ability based Grammarly (Grammarly, 2023). Grammarly is prevalent English typing assistant that reviews spelling, grammar, punctuation, clarity, engagement, and delivery mistakes in English texts, detects plagiarism, and suggests replacements for the identified errors (Wu et al., 2023).
5.6 Case Study
To have close look, we randomly select two instances from SemEval for the case study. Ta- ble 5 shows the original and augmented instances by CCG and other three typical methods. Clearly, the CCG counterfactuals conform to commonsense and align well with labels. In contrast, the counter- factuals of AutoCAD or CoCo violate the common-

---

RBEERT Ro
Method
R-BERT
R-RoBERTa
Original
88.070.47
88.22p. 11
F1-Score (%)
79
MICE
88.166.25
87.85p. 49
78
AutoCAD
88. 18.38
87. 87.96p.52
78-
CoCo
88.22
88.320.31
76
ChatGPT 87.521.18
87.766.26
77
CCG
88.31o.16
88.450.35
K
Table 6; Results conventional setting study on Se- mEval in terms of Fi-scores. Numbers in bold indicate the best results, and the underlined ones are the second best. The subscript denotes the standard deviation.
Figure 5: Results of parameter study. We report the results in 10% setting on the SemEval dataset. H de- notes the ratio threshold to control the scope upward retrieval in relation expansion. I denotes the pick-up number in the ranked potential relations, i.e., top K..
ting, we conduct a conventional setting study, The results presented in Table 6. Consistent with the previous experimental findings (Kaushik al., 2019; Sen al., 2021: Wang and Culotta, 2021; Geva al., 2022), the conventional setting cannot appropriately validate the effectiveness of coun- terfactuals. The benefits of counterfactuals, from either CCG or CoCo, are quite small under this set- ting. Other methods even have the opposite effect. In this situation, the spurious correlations present in the test set usually contained within the training set. This means that the spurious corre- lations can assist the model in finding shortcuts and improving accuracy (Sen et al., 2021). There- fore, when counterfactuals block spurious corre- lations, they may not help the model in terms of accuracy and could even have counterproductive effect (Kaushik et al., 2019; Sen et al., 2021; Wang and Culotta, 2021; Geva et al., 2022).
Specifically, we randomly select 100 generated counterfactuals on SemEval from each method. We complete doc-
then treat these counterfactuals ument and employ the Grammarly tool to check it. The results are shown in Figure 4. The results demonstrate that the grammatical correctness and semantic readability of CCG are only inferior to ChatGPT, but better than all other compared meth- ods. After all, within an acceptable range of read- ability, what we focus on more is commonsense.
5.8 Parameter Study
The parameters H and IC affect the amount of data augmentation, and we conducted parameter study and show the results in Figure 5. Increasing the value of H. indicates an expanded scope for upward retrieval in hypemyys, allowing more entity pairs to uncover potential relations. This continuously improves the model performance within a certain range. However, after reaching certain threshold, they all start to decline, which validates that aug- menting the training set with small portion leads to more robust models (Treviso et al., 2023).
Conclusion
In this puper, we solve the problems in existing methods of CAD, i.e. struggling to accurately iden- tiy causal terms under the invariant entity con- straint and ignoring the commonsense constraint. We aim to produce the most human-like counter- factuals, i.e., not only grammatically correct and semantically readable but also consistent with com- monsense. To this end, we need to satisfy the vari- ant entity and commonsense constraints. To meet the first one, we present a novel intervention-base strategy, which can accurately identify the editable words by potential interventions. To meet the sec- ond one, we exploit WordNet to expand potential relations such that the counterfactual generation is constrained by commonsense. Extensive experi- mental results prove that our framework generates commonsense counterfactuals and outperforms the state-of ofie--oftteer t-ehhartt baselines. also consistently en- hances the base models' stability across all settings.
On the other hand, when more potential relations are taken into account, i.e. increasing the value of K. the performance of the base model shows a continuous downward trend. This phenomenon illustrates that the potential relations discovered earlier are more commonsense, CCG can correctly rank potential relations aligned with commonsense.
5.9 Conventional Setting Study
For common datasets, the quantity ratio of the train- ing set the test set is significantly larger than I (e.g. 2.65 in SemEval) and the distribution com- plies with the IID (Independent Identically Distri- bution) assumption. To explore the effectiveness the main counterfactual methods under this set-

---

Limitations
for relation extraction. In Proceedings of the ACM Web Conference 2022, pages
Despite the effectiveness of the relation expansion module for mining commonsense relations in CCG, the granularity of WordNet can affect the accuracy static knowledge
Xuanting Chen, Junjie Ye, Can Zu, Nuo Xu, Rui Zheng, Minlong Peng, Jie Zhou, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023. How robust gpt-3.5 to pre- decessors? a comprehensive study on language un- derstanding tasks. arXiv preprint arXiv:2303.0000 002933
of the results, and WordNet graph also faces the issue of temporal relevance, which makes it challenging for our method to han- dle emerging concepts. Therefore, we plan to inte- grate the principles of our algorithm with LLMs at a deep level, as these models encompass rich knowl- edge. Another limitation of our work is that CCG is still pipeline framework. Some recent research attempts to jointly optimize a rationale extractor and classifier in an end-to-end fashion (Paranjape al., 2020; Yu et al., 2021). We plan to explore this direction.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training deep bidirectional transformers for language understand- ing, arXiv preprint 0rXiv:/810.04803.
Tanay Dixit, Bhargavi Paranjape, Hannaneh Hajishirzi, and Luke Zeitlemoyer. 2022. Core: A retrieve-then- edit framework for counterfactual data generation. arXiv preprint arXiv: 2tt::22210.0445 10.04873.
Sudhakaran Gajendran, D Manjula, Vijayan Sugumaran, and R Hema. 2023. Extraction of knowledge graph of covid-19 19 through mining of unstructured biomedi- cal corpora. Computational Biology and Chemistry, page 107808.
Acknowledgments
This work was supported by a grant from the Na- tional Natural Science Foundation of China (NSFC) project (No. 62276193). It was also supported by the Joint Laboratory on Credit Science and Tech- nology of CSCI-Wuhan University.
Siddhant Garg and Goutham Ramakrishnan. 2020. Bae: Bert-based adversarial examples for text classifica- tion. arXiv preprint arXiv:2004.0199
Robert Geirhos, Henn-Kerikk Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. 2020. Shortcut learning deep neural networks. Nature Machine Intelligence, 2(1 11:665-6773 673..
Ethics Statement
Our work aims to explore the commonsense coun- terfactual generation, which is entirely at the methodological level and therefore does not have any negative social impact.
Mor Geva, Tomer Wolfson, and Jomathan Berant. 2022. Break, perturb, build: Automatic perturbation of rea- soning paths through question decomposition. Trans- actions of the Association for Computational Linguis- rics, 10:111-124
References
Grammarly. 2023. http://www.gramaaty.cooom//
Nitay Calderon, Eyal Ben-David, Amir Feder, and Roi Reichart. 2022. Docogen: Domain counterfactual generation for low resource domain adaptation. arXiv preprint arXiv:2202.1 12350.
Ralph Grishman, David Westbrook, and Adam Meyers. 2005. Nyu's english ace 2005 system description. ACE, 5.
Zhijiang Guo, Yan Zhang, and Wei Lu. 2019. Atten- tion guided graph convolutional networks for relation extraction. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 241-251, Florence, Italy, Association for Com- putational Linguistics.
Fredrik Carlsson, Joey Öhman, Fangyu Liu, Severine Verlinden, Joakim Nivre, and Magnus Sahlgren. 2022. aen--grained controllable text generation using non- residual prompting. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6837- 6857.
Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid O Séaghdha, Sebastian Pad6. Marco Pennacchiotti. Lorenza Romano, and Stan Szpakowicz. 2019. Semeval-201 task 8: Multi- way classification of semantic relations between pairs of nominals. arXiv preprint arXiv:1911.10422
Hao Chen, Rui Xia, and Jianfei Yu. 2021. Reinforced counterfactual data augmentation for dual sentiment classification. In Proceedings of the 202/ Confer- ence on Empirical Methods in Natural Language Processing, pages 269-278.
Phillip Howard, Gadi Singer, Vasudev Lal, Yejin Choi, and Swabha Swayamdipla. 2022. Neuro- counterfactuals: Beyond minimal-edit counterfac- tuals for richer data augmentation. arXiv preprint Sti::2200.12365
Xiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng, Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022. Knowprompt: Knowledge- aware prompt-tuning with synergistic optimization

---

Wenlong Huang, Picter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Language models as zero-shot planners: Extracting actionable knowledge for em- bodied agents. In International Conference on Ma- chine Learning, pages 9118-9147. PMLR.
Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian- Saan-Sheng Hua, and Ji- Ji-Rong Wen. 2021. Counter- factual vqa: A cause-effect look at language bias. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 12700-12710.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. Tinybert: Distilling bert for natural language under- standing, arXiv preprint arXiv:/909, 10351.
Bhargavi Paranjape, Mandar Joshi, John Thickstun, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. An information bottleneck approach for controlling conciseness in rationale extraction. arXiv preprint 2tl::000500055...
Nitish Joshi and He He. 2021. An investigation of the (in) effectiveness of counterfactually augmented data. arXiv preprint urXiv:2107.00753.
Judea Pearl. 2009, Causal inference in statistics: An overview.
Divyansh Kaushik, Eduard Hovy, and Zachary C Cipton. 2019. Learning the difference that makes a differ- ence with counterfactually augggteeettlll-eeeeeeeeeeeeeeedd data. arXiv prepriat arXiv:1909.1 19K.:11999.1233 12434.
Judea Pearl and Dana Mackenzie. 2018. The book of why: the new science of cause and effect. Basic books.
Luoqiu Li. Xiang Chen, Hongbin Ye, Zhen Bi. Shumin Deng, Ningyu Zhang, and Huajun Chen. 2021. On robustness and bias analysis of bert- relation ex- traction. In China Conference on Knowledge Graph and Semantic Computing, pages 43-59. Springer.
Yujia Qin. Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang, Maosong Sun, and Jie Zhou. 2021. ERICA: Improving entity and rela- tion understanding for pre-trate pre-trained language models via contrastive learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Confer- ence on Natural Language Processing ( (Volume 1: Long Papers), pages 3350-3363, Online. Association for Computational Linguistics.
Wanli Li, Tieyun Qian, Ming Zhong, and Xu Chen. 2022a. Interactive lexical and semantic graphs for semisupervised relation extraction. IEEE Transac- tions on Neural Networks and Learning Systems.
Ziyan Li, Kan Ni, Haofen Wang, and Wenqiang Zhang. 2022b. Relation extraction text matching: A scheme for multi-hop knowledge base question an- swering. In CCKS E22--Eval Track: 7th China Conference Knowledge Graph and Seman- tic Computing Evaluations, CCKS 2022, Qinhuang- dao, China, August 24-27, 2022, Revised Selected Papers, pages 191-201. 2.rine0..Srrnner.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9.
Marcel Robeer, Floris Bex, and Ad Feelders. 2021. Gen- erating realistic natural language counterfactuals. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 3611-3625.
Bill Yuchen Lin, Wenyang Gao, Jun Yan, Ryan Moreno, and Xiang Ren. 2021. Rockner: A simple method to create adversarial examples for evaluating the ro- bustness of named entity recognition models. arXiv preprint 210.0000..0..0.
Alexis Ross, Ana Marasović. and Matthew E Peters. 2021. Explaining nip models via minimal con- trastive editing (mice). In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3840-3852.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907. 1692.
Indira Sen, Mattia Samory, Fabian Flück, Claudia Wagner, and Isabelle Augenstein. 2021. How does counterfactually augmented data impact mod- els for social computing constructs? arXiv preprint 200.02210..07022..
Angrosh Mandya, Danushka Bollegala, and Frans Co- enen. 2020. Graph convolution over multiple depen- dency sub-graphs for relation extraction. In Proceed- Com-
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Improving neural machine translation models with monolingual data. arXiv preprint 11X1v:1511.06709.
ings of the 28th International Conference putational Linguistics. pages 6424-6435, Barcelona, Spain (Online). International Committee on Compu- tational Linguistics.
Marcos Treviso. Alexis Ross, Nuno M Guerreiro, and André FT Martins. 2023. Crest: A joint framework for rationalization and counterfactual text generation. arXiv preprint 2.05.
George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39-41
Guoshun Nan, Jiaqi Zeng, Rui Qiao, Zhijiang Guo, and Wei Lu. 2021. Uncovering main causalities for long-tailed information extraction. arXiv preprint arXiv:2109.05213.
Zhao Wang and Aron Culotta. 2021. Robustness to spurious correlations in text classification via auto- matically generated counterfactuals In Proceedings

---

of the AAAI Conference on Artificial Intelligence. volume 35, pages 14031.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character- convolutional networks for text classi- fication. Advances in neural information processing systems, 28.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. 2022.
Chain thought prompting elicits reasoning in large language models. arXiv preprint arXi::2201.1199) 1903.
Yuhao Zhang. Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 35-45, Copenhagen, Denmark. Association for Com- putational Linguistics.
Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang. Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, et al. 2023. Zero- shot information extraction via chatting with chatgpt. arXiv preprint arXiv:2302.1020:
Zexuan Zhong and Danqi Chen. 2021. A frustratingly easy approach for entity and relation extraction. In North American Association for Computational Lin- guistics (NAACL).
Jiaxin Wen, Yeshuang Zhu, Jinchao Zhang. Jie Zhou, and Minlie Huang. 2022. Autocad: Automatically generating counterfactuals for mitigating shortcut learning. Xixiv preprint arXiv:2211.1663
Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li. Hongwei Hao, and Bo Xu. 2016. Attention-basee bidirectional long short-term memory networks for relation classification. Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 207- 212, Berlin, Germany. Association for Computational Linguistics.
Haoran Wu, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao, and Michael Lyu. 2023. Chatgpt or grammarly? evaluating chatgpt grammatical error correction benchmark. arXiv preprint 2rXiv:2333.1360
Proceedings of the 54th
Shanchan Wu and Yifan Hc. 2019. Enriching pre- trained language model with entity information for relation classification. In Proceedings of the 28th ACM international conference on information and knowledge management, pages 2361-2364.
Tongshuang Wu, Marco Tulio Ribeino, Jeffrey Heer, and Daniel Weld. 2021. Polyjuice: Generating coun- terfactuals for explaining, evaluating, and improving models. arXiv preprint arXiv: 200. 200.00088...
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: Deep contextualized entity representations with entity- aware self-attention. Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 6442 6454, On- line. Association for Computational Linguistics.
Kexin Yang, Dayiheng Liu, Wenqiang Lei, Baosong Yang, Mingfeng Xue, Boxing Chen, and Jun Xie. 2022. Tailor: prompt-based approach to attribute- based controlled text generation. arXiv preprint arXiv:2204.1323 133622.
Linyi Yang, Jiazheng Li, Pádraig Cunningham, Yue Zhang, Barry Smyth, and Ruihai Dong. 2021. Ex- ploring the efficacy of automatically generated coun- terfactuals for sentiment analysis. arXiv preprint 10xiv:2106.152..
Mo Yu, Yang Zhang, Shiyu Chang, and Tommi Joak 2021. Understanding interlocking dynamics of coop- erative rationalization. Advances in Neural Informa- tion Processing Systems, 34:12821 12835.
Mi Zhang, Tieyun Qian, Ting Zhang, and Xin Miao. 2023. Towards model rohustness: Generating contex- tual counterfactuals for entities in relation extraction. Proceedings of ACM Web Conference 2023, pages 1832-1842.

---

A Experimental Details
Setting
Dataset
Split
Sentences
Types
Direction
Train
7200
A.1 Datasets
Low
SemEval
Dev
800
Bi direectiion
Test
2715
Low-resource Settings Following previous re- lated works (Li et al., 2022a; Chen et al., 2022), evaluate models' performance under two types of low-resource scenarios. Firstly, Li al. (2022a) proposed a proportionally divided scenario, includ- ing 5% and 10% settings. Namely, randomly sam- pling 5% or 10% of data from a training set. In addi- tion, we supplement 1% and 3% settings. Secondly, Chen et al. (2022) proposed a scenario divided by a fixed number of instances per relation, including 8-shot, 16-shot, and 32-shot settings. For example, for 8-shot, we randomly extract 8 instances from the training set for each relation. Furthermore, we include 2-shot and 4-shot settings. We introduce these more challenging settings for both scenarios in order to thoroughly assess models' effectiveness.
Train (WL)
576
Dev (WL)
192
Out-of-domain
ACE 2005
Test (BC)
1607
Uni-direction
(BN)
Test (NW)
2680
Train
7200
SemEval
Adversarial attack
directioon
RB-Amack
Test
2391
Table 7: Statistics of experimental datasets. Be aware that the training and development set of RE-Attack remain consistent with those of SemEval.
given instance, we first substitute its entities with different entities from instances with the same label. After obtaining various candidates, to pick up high- quality sentences, we also apply the GPT-2-based strategy for automatic filtering. Finally, we man- ually choose at most one semantically fluent and relation-complian sentence for each instance from the filtered candidates and apply it to RE-Attack
Out-of-domain Settings ACE 2005 multilingual training corpus (Grishman et al., 2005) contains the complete set of English, Arabic, and Chinese train- ing data for the 2005 Automatic Content Extraction (ACE) technology evaluation. line with previ- ous work (Zhang et al., 2023), we only conduct experiments on the English data. Due to copyright issues, please download the dataset from the pro- vided link Refer to the processing approach of previous work (Zhong and Chen, 2021), utilize the preprocessing code from DyGIE repository".
A.2 Baselines
Conventional Methods Following previous work (Wen et al., 2022), the word-replacement ra- tio of Synonym Replacement (Zhang et al., 2015) and BERT-MLM (Jiao et al., 2019) is set to 30%. Specifically, we randomly select 30% of words in sentence for replacement. For Synonym Replace- ment, the synonyms come from WordNet (Miller, 1995). For BERT-MLM, we employ BERT-base as the MLM for word substitution. We implement Back Translation (Sennrich et al., 2015) by using an online translation APT". Specifically, we first translate English sentences into Chinese text and then translate them back. To be consistent with previous work (Wen et al., 2022), all the methods mentioned above augment each instance once.
Adversarial-attack Settings To balance the tradeoff' between time consumption and data qual- ity, this adversarial-attack dataset is built in semi- automatic way upon the test set of SemEval (Hen- drickx al., 2019). Specifically, we intervene each instance through label-flipped and label-invariant approaches. In the label-flipped case, for given instance, we first replace its causal term with the different ones from other relations and flip its la- bel. This allows us to obtain a large number of candidates, but only small portion of them are readable. Then, we employ GPT-2 (Radford et al.. 2019) calculate their perplexity and select the top 10 sentences with the lowest perplexity for man- ual verification. Finally, we manually pick at most one sentence that meets the three aforementioned requirements in $3.1 for each instance and apply to RE-Attck. In the label-invariant situation, for a
Counterfactual Methods The current counter- factual methods lack specialized design to con- form with the commonsense constraint, including MICE (Ross et al.. 2021), AutoCAD (Wen et al., 2022), and CoCo (Zhang et al., 2023). Therefore, during the implementation, these methods assume that all other relations be potential ones. The ultimate data augmentation in quality control relies entirely on consistency filtering. Simultancously. the quantity and ratio of data augmentation also de- pend on the filtering model. The detailed statistics of data augmentation are displayed in Table
"nttps://catalog..dd. upc.u wwwpen.edd/LLC2ee0000 LDCC2000666666 /https://iithhb. wttgg/thtb..mmmag///./ wwwp//aaw.a/cc///a////.aoaaaaaat /tyg//layayyyyy////aaaa preprocessing
hhttps://huggingface ww/berttbase-uncased "https://api fanyi. baidu. com/

---

Low-resource 1
Low-resource
RE-Attack
ACE 2005
Method
Num.
Ratio
Num.
Ratio
Num.
Ratio
Num.
Ratio
MICE
111%
282
67%
12894
179%
AutoCAD
712
141%
368
87%
16839
234%
3
1%
CoCo
40
8%
17
4%
677
9%
22
4%
CCG
266
51%
54%
4619
47%
Table 8: The statistics of the augmentation quantity and ratio for counterfactual methods. The num. indicates augmentation quantity and the ratio the proportion of augmentation quantity to the original data volume. Note that the values are averages under two types of low-resource settings, CCG does not require consistency filtering.
ChatGPT In order to thoroughly unleash the counterfactual reasoning capability of ChatGPT, we design clearly described and example-guides prompt, described in Table 9. In this prompt, we incorporate the Chain-of-thouglh (COT) for step-wise decomposition, which has been proven effective in reasoning and commonsense to solve tasks (Huang et al., 2022; Wei et al., 2022). We di- vide the counterfactual generation of RE into three steps and provide specific descriptions for each one. Furthermore, we provide example to help ChatGPT understand the assignment and standard format. In the implementation, we make use of the official AP provided by OpenAI to perform all of our experiments. Specifically, taking into account both performance and resource consumption, the specific version we have chosen is gpt-3.5-turbo Align with the conventional methods, ChatGPT also augments each instance once.
tual instances generated by each method for rating, respectively. Specifically, we assess an instance whether aligns with commonsense based on the relation role of entities. For instance, entity "buck- ets" can take on the role of "destination", but not as "effect", as shown in Figure 1. Note that all annotations were completed by three individuals, who possess relevant research experiences, and the final decisions were determined through voting.
A.3 Human Study
To intuitively analyze different strategies, we con- duct a small-scale human study. Firstly, to evaluate the performance of various causal term identifica- tion strategies, we randomly select 100 instances from SemEval (Hendrickx et al., 2019) for man- ual annotation, where we manually identify the words that determine the relations between enti- ties. Since we need adhere to the principle of minimal edits, i.e. identifying causal words (re- call) while avoiding affecting non-causal words (precision), we naturally utilize the Fl-score for evaluation. It is worth noting that each instance is the smallest unit of data augmentation, hence each instance should be assigned the same weight. Treating each instance as an evaluation category, we employ the Macro-Fl score specific eval- uation metric. Secondly, directly evaluate the commonsense degree generated counterfactuals, from SemEval, we randomly select 100 counterfac-
"https: httttts://platfomm. wttp://atooroooppaiiiiomo openai..oom

---

Task Definition: changing the relation between entity pair with minimal contextual edits.
Instruction Description: the process can be divided into the following three steps. (1) causal term identification, finding as few contextual words (except for entity pair) as possible that can change the relation when they are re- placed; (2) potential relation discovery, picking another commonsense relation from candidate relations; (3) caus-
al term replacement, replacing the causal term with appropriate words to change the original relation to the potential one.
Candidate Relations: message-topiics topic-messse entity-destinationn destination entity, content-container, co- ntainer-content, effect-cause, cause-effect, chole-componen, cemponent-whole collection-member member-co- llection, ageney-instrumenn, instrument-agency, producer-ppoouucl, product- entily-origin, origin-entily.
Example:
Input: the key is moved into chest.
Entity Pair: key-chest
Relation: entity- destyddestinntion
Causal Term Identification: relation "entity-destination depends contextual words "moved into". Potential Relation Discovery: "key" can be "entity", "chest" can be "origin", thus another commonsense relation from candidate relations is "entity-origin..
Causal Term Replacement: to change original relation to the potential one "cntity-origin" contextual words "m- oved into" are replaced with "from".
Output: the key is from chest.
Relation: entity-origin
Inference (completing the remaining content and maintaining consistency with the format of example): Input: eggs removed into buckets.
Entity Pair: eggs-buckets
Relation: -nstiyddet
Table 9: The prompt designed for directing ChaiGPT to generate RE counterfactuals.


Reviews:
[{"review_id": "WpnUtLPHhV", "reviewer": "Reviewer_fvyK", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your valuable suggestions. We will take them into consideration and make the necessary modifications to the paper.\n\n***Suggestion:** sort citations by date i.e. (Wang and Culotta, 2021; Garg and Ramakrishnan, 2020; Kaushik et al., 2019) => Kaushik et al., 2019 Garg and Ramakrishnan, 2020, Wang and Culotta, 2021)*  \n**Response:** So many thanks for your intensive reading! We will carefully edit the paper and sort all citations by date per your advice.\n\n***Suggestion:** Conclusions seem a bit misaligned with abstract. Authors note in the abstract that ”existing methods [of counterfactual augmented data] encounter [..] problems when dealing with the fine-grained relation extraction tasks”. In the conclusions authors claim that they “introduce the problem of commonsense counterfactual generation into the relation extraction field”, which is not the same. Also, “commonsense counterfactual generation” is more a technique rather than an problem. The problem, as stated in the abstract is the weak quality of that generation due to: “struggle to accurately identify causal term under the invariant entity constraint” and “ignore the commonsense constraint”.*  \n**Response:** Thank you for pointing this out! We will revise the conclusion part as follows: “… solve the problems in existing methods of counterfactual generation, i.e., struggling to accurately identify causal term under the invariant entity constraint and ignoring the commonsense constraint.”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Approach uses a known method (counterfactual augmented data) on an important NLP task (relation extraction).\n\nProposal seems fairly grounded by previous work (authors did a good job in exploring previous work, locating this work in the research space and providing clear motivations).\n\nExperimentation was carried out on reasonable datasets. ACE2005 is standard in relation extraction domain.\n\nExperiments show an improvement in performance compared to previous work.\n", "reasons_to_reject": "Solid work… no clear reasons to reject \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work proposes to use counterfactual augmented data to improve relation extraction methods. Concretely authors aim to use counterfactuals to avoid relying on spurious correlations. \nThe method relies on three aspects. To generate counterfactual data only affects the context, to keep the entity pair while changing the relation entity. The method identifies the causal term, and removes the minimal phrase of it. Identifies hypernyms of the participating entities and their relation to do relation expansion.\nThe model is evaluated in three different settings: low resource setting, out of domain and adversarial setting. For the last one, a new dataset “REAttack” was introduced.\nModel is compared with reasonable baselines such as synonym replacement, back translation BERT-MLM, MICE (Ross et al., 2021) AutoCAD (Wen et al., 2022), CoCo (Zhang et al., 2023) and ChatGPT.\nNumbers show that the proposed model Commonsense Counterfactual Generation (CCG) presents better behavior than its competitors. \nA human evaluation is also performed in order to evaluate the quality of the counterfactuals of CCG compared to CoCo and AutoCAD, showing a clear differentiation in terms of F1 for causal term validity and score for commonsense rating\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "In general: sort citations by date i.e.  (Wang and Culotta, 2021; Garg and Ramakrishnan, 2020; Kaushik et al., 2019) => Kaushik et al., 2019 Garg and Ramakrishnan, 2020, Wang and Culotta, 2021)\n\nConclusions seem a bit misaligned with abstract. Authors note in the abstract that ”existing methods [of counterfactual augmented data] encounter [..] problems when dealing with the fine-grained relation extraction tasks”. In the conclusions authors claim that they “introduce the problem of commonsense counterfactual generation into the relation extraction field”, which is not the same.\nAlso, “commonsense counterfactual generation” is more a technique rather than an problem. The problem, as stated in the abstract is the weak quality of that generation due to: “struggle to accurately identify causal term under the invariant entity constraint” and “ignore the commonsense constraint”\n"}}, {"review_id": "ito3LXHn0F", "reviewer": "Reviewer_Uqod", "summary": "", "questions": "The ChatGPT prompt divides the task into three steps. I wonder in what steps ChatGPT does not perform well. A defect shown in the case study is that ChatGPT may generate illusory relation in potential relation discovery. If this is the main problem, can it be easily solved by filtering?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable suggestions. Based on your advice and questions, we have made the following clarifications in an effort to address your concerns.\n\n***Suggestion:** CCG performs well when reading the numbers in tables, but it doesn't appear to be that good when seeing the generated outputs in case study. In both randomly selected cases, it replaces verb phrase with a single preposition, which undermines the integrity of the sentence, while other methods generate grammatically correct sentences. A human evaluation on the grammatical correctness and semantical readability is possibly needed to address the problem.*  \n**Response:** Regarding the grammatical correctness and semantical readability problem, the first case by our CCG is correct since we replace the past participle with a preposition, and the readability of the second case by CCG, while satisfying the commonsense constraint, is not as good as those by CoCo and ChatGPT.  \n&emsp; Please note that other methods also generate grammatically incorrect and semantical unreadable sentences, as shown in two cases. To thoroughly evaluate the quality of generated counterfactuals, we conduct evaluation experiments to verify grammatical correctness and semantic readability based on Grammarly [1]. Grammarly is a prevalent English typing assistant that reviews spelling, grammar, punctuation, clarity, engagement, and delivery mistakes in English texts, detects plagiarism and suggests replacements for the identified errors [2]. Specifically, we randomly select 100 generated examples from each counterfactual-based methods. We then treat these examples as a complete document and let the Grammarly tool check it. The results are shown in Table 1.\n\n| Method | Suggestion (↓) | Score (↑) |\n| --- | --- | --- |\n| MICE | 114 | 35 |\n| AutoCAD | 113 | 35 |\n| CoCo | 95 | 45 |\n| ChatGPT | $\\textbf{41}$ | $\\textbf{76}$ |\n| CCG | $\\underline{90}$ | $\\underline{47}$ |\n|  |  |  |\n\nTable 1: Results for grammatical correctness and semantic readability evaluation on SemEval, where ‘*suggestion*’ denotes the number of grammatical suggestions by Grammarly, and ‘*score*’ denotes the overall quality of writing in this document including readability. ↓ indicates that a lower value is better, and ↑ indicates that a higher value is better. Numbers in **bold** indicate the best results, and the $\\underline{underlined}$ ones are the second best. For example, ChatGPT achieves a score of 76, indicating that the text by ChatGPT is better than 76% of all texts checked by Grammarly including those by humans.\n\nThe results demonstrate that the grammatical correctness and semantic readability of CCG is only inferior to ChatGPT, but better than all other methods. We will further conduct a human evaluation and will report the results on randomly selected 10 instances (due to the space limit) from the above set in the appendix. \n\n***Suggestion:** The experiments focus on a small amount of training data, and as the experiments on SemEval show, the performance gain of CCG diminishes when the training data increases. When the training data comes to 10% and 32-shot, the difference between CCG and CoCo is smaller than the standard deviation. I find that CoCo, the only existing research on this task, experiments on some large-scale settings, like the whole SemEval training set and all domains except one in the out-of-domain setting. So adding experiments following the setting of CoCo may help to compare the two methods comprehensively.*  \n**Response:** According to our experience and the results of previous work [3-6], we believe that the large-scale setting under the IID (Independent Identically Distribution) scenario cannot effectively validate the effects of counterfactuals. Usually, the ratio of a training set to a test set is significantly larger than 1 (e.g. 2.65 in SemEval), and if under an IID scenario, the spurious correlations present in the test set are also contained within the training set. In this situation, the spurious correlations can assist the model in finding shortcuts and improving accuracy [4]. Therefore, when counterfactuals block spurious correlations, they may not help the model in terms of accuracy and could even have a counterproductive effect [3-6].  \n&emsp; To eliminate such “benefit” of spurious correlations and accurately validate the effects of counterfactuals, we need to disrupt this contained relationship of spurious correlations between training and test set. For the training set, we reduce the data size to reduce its overlap with the spurious associations present in the test set. This corresponds to the low-resource setting that we introduced (note that the test set remains unchanged in this case). For the test set, we can introduce out-of-domain and out-of-distribution instances to make its distribution different from the training set. These correspond to the our-of-domain and adversarial-attack settings that we introduced. Please note that the training set remains unchanged in this case, e.g., we use the whole SemEval training set in the adversarial-attack setting. As a result, both low-resource and out-of-domain settings have been considered as important ways to validate counterfactuals [7].  \n&emsp; Nonetheless, we have reported the F1-socre compared with other counterfactual-based methods on the whole SemEval training set. The results are presented in Table 2. Due to the reasons mentioned above, the benefits of counterfactuals, from either CoCo or CCG, are quite small under this setting. Other methods even have the opposite effect. This proves that such a setting is impropriate to truly showcase the quality differences of counterfactuals.  \n\n| Method | R-BERT | R-RoBERTa |\n| --- | --- | --- |\n| Original | 88.07 (± 0.47) | 88.22 (±0.41) |\n| MICE | 88.16 (±0.25) | 87.85 (±0.49) |\n| AutoCAD | 88.18 (±0.38) | 87.96 (±0.52) |\n| CoCo | $\\underline{88.22}$ (±0.20) | $\\underline{88.32}$ (±0.31) |\n| ChatGPT | 87.52 (±0.29) | 87.76 (±0.26) |\n| CCG | $\\textbf{88.31}$ (±0.16) | $\\textbf{88.45}$ (±0.38) |\n|  |  |  |\n\nTable 2: The comparison results on the whole SemEval training set. Numbers in **bold** indicate the best result, and the $\\underline{underlined}$ ones are the second best. The numbers within parentheses indicate the standard deviation.\n\nIn addition, there is one more point we need to clarify. In the out-of-domain setting, our experimental setup is consistent with CoCo, that is, one domain serves as the training set, while the remaining domains are used as separate test sets. The results have been presented in Table 2 of our submitted paper.\n\n***Suggestion:** The ChatGPT prompt divides the task into three steps. I wonder in what steps ChatGPT does not perform well. A defect shown in the case study is that ChatGPT may generate illusory relations in potential relation discovery. If this is the main problem, can it be easily solved by filtering?*  \n**Response:** To analyze which step ChatGPT does not perform well, we conduct a human study. Specifically, we randomly select 100 examples generated by ChatGPT. And then, we count the number of errors for each step and calculate the proportions. The errors in the first, second, and third steps account for 10%, 48%, and 42%, respectively. We can observe that the second step, where potential relations identification, performs the worst, and the third step is also affected. Therefore, generating illusory relations is the main problem.  \n&emsp; Afterward, we attempt to apply a filtering mechanism to ChatGPT. We report F1-socre in Table 3 and Table 4. Table 3 includes low-resource and adversarial-attack settings, and Table 4 corresponds to out-of-domain setting.\n\n| Method | | | R-BERT | | | | | R-RoBERTa | | |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| | 1% | 3% | 5% | 10% | Adv. | 1% | 3% | 5% | 10% | Adv. | 1% | 3% | 5% | 10% | Adv.\n| Original | 33.26 (±1.43) | 59.31 (±1.46) | 68.66 (±1.77) | 76.47 (±1.14) | 53.34 (±1.78) | 35.77 (±2.41) | 64.27 (±3.20) | 69.99 (±1.84) | 78.27 (±1.07) | 64.16 (±1.19) |\n| ChatGPT | 38.78 (±2.71) | 61.84 (±1.23) | 67.90 (±2.14) | 75.15 (±1.10) | 56.15 (±1.18) | 38.71 (±2.11) | 64.44 (±1.34) | 70.14 (±2.11) | 76.25 (±0.52) | 65.78 (±1.31) |\n| w/ Filtering | 32.47 (±2.87) ↓ | 59.85 (±2.06) ↓ | 68.39 (±2.52) ↑ | 78.64 (±0.37) ↑ | 55.70 (±2.30) ↓ | 35.46 (±2.54) ↓ | 65.37 (±1.68) ↑ | 72.41 (±2.78) ↑ | 79.33 (±1.27) ↑ | 64.67 (±1.37) ↓ |\n|  |  |  |  |  |  |  |  |  |  |  |\n\nTable 3: The results of low-resource and adversarial-attack settings. 1%-10% denotes the low-resource setting, and Adv. denotes the adversarial-attack setting. w/ Filtering represents ChatGPT combined with the filtering mechanism. ↓ indicates a decrease compared to the original result, while ↑ indicates an increase.\n\n| Method | | R-BERT | | | R-RoBERTa | |\n| --- | --- | --- | --- | --- | --- | --- |\n| | WL→BC | WL→BN | WL→NW | WL→BC | WL→BN | WL→NW |\n| Original |   70.43 (±2.45) | 70.55 (±2.51) | 69.42 (±1.41) | 74.17 (±0.70) | 70.54 (±0.87) | 74.93 (±0.74) |\n| ChatGPT | 52.70 (±0.99) | 55.94 (±1.21) | 54.51 (±0.63) | 59.55 (±0.50) | 60.11 (±0.89) | 61.74 (±1.13) |\n| w/ Filtering | 69.27 (±2.37) ↑ | 70.46 (±1.93) ↑ | 69.55 (±1.63) ↑ | 74.58 (±1.37) ↑ | 70.33 (±1.36) ↑ | 74.85 (±0.74) ↑ | \n|  |  |  |  |  |  |  |\n\nTable 4: The results of out-of-domain setting. WL→BC denotes that the training set is in the WL domain and the test set is in the BC domain, the same for others. w/ Filtering represents ChatGPT combined with the filtering mechanism. ↓ indicates a decrease compared to the original result, while ↑ indicates an increase.\n\nBased on the experimental results, we can conclude that the problem of illusory relation can be partially alleviated through filtering in some cases, but it cannot be easily solved.  \n&emsp; Firstly, although the filtering mechanism can remove noise data, it heavily relies on the performance of the filter, i.e., the base model. In settings with relatively abundant training resources, the filter is adequately trained and the filtering mechanism might be effective, such as the cases in the 5%, 10%, and the out-of-domain settings. However, in more extreme scenarios such as very low-resource or adversarial-attack settings, the filter itself may become ineffective.  \n&emsp; Secondly, the filtering mechanism, which optimizes data solely through subtraction, always has its limitations. Filtering only mitigates the negative impact of low-quality data without generating higher-quality data. Therefore, even with the inclusion of the filtering mechanism, the vast majority of results still exhibit significant differences from CCG.\n\n***Question:** Line 97: what does “they” refer to?  \nLine 230: why determining the change of prediction is much simpler than predicting the actual outcomes?  \nAlgorithm 1: the hyperparameter K is missing.*  \n**Response:** In Line 97, “They” refers to the current counterfactual generation methods in the natural language processing community, including MICE, AutoCAD, and CoCo.  \nIn Line 230, the decision space for “determining the change of prediction” (binary classification) is much smaller than the decision space for“predicting the actual outcomes” (multi-class classification, directly proportional to the number of relations), which greatly reduces the complexity of the problem.  \nThank you for your reminder. We will incorporate the hyperparameter K into the Algorithm 1.\n\n***Suggestion:** Line 214: an intervention-based strategy  \nLine 369: The constituency parser is from CoreNLP.  \nCaption of Table 7: The instances are from SemEval dataset.*   \n**Response:** So many thanks for your intensive reading! We will carefully edit the paper and make revisions per your suggestions!\n\n**Rerefrence**  \n[1] Grammarly. 2023.https://www.grammarly.com/about.  \n[2] Wu H, Wang W, Wan Y, et al. Chatgpt or grammarly? evaluating chatgpt on grammatical error correction benchmark[J]. arXiv preprint arXiv:2303.13648, 2023.  \n[3] Kaushik D, Hovy E, Lipton Z C. Learning the difference that makes a difference with counterfactually-augmented data[J]. arXiv preprint arXiv:1909.12434, 2019.  \n[4] Sen I, Samory M, Flöck F, et al. How Does Counterfactually Augmented Data Impact Models for Social Computing Constructs?[J]. arXiv preprint arXiv:2109.07022, 2021.  \n[5] Wang Z, Culotta A. Robustness to spurious correlations in text classification via automatically generated counterfactuals[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2021, 35(16): 14024-14031.  \n[6] Geva M, Wolfson T, Berant J. Break, perturb, build: Automatic perturbation of reasoning paths through question decomposition[J]. Transactions of the Association for Computational Linguistics, 2022, 10: 111-126.  \n[7] Calderon N, Ben-David E, Feder A, et al. Docogen: Domain counterfactual generation for low resource domain adaptation[J]. arXiv preprint arXiv:2202.12350, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe paper is well organized. It clearly defines the requirements of counterfactual data augmentation for relation extraction, and summarizes two reasonable challenges for this task. The proposed method fits the motivations nicely.\n2.\tThe authors conduct multidimensional experiments and analysis. They explore three different scenarios: low-resource, out-of-domain, and adversarial-attack, and CCG exhibits its advantage in all the scenarios.", "reasons_to_reject": "1.\tCCG performs well when reading the numbers in tables, but it doesn't appear to be that good when seeing the generated outputs in case study. In both randomly selected cases, it replaces verb phrase with a single preposition, which undermines the integrity of the sentence, while other methods generate grammatically correct sentences. A human evaluation on the grammatical correctness and semantical readability is possibly needed to address the problem.\n2.\tThe experiments focus on a small amount of training data, and as the experiments on SemEval show, the performance gain of CCG diminishes when the training data increases. When the training data comes to 10% and 32-shot, the difference between CCG and CoCo \tis smaller than the standard deviation. I find that CoCo, the only existing research towards this task, experiments on some large-scale settings, like the whole SemEval training set and all domains except one in the out-of-domain setting. So adding experiments following the setting of CoCo may help to compare the two methods comprehensively.", "questions_to_authors": "The ChatGPT prompt divides the task into three steps. I wonder in what steps ChatGPT does not perform well. A defect shown in the case study is that ChatGPT may generate illusory relation in potential relation discovery. If this is the main problem, can it be easily solved by filtering?", "paper_topic_and_main_contributions": "This paper introduces Commonsense Counterfactual Generation (CCG), a new counterfactual data augmentation method for relation extraction. It tries to resolve two challenges: the first is to accurately identify causal terms, and the second is to be consistent with commonsense. The method is divided into three steps, causal terms identification, relation expansion, and controlled editing. In causal terms identification, the authors design an intervention-based strategy to identify editable words and use a constituency parser for correction. In relation expansion, they uncover possible relations that meet commonsense with the help of WordNet. In controlled editing, a content generation model is trained to generate the masked content given a relation. Experiments in various settings demonstrate the effectiveness and robustness of CCG.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1.\tMissing details: \n\na)\tLine 97: what does “they” refer to?\n\nb)\tLine 230: why determining the change of prediction is much simpler than predicting the actual outcomes?\n\nc)\tAlgorithm 1: the hyperparameter $\\mathcal{K}$ is missing\n\n2.\tTypos:\n\na)\tLine 214: an intervention-based strategy\n\nb)\tLine 369: The constituency parser is from CoreNLP.\n\nc)\tCaption of Table 7: The instances are from SemEval dataset."}}, {"review_id": "VBhxvoWKA0", "reviewer": "Reviewer_Ck7v", "summary": "", "questions": "Line 141: What does “its performance is entangled with the base model” mean? \n\nLine 290: Why is it obvious? Perhaps it could be useful to briefly state it.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments. We hope the following clarifications can address the reviewer’s concerns.\n\n***Question:** Line 141: What does “its performance is entangled with the base model” mean?*  \n**Response:** As mentioned in Line 060, current counterfactual augmentation methods rely on filtering strategies to ensure the quality of the generated data. In this process, the filter is the relation extraction model (the base model) trained on existing data, and the data the base model filters is used to improve itself. We define this contradictory phenomenon as the entangled problem. Note that our method identifies explicit targets through potential relation identification, which enables the direct generation of high-quality counterfactuals. As a result, this process can be bypassed, i.e., we effectively address this issue at its core.\n\n***Question:** Line 290: Why is it obvious? Perhaps it could be useful to briefly state it.*  \n**Response:** As introduced in Line 260, hypernymy (super-name) are transitive relations between concepts. Moreover, concepts gradually become broader in the hierarchy of hypernyms, as illustrated in Figure 2 in our submission (bin->container->artifact, the container is hypernymy of the bin, the artifact is hypernymy of the container). Therefore, between any entity pairs, the lower common hypernym in the hierarchical structure, the closer their semantics. According to this nature, for a given entity pair, our bottom-up retrieval strategy can proactively identify semantically closer entity pairs with different relations. As a result, the relations discovered earlier come from the entity pairs with more similar semantics, hence it should be given a higher priority. We will add these statements to the paper per your suggestion.  \n\n***Comment:** \"Improving commonsense causal reasoning by adversarial training and data augmentation\" by I Staliunaite, PJ Gorinski, I Iacobacci (2021) have presented a very similar method of generating confounders as well as generating adversarial examples to improve commonsense causal reasoning models.*  \n**Response:** Regarding the reference you introduced, we respectfully disagree with your mention of \"a very similar method of generating confounders as well as generating adversarial examples\". Our reasons are as follows.  \n1) There is an essential distinction between adversarial examples and counterfactual samples. The methods in the reference utilize synonym substitution to generate adversarial samples, whereas our focus is on generating counterfactual samples. From a causal perspective, the synonym substitution approach belongs to the second level of the causal ladder, namely intervention. It does not require a causal discovery process and relies on the principle of semantic invariance, and implementing the intervention (with unchanged labels) through the simple synonym substitution. In simpler terms, it guides the model by saying \"you shouldn't be affected by irrelevant perturbations\". In contrast, counterfactuals belong to the third level of the causal ladder, where they require causal discovery methods to identify causal words and then intervene on them to emphasize decision boundaries (label flipping). In simpler terms, the counterfactual-based approach directly guides the model by saying \"you should make judgments based on causal-relevant information\". Therefore, our method does not generate confounders. Moreover, we have taken a step further by first addressing and validating the significance of commonsense in counterfactual generation, which is evidently different from the commonsense classification task that the reference has focused on. Additionally, the adversarial-attack setting we employed is just one way to validate the effectiveness of counterfactuals, rather than directly attacking the target model.  \n&nbsp;\n2) Due to the distinction of the problems, there naturally exist significant differences in the implementation of methods.  \na) Firstly, the reference method does not require a causal discovery process, as mentioned earlier.  \nb) Secondly, the reference method solely relies on the node information from WordNet while our method leverages both the node and edge information. WordNet is a knowledge graph based on human commonsense, consisting of the synsets (conceptual nodes) and their relations (edges like hypernyms or hyponyms). Our method not only utilizes the nodes for concept localization, but also leverages hypernym relations between concepts to generalize entities. This enables the propagation of relations, thereby uncovering potential relations aligned with commonsense, as detailed in Algorithm 1.  \nc) Thirdly, the data generation process of the reference involves only substitution and does not involve generative models. On the contrary, generating counterfactual examples involves controlled text generation, which is also a challenge. After obtaining potential relations, the generative model needs to produce causal words consistent with the new relations.  \nd) Finally, the counterfactual data we generate can be directly used for data augmentation to enhance model stability. This process eliminates the need for the attack procedure relied upon by the reference, resembling the entangled issue we mentioned earlier.  \n&nbsp;\n3) Accompanied by the challenge during the counterfactual generation, counterfactuals are more effective in eliminating spurious correlations. The reference method is similar to the synonym substitution method Synonym Rep. which we have used as a baseline. The comparative analysis of the results between Synonym Rep. and our CCG indicates that synonym substitution data eliminates spurious correlations through a process of exclusion, whereas counterfactuals clearly emphasize the decision boundary directly, which proves to be more efficient.  \n\n***Suggestion:** Line 001: The start of the abstract is very unclear, the task that the paper tackles should be introduced at the very start, before stating that there are problems in it that most NLP models don’t address.  \nLine 027: The writing could use with some restructuring, many sentences end with a subclause that makes a side note, which is confusing to read.  \nLine 070: Invariant entity constraint and commonsense constraint should be clearly defined.*   \n**Response:** So many thanks for your intensive suggestions, we will make adjustments to the abstract and the main text accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A good overview of related work and clearly delineated unsolved issues and concrete contributions of this paper. \n\nAn interesting application of counterfactual generation for relation extraction.", "reasons_to_reject": "The paper is rather difficult to follow as it lacks structure. Most importantly, terms should be more clearly defined before being used.", "questions_to_authors": "Line 141: What does “its performance is entangled with the base model” mean? \n\nLine 290: Why is it obvious? Perhaps it could be useful to briefly state it.", "paper_topic_and_main_contributions": "The paper presents a novel approach to relation extraction by leveraging commonsense counterfactuals. The authors generate counterfactuals with the view to uncovering relationships between entities. They conduct extensive evaluations and demonstrate that the methods improves the models’ robustness.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "\"Improving commonsense causal reasoning by adversarial training and data augmentation\" by I Staliunaite, PJ Gorinski, I Iacobacci (2021) have presented a very similar method of generating confounders as well as generating adversarial examples to improve commonsense causal reasoning models.", "Typos_Grammar_Style_and_Presentation_Improvements": "Line 001: The start of the abstract is very unclear, the task that the paper tackles should be introduced at the very start, before stating that there are problems in it that most NLP models don’t address. \n\nLine 027: The writing could use with some restructuring, many sentences end with a subclause that makes a side note, which is confusing to read. \n\nLine 070: Invariant entity constraint and commonsense constraint should be clearly defined."}}]

Author rebuttal:

Meta-review: This paper presents a novel approach to data augmentation for relation extraction, consisting of three key steps: causal term identification, relation expansion, and controlled editing. Generally speaking, this is a solid data augmentation work for relation extraction. While the paper employs terms such as "intervention," "counterfactuals," and "causal," it is my viewpoint that the method proposed in this paper has a relatively weak connection to causal theory and does not operate within a causal theory framework. Therefore, I recommend that the authors carefully reconsider this aspect, make necessary revisions to the paper, and provide a clearer elucidation of the relationship between their method and causal theory, or employ more appropriate descriptions.
Decision: Accept-Main