CoVariance-based Causal Debiasing for Entity and Relation Extraction
Authors: Lin Ren
Yongbin Liu
Yixin Cao
Chunping Ouyang
Venue: EMNLP 2023 Findings (2023)
Keywords: Causal Debiasing
Entity and Relation Extraction
Covariance Optimizing
Variance Optimizing

Abstract: Joint entity and relation extraction tasks aim to recognize named entities and extract relations simultaneously. Suffering from a variety of data biases, such as data selection bias, and distribution bias (out of distribution, long-tail distribution), serious concerns can be witnessed to threaten the model's transferability, robustness, and generalization. In this work, we address the above problems from a causality perspective. We propose a novel causal framework called c$\underline{\textbf{o}}$variance and $\underline{\textbf{v}}$ariance $\underline{\textbf{o}}$ptimization framework (OVO) to optimize feature representations and conduct general debiasing. In particular, the proposed $\underline{\textbf{c}}$ovariance $\underline{\textbf{op}}$timizing (COP) minimizes characterizing features' covariance for alleviating the selection and distribution bias and enhances feature representation in the feature space. Furthermore, based on the causal backdoor adjustment, we propose $\\underline{\textbf{v}}$ariance $\underline{\textbf{op}}$timizing (VOP) separates samples in terms of label information and minimizes the variance of each dimension in the feature vectors of the same class label for mitigating the distribution bias further. By applying it to three strong baselines in two widely used datasets, the results demonstrate the effectiveness and generalization of OVO for joint entity and relation extraction tasks. Furthermore, a fine-grained analysis reveals that OVO possesses the capability to mitigate the impact of long-tail distribution.

Covaiance-bbsed Causal Debiasing for Entity and Relation Extraction
Lin Ren', Yongbin Liu"; Yixin Cao², Chumping Ouyang 'University of South China, "Singapore Management University (homurat.rl, yongbinliu03, caoyixin2011 gmmil.com, ouyangep @ 126.com
Abstract
Furthrmonn we show lans the socarll doncippal officinn safeeindeor remeohiSSS procelure intermed sags
Inference
Joint entity and relation extraction tasks aim to recognize named entities and extract relations simultaneously. Suffering from variety of data biases, such as data selection bias, and dis- tribution bias (out of distribution, long-tail dis- tribution), serious concerss can be witnessed to threaten the model's transferability, robustness, and generalization. In this work, address the above problems from a causality perspec- tive. We propose novel causal framework called covariance and yariance optimization framework (OVO) to optimize feature repre- sentations and conduct general debiasing. In particular, the proposed covariance optimizing (COP) minimizes characterizing features' co- variance for alleviating the selection and distri- bution bias and enhances feature representation the feature space. Furthermore, based on the causal backdoor adjustment, propose yariance optimizing (VOP) separates samples terms of label information and minimizes the variance of each dimension in the feature vectors of the same class label for mitigating the distribution bias further. By applying to three strong baselines in two widely used datasets, the results demonstrate the effective- ness and generalization of OVO for joint tity and relation extraction tasks. Furthermore, a fine-grained analysis reveals that OVO pos- sesses the capability to mitigate the impact of long-tail distribution. The code is available at https://githh: thub.....tttt. com/HomuraT/Ovo.
MOSTLSTION
HOVO
27%
Blas
FARTOO
PARTOP
Figure An example of RE on SciERC (the label bias increases the risk of predicting some label possessing majority samples). The confidence at the upper right is predicted by PL-Marker (Ye et al., 2022). The ground- truth relation of the example is PART-OF, but due the bias in the train set, the NO-RELATION is inferred. After using our OVO, PART-OF is predicted correctly.
One is to build separate model (pipeline) to get the result in step (Shang et al., 2022), and the other is tities and relations in turn (Zhong and Chen, 2021; Ye et al., 2022).
multiple models (joint) to extract
Suffering from various biases in data, such as data selection bias, out-of-distributtin, and long- tail distribution (Lin et al., 2022; Wang et al., 2022), models may learn spurious correlations and make erroneous predictions. Although we would like to be able to obtain uniform (or unbiased) data set, such dataset almost nonexistent in real applications. Various biases are always mixed up in real scenarios, for example, selected data in a way that differs from the target population (data se- lection bias), training and testing data distribution shift (out-of-distributtio)) the labels with numer- ous samples may have more chances to inference (long-tail distribution lead to predicting majority bias), and some of the data biases are even un- known. Because statistical correlation is the basis of supervised learning algorithms, the learning al- gorithms may learn many spurious correlations due to these biases. Finally, this leads to a greatly re-
1 Introduction
Named Entity Recognition (NER, (Ratinov and Roth, 2009)) and Relation Extraction (RE, (Bunescu and Mooney, 2005)) are both the key fun- damental tasks of Information Extraction and both significant predecessor tasks for building knowl- edge graphs that are used widely in daily life (Chen al., 2021; Ji al., 2022; Du al., 2022a). There are two major methods to extract the final result,
"Corresponding Author.

---

duced predictive ability of the models. As shown in Fig. 1 (left), there are different degrees of the data bias of long-tail distribution in SciERC (Luan et al., 2018), which is common for other datasets as well, making models focus more on the labels with numerous samples and performing poorly on the others (fewer chances to the labels with few samples).
B
L: Real Label
F : Feature vector
Prediction
B: Bins
F
P
Figure The causal graph of the joint entity and rela- tion extraction task in the classification phase. Bias is something that we can not avoid, such as agnostic data selection bias, data labeling error, and unobservable fea- tures.
In recent years, debiasing has become a signifi- cant research focus. There are different ways to ad- dress the data bias by reweighting samples (Dixon et al., 2018; 8; Rayhan et al., 2017; Wang et al., 2022). These methods focus on the reweights in the sam- ple space and ignore spurious correlations due to data selection bias. Recently, stable learning (Shen al., 2018, 2020; Zhang al., 2021) proposed serial ways de-bias the spurious correlation to find the causal effect of each input feature on the basis of the correlation. Specifically, these meth- ods require extensive calculation resources to iden- tify causality and focus solely on causal effects between features. Additionally, they fail to con- sider the causal effects for each feature dimension within the same class label.
ture dimensions of the same label and the out- come targets and mitigate the long-tail dis- tribution bias further. We call this method variance optimizing (VOP).
We combine COP and VOP as OVO. To access the effectiveness of OVO, we conduct experi- ments on the joint entity and relation extrac- tion task. By applying OVO strong base- lines, we observe significant improvements in the performance of this task, which demon- strates the effectiveness and generalization of our proposed method. These results suggest that optimizing feature representation directly is actually an effective way to get improve- ment and reduce the bias from datasets with- out model redesigning, and OVO is feasible framework in this way.
In this work, we introduce a method called the covariance and variance optimization framework (OVO), to optimize the feature vectors from two aspects, decorrelating between features and each feature dimension of the same class label. Our contributions can be summarized as follows:
From decorrelating between features, inspired by stable learning, confounder degree of features and try mini- mize it in each batch of every epoch learn the causality between features and targets. We minimize the covariance to reduce the collinearity of each feature, get the causal effects, alleviate the data selection and dis- tribution biases, and enhance feature repre- sentation. We call this method covariance optimizing (COP).
covariance as the
Related Work
Stable Learning aims research how to reduce the agnostic data selection bias and model mis- specification problem and make the feature more causal by reweighting samples. CRLR (Shen et al.. 2018) proposes an equation to evaluate the degree of confounders called Causal Regularized Logis- tic Regression and gains the optimal weight by minimizing the confounder. SRDO (Shen et al., 2020) considers the smallest feature of the covari- ance matrix as the degree of confounder. StableNet (Zhang et al., 2021) adopts stable learning tech- nology into image classification tasks using deep learning models and the results show the strong domain adaptation of the proposed models.
From feature dimensions of the same class label, according to the variance of each di- mension, using the intervention of causal in- ference (Pearl et al., 2016), it eliminate the confounding bias from the backdoor path by blocking F - L P as the causal graph shown in Fig.2, so that model can learn the causality from the causal path F + P. We explore the causal correlation between the fea-
Span-level NER and RE Models are the ap- proaches using the boundaries of spans as a total feature to recognize the entities and the relation

---

Covariance Optinizing(COP)
Loss
Samplee Number
LossgoP
Feature Voctors
Covarance Marnx
Conariancy
Variance Optimizig
Backdoor Adjustment
Distance Vgrionce
Group
Intervention
Label
Intervention P()
Lasevop
Label ug..
3
Label
Figure 3: The overview OVO. the training phase of the model, we insert COP and VOP as extra optimization for feature vectors. COP, there an inner loop learn the optimal using gradient descent, and Lossw is the loss function of W calculated according to eq. 4. For VOP, Backdoor Adjustment is used to reduce the bias from datasets. Note that Lossw only computes the gradient of W, and only the final optimal weights W created in COP will be applied the model training process. The Lossyor will be used in model training directly. Los8orig means the loss employed on the original method: O means zero matrix.
between two entities. DyGIE++ (Wadden al., 2019) and SpERT.PL (Sai al., 2021) extract features spans and share them between NER and RE tasks. PURE (Zhong and Chen, 2021) uses two separate span models to recognize en- tities and relations without interaction with each other and proposed levitated markers to reinforce the boundaries information of spans. PL-Marker (Ye et al., 2022) proposes neighborhood-oriente packing and subject-oriented packing methods to leverage the levitated markers.
extraction methods. Then, will describe covari- ance optimizing (COP) and variance optimizing (VOP).
3.1 Background: Inference Process of Joint Entity and Relation Extraction Methods In the joint entity and relation extraction task, the goal is to extract both entities and the relations between them from given text. The general pro- cess of this task involves several steps. Firstly, an encoder module is used to transfer pure text to numerical representation that captures the seman- tic information. This can be done using techniques like word embeddings or pre-trained language mod- (PLMs).
Variance and Covariance are two important metrics in deep learning. In confidence learning (Mukherjee and Awadallah, 2020), they are neg- atively correlated with the confidence of models and samples. Moreover, in active learning (Cohn et al., 1996), they are correlated with generaliza- tion error and bias which can be indirectly reduced by minimizing variance or covariance. Recently, IDEAL (Guo et al., 2022) uses variance as one of the measures for the inconsistency of unlabeled samples. In this work, we use them estimate the feature representation and correlation between features respectively.
S 1l, ....... [6
(1)
encoder(S)
Where w, indicates the i-th word of the given text; means the length of the text; H E R !xn represents the representation of the text and n is the dimension size.
For NER, the encoded representations are passed through an entity extraction module to identify and classify entities in the text. The formulations of span-level methods are as follows:
3 Method
pspon extractor H (. (iJ)
this section, we first briefly introduce the general inference process of the joint entity and relation
(2)
label5pan classifier( ( espan (i.j)
(i.j)

---

Where 0 sisis indicates the boundary of span; | means the length of the given text: pépan (ig)) is the feature of span; extractor(-) a function to gain the feature of given spans; classifier(-) is a function to predict the label of spans according to its representation.
Where F and W mean the feature vectors and weights; W R is vector representing all sam- ple weights and n the number of samples, hold- ing Σ, W; = 1 by softmax function. F means the i-th dimension feature of all samples. cor(.,) is the covariance function.
For RE, H and the spans of identified entities are used to determine the relations between the entities.
Next, W is used weigh the loss of the corre- sponding sample.
erol extractor( H, spans, spang) (6))
LossCoP =
Σ ==0
(5)
labelte (i.)) = classifier (3.)
Where span; is the entity identified by NER; extractor(-) is a function to acquire the feature of given relations; classifier()) is function to predict the label of relations. In this work, OVO is used to optimize the ((J) and (()) during training.
Where wi E W is the final weight for the i-th sample; pt and & mean the prediction distribution and real label respectively; Lossorig means the loss employed on the original method.
However, there are two difficulties in perform- ing the above process directly. Firstly, the major- ity of PLMs have extensive number of param- eters, ranging from hundreds of millions (Devlin et al., 2019) billions (Du al., 2022b) and even more, thus the computation of weights for the en- tire dataset requires significant computational re- sources, making it expensive and potentially unaf- fordable during training. Secondly, incorporating features from previous batches or epochs may lead to feature inconsistency due to the continuous evo- lution of features during training.
3.2 Covariance Optimizing
When the features cxhibit linear correlations with each other, it can lead to instability in parameter estimation. Consequently, the optimal model be- comes uncertain, and the presence of selection and distribution biases is amplified during inference. This phenomenon is called collinearity (Dormann et al., 2013). For example, y = .1 + 22 and y = 100x; - 98x² have identical performance if assumption क1 = 22 holds in train datasets, but if æ1 = 1.01.rg holds in test datasets, the gaps of them will be huge. Besides collinearity, the corre- lation between features also poses the same impact in deep learning. One straightforward way to re- duce the effect of collinearity and correlation is to analyze them among features and then remove some relevant features. However, due to the in- explicability and excessive number of features in PLMs, this method is difficult to afford.
To overcome the aforementioned difficulties, in- spired by MoCo (He et al., 2020), we only com- pute optimal weights for samples in the current batch, utilizing previous features and weights as ad- ditional fixed information. Furthermore, to ensure feature consistency, we discard the earliest features and weights:
F, F IF 1)] F),FF... Fowr] W, 1W.-ill/w.// + 1)J :], W W
The methods of stable learning (Zhang et al., 2021) show that feature representations will be more causal when giving suitable weights for sam- ples in loss calculating during training. Inspired by this, we propose govariance optimizing (COP) to alleviate the collinearity and correlation between features with low resource consumption, as well as improve the causality of features.
(6)
Where F and W mean the feature vectors and weights; i means the index of the current batch; Lg is the length of #; Sour means feature vectors weights in the current batch; n is the batch number of feature vectors and weights needed to saved; [.] is the floor function; [ means to delete first data.
In COP, covariance is used to measure the cor- relation between features, and the target is ac- quire suitable weights by minimizing the covari- ance among all features using gradient descent:
To further mitigate the feature inconsistency, we fuse and emphasize the current information with previous features and weights. Firstly, we ran- domly sample from Four and the corresponding W ouur until the number of samples is the same as
W = argmin
Σ Ljft)
(cov(WF) 2
(4)
W

---

R
Moreover, the effectiveness of mitigating this bias can be easily quantified.
Causal Intervention
((P)do(F))
As shown in Fig. 2. <- L path, a path that connects F and P but does not start with F, from F to P in which F, I,, and P are feature vectors, real labels, and model predic- tions respectively. Thus, the label distribution from the datasets is the confounder between the feature vectors and the final predictions, creating a spuri- ous correlation that disturbs model inference and causes lower performance. For better understand- ing of the causal graph, we introduce it as follows: The labels datasets are the optimizing target of model training (L + P) which also takes effect on the feature vector learning (I. -> F). Appendix. A shows more detail of the causal graph.
a backdoor
F
Figure 4: The original causal graph of the joint entity and relation extraction task in the classification phase (left) with its alternate after intervention (right).
Fi.
F Four RL]] W cur W cur [2 W our [z]]
F
cur = Feur R
Where F and W our the sets after random sampling: L is the length of F,, R2 is random integer from L, and [i] means getting the i-th element. And then, we fuse current information with the previous.
In order to remove the confounder caused by backdoor path F t L - P and make the model learn direct causality between feature vectors and predictions, we introduce do-operator (Pearl et al., 2016) on feature vectors F to gain the causality of F on P called p(P do( F))- As shown in Fig. 4 right, through do-operator, the edge causing con- founder, F + L. + P, is blocked, so that the spurious correlation has been removed.
F = aF, F1 cur
W = a W; + (1- a)
Where a is hyperparameter between [0, 1] re- garding how much previous information needs to be saved. Finally, F; and Wi will be applied to eq. 4 W and F. Note that F is used to gain
Backdoor adjustment (Pearl, 1995) is method to reduce confounders in causal inference, and the optimal W but only the part Wew will be updated and used to weight sample loss.
can perform do-operator on the observational datasets. Formally, the backdoor adjustment re- duces the bias as:
To enhance the applicability of COP, we use a hyperparameter called COP rate to adjust the degree of influence of COP during training. After acquiring the optimal weights, the loss function is follows:
p(P)do(F)) (
(10)
Where the detail for the equation in Appendix B. Inspired by the eq. 10, it can reduce the spurious correlation if we process the feature vectors of the same label separately. So we propose a method to optimize the feature vectors in the same label by using the property of variance.
wis' = - (1-rcop) + TCOP * :
a
LossCop
ui Lossorig 4))
When the fluctuation among samples is serious, the variance will be large. and vice versa. In other words, we can minimize the variance between the samples of the same label to bring them closer to- gether and at the same time achieve intervention. Along this way, we introduce new loss function using the variance to optimize the feature represen- tation as follows:
Where ui E W the final weight of the i-th sam-
ple: rcop E (0,1) indicates COP rate.
3.3 Variance Optimizing
Inspired by causal intervention (Pearl et al., 2016),
we construct causal graph (Fig. 2) regarding the causality from feature vectors to the final targets. To clarify our objective, in our causal analysis, we focus on the mitigation of bias arising from the long-tail distribution. This bias tends to make la- bels with substantial number of samples more readily predictable during the inference process.
L
Lossvor = ΣΣ var var (F
(11)
1=0 0=
Where is the label amount in one batch of data, N the dimension amount of feature vectors, and

---

F(I.)) means the n-th dimensional feature of all samples of label var(-) is the operation of com- puting sample variance. Note that missing several labels is possible in one batch of data.
task, we use the span-level benchmark evaluate. which one entity is considered recognized correctly if and only both its boundary and type are pre- cisely predicted. For the RE, we adopt two evalua- tion metrics as follows: (1) boundaries evaluation (Rel): a relation is considered recognized correctly if and only if the boundaries of head and tail enti- ties are conrect, and their relation label is correct; (2) strict evaluation (Rel+): in addition the quirement of Rel, the types of two entities are also needed to be correctly predicted.
This method is called yariance optimizing (VOP). Additionally, benefiting from the random mechanism of mini-batch gradient descent, one sample can combine with different samples into the same batch in each training epoch, which brings two main udvantages as follows: Firstly, the diver- sity of sample combination is ensured automati- cally, Secondly, the impact of abnormal samples is mitigated due to the overall optimization.
4.3 Implementation Details
We apply OVO to 3 of out of top-5 strong base- lines specifically PURE (Zhong and Chen, 2021), SpERT.PL (Sai et al., 2021), and PL-Marker (Ye et al., 2022). For ACE2005, the BERT encoders bert-base-uncased (Devlin et al., 2019) and lubbert-xlarge-vIle (Lan et al., 2020) which are in the common domain. For SciERC, the BERT en- coder is criber-sccvvcaab-uncaald (Beltagy et al.. 2019) which is the scientific domain. We use our proposed method, OVO, to optimize the fea- ture vectors as the extra plugin beyond the common model training with all hyperparameters the same as the base models. To ensure robustness, duct experiments with different seeds and report the average scores.
3.4 Overall Optimization Framework
We combine COP and VOP into a unified frame- work called covariance and variance optimization framework (OVO), as shown in Fig. 3, and the al- gorithm. (Appendix. C) illustrates the detailed training procedure of OVO. The final loss is as follows:
Loss final = Losscop + ALossvop (12)
Where À a hyperparameter for the weight of Lossvop-
4 Experiment
4.1
Dataset
We conduct experiments to evaluate our approach two popular joint entity and relation extraction datasets: ACE05 and SciERC² (Luan al., 2018). For ACE2005, we use the same split following (Luan et al., 2019) in the train, development, and test sets. ACE2005 dataset includes various domain data, such as newswire and broadcast news, while the SciERC dataset includes abstracts collected from AI conferences/workshopps. Additionally, to evaluate the effectiveness of OVO in diito---ittribttion settings, conduct experiments on CrossRE (Bassignana and Plank, 2022) which dataset for the relation extraction task across 6 different domains, namely news, poli- tics, natural science, music, literature, and artificial intelligence.
4,4 Baselines
For comparison, we choose serials of state-of-the art models as follows: (1) PURE (Zhong and Chen, 2021) uses two independent models for NER and RE respectively in frustratingly easy way, and is the first to propose levitated markers for RE task called PURE(Approx) to accelerate the inference speed, but with a slight drop in performance; (2) SpERT.PL (Sai al., 2021) uses part-of- part-of-speech and logits entities to enhance the features enti- ties and relations which shows strong performance in the SciERC dataset. (3) PL-Marker (Ye et al.. 2022) is the first to leverage levitated markers to both NER and RE tasks using packed method with great improvement.
4.2 Evalution metries
Additionally, for comparison with other causal debiasing methods, we choose two related methods as follows: (1) CFIE (Nan et al., 2021) 1) uses causal inference to tackle the CoRE (Wang et al., 2022) designs a counterfactual
We follow the evaluation metrics in previous close works to evaluate the experiment results' perfor- mance using the micro Fl measure. For the NER
IE issues; (2)
'https://catalo 1dc. upenn. edu/LDC28e6T66 "http://nlp.a cs. washington. edu/sciIE/ Phttps:
htttp:: //ppper//ppprww////// thoode. com/sota/ sentt-entity-ndee reelation-exracttoon
http:///ggggiu...om//////////// com/maainnp/Crosstt

---

analysis to debias RE by reducing the side-effects of entity mentions.
all weights of feature vectors for computing LossCop. As shown in Table 3, the performance of Rel and Rel+ drop 0.8%-2.2% without COP, which can represent the importance to reduce the spurious correlation among features.
4.5 Results
Table shows the results of comparative experi- ments between proposed method and baselines. As shown, with the scibert-scivocabbmmaati en- coder, our proposed method, OVO, outperforms PURE-F, SpERT.PL, and PL-Marker by +3.3%, +2.1 %, and +2.9% with Rel+ on SciERC respec- tively. Furthermore, on ACE2005, with both bert- base-uncased encoder and albert-xxlarge-v/ en- coder, our method also outperforms the baselines on Rel+. Such improvements over baselines indi- cate the effectiveness and generalization of OVO, Additionally, due to CoRE (Wang et al.. 2022) being only for RE, we compare OVO with CFIE (Nan et al., 2021) and CoRE only in RE using the result of OVO in NER. For the base model, we chose PURE because of SL is joint model that cannot handle NER and RE tasks separately and PL-Marker uses packed method that cannot handle only one pair of entities in RE. As shown in Table 1, CFIE and CoRE can improve performance somewhat but are still lower than OVO. The results can show the effectiveness of OVO compared with other methods. Specifically. on SciERC, the REL+ of OVO was 1.6% higher than CFIE; on Ace2005, the REL+ of OVO was 0.2% higher than CoRE.
w/o. VOP: To verify the effectiveness of variance optimization, make VOP disabled and let only COP work in the training procedure. As shown in Table when removing VOP, the performance of Rel and Rel+ drop 2.0%-3.44% and 1.6%-2.2% respectively, which can indicate the effect of the causal intervention and the VOP play an important role in OVO.
W. SNet: ilar work with COP which use Random Fourier Features to enhance the non-linearity of enhanced features. To compare the effectiveness between COP and StableNet, we replace COP with Sta- bleNet keeping other settings the same, and then negotiate the final performance between them. The performance shows that using COP gains +0.7% and +0.4% improvement in Rel+ on ACE2005 and SciERC respectively. Moreover, from the results in Table 5, the consumption of COP is much smaller than that of StableNet.
StableNet (Zhang al., 2021) is sim-
Analysis
To evaluate the effectiveness of OVO on out-of- distribution, we conduct experiments on an out- domain setting as follows: using the train set of one domain during training, and using the test sets the other five domains during testing. We choose PL-Marker as the basc model and then apply OVO on each domain. As shown in Table. 2, using OVO gains improvements in all cases with average improvement of 1.5% and best improvement of 2.4% in the ai domain.
5.1 Fine-grained analysis
To analyze the debias ability of OVO, we conduct experiments to evaluate the performance of each label on PL-Marker. The detailed results for each relation label on SciERC are shown in Table 4. From the table, the number of USERD-FOR ac- counts for more than half, 54.7% of the total, but that of COMPARE is only 3.9% of the total.
From the results, the two porst-performing la- bels both have a small number of samples which are only 6.9% and 6.4% of the total, respectively. For the comparison between with or without OVO, we find that the performances in all labels gain improvement, except for USED-FOR, which is one that has the most samples in the dataset and keeps comparable performance. Additionally, the improvement increases as the number of samples decreases. The most improvement is on PART-OF, which outperforms +9. 1% in F1 than without VOP. Moreover, the overall results average F1 gain +4.0% improvement.
4.6 Ablation Study
In this section, conduct ablation studies to elu- cidate the effectiveness of different components of our proposed method on the baseline PL-Marker (Ye et al., 2022). We use the evaluation metrics the same as in Table 1 on both ACE2005 and Sci- ERC datasets using bert-be serr-base-unt unse-dd and scibert- scivocab-unccaa as bert encoders respectively.
To evaluate the effectiveness of co- variance optimizing, we remove COP and only use VOP in the model training process and set
w/o. COP:

---

Encoder
ACE200S Rel
SCIRRC
Ent
Rel+
Ent
Rel
Rel+
SPure (Miwa and Bansal, 2016)
LSTM
83.4
55.6
DyGlF, (Luan al., 2019)
ELMo
88.4
63.2
65.2
41.6
OnelE? (Lin et al., 2020)
B1.
88.8
67.5
DyGiE+++ (Wadden
2019)
88.6
63.4
67.0"
47.5"
SPERT (Eberts and Ulges. 2020)
Ba/SaB
86.8
63.8"
70.3
50.8
UniRE' (Wang al.. 2021)
88.8
65.7"
64.3
68.4
40.2"
36.9
SpERT.PL (Sai et al., 2021)
87.4"
65.3"
62.1"
70.5
51.3
39.4
+OVO
87.9
67.9
65.3
71.7
52.6
41.5
PL-Marker' (Ye al., 2022)
89.8
69.0
66.5
69.9
53.2
41.6
+OVO*
89.6
70.6
67.7
70.7
44.5
PURE-FI (Zhong and Chen, 2021)
Ba/SciB
90.1
67.7
64.8
36.8
+CFIE (Nonet al., 2021)
67.7
64.5
50.9
38.5
+CoRF (W anget al., 2022)
67.6
64.9
50.7
38.0
+OVO
90.5
68.0
65.1
70.0
51.6
40.1
UniRE (Wang et al., 2021)
90.2
66.0
PURE-F! (Zhong and Chen, 2021)
ALBxx
90.9
69.4
67.0
PI.-Marker' (Ye al., 2022)
91.1
73.0
71.1
+OVO*
91.2
74.6
72.3
Table 1: Overall test results F1 in joint entity and relation task on the dataset of ACE2005 and SciERC. t: the model or method leverages cross-sentence information extra features. The encoders used in different
models: BB = base-uncased. Bı = Sert-large-uccasee SciB = pinpeert-civoccb-nnccce
ALBxx =
xlbare-- Model name abbreviation: PURE-F PURE(Full), Evaluation metric abbreviation: Rel denotes the boundaries evaluation and Rel+ denotes the strict evaluation. *: the result is not given in the original works, and we rum the corresponding experiment, reporting the results the table.
Rel(%)
news
science
polities
literature
ai
music
avg
PL-Marker
9.7
13.0
14.4
14.8
11.6
14.1
12.9
+OVO
10.3
14.3
15.4
16.4
14.0
16.4
14.5
Table 2: Results in the out-domain setting on the CrossRE dataset.
ACE2005
SciERC
reported in Appendix. D.
Ent
Rel
Rel+
Ent
Rel
Rel+
5.3
Effect of Variance Optimizing
OVO
89.6
70.6
67.7
70.7
56.1
44.5
w/o. COP
89.7
69.7
66.9
70.0
54.1
43.3
To enhance adaptation to diverse methods, we intro- duce weight for Lossvop to mitigate its impact on the normal training of the model and avoid per- formance degradation. The optimal weight varies depending on the specific dataset and method em- ployed, as different weights yield the best perfor- mance. The detailed results are reported in Ap- pendix. E.
VOP
65.5
70.7
52.7
SNet
89.4
70.1
67.0
70.5
55.0
44.1
Table 3: Ablation on ACE2005 and SciERC test set with different component setups. w. SNet means replacing COP with StableNet in model training.
5.2 Effect of Covariance Optimizing
Conclusion
As the results are shown in Table 5, by using Sta- bleNet, it gains +0.6% improvement on Ent, and COP improves on it +0.2%. Table shows more results between COP and StableNet. So, we can conclude that stable learning technology can ac- tually achieve improvement for our task. But the consumption of computing resources is relatively high and it costs approximately and 1.7 times for training time and GPU memory respectively than the original. In comparison, COP uses fewer resources but gets better performance. Addition- ally, more detailed results of hyperparameters are
In this work, we propose the OVO, a novel model training framework for optimizing feature vectors using causal technology. There are two main parts of OVO which are called COP and VOP. From the perspective of the correlation among features, COP reduces the spurious relation by minimizing the covariance computed from all features. From the viewpoint of each feature, VOP makes the feature more causal by minimizing the variance calculated from each feature of the same label. After optimiz- ing the features toward causal, our method achieves

---

Metric (Rel)
UF
CON
EF
HO
PO
FO
COM
avg.
Proportion (%)
54.7
12.6
9.3
6.9
6,4
6.0
3,9
Recall
55.2
62.6
46.2
64.2
28.6
23.7
52.6
47.6
w/o. OVO (%)
Precision
53.6
55.0
51.2
58.1
48.7
25.9
50.0
48.9
F1
54.3
58.6
48.6
60.1
36.0
24.8
$1.3
47.7
Recall
53.1
65.7
49.5
70.2
36.5
25.4
55.3
50.8
W. OVO (%)
Precision
55.4
53.3
54.2
59.5
59.0
35.7
60.0
53,9
Fl
54.2
58.9
51.7
64.4
45.1
29.7
57.5
51.7
-0.1
+0.3
+3.1
+4.3
+9.1
+4.9
+6.2
+4.0
Table 4: The fine-grained results the SciERC dataset.
Relation label abbreviation: UF=USED- FOR, CON=CONIUNCTION, EF=EVALUATE FOR, HO-HYPONYM--0 PO-PART-OF, FO-FEATURE COM-COMPARE, Proportion means the percentage of the number of relation label in total data.
Acknowledgements
Name
Ent
GPU (GB)
Time (h)
PL-Marker
69.9
19
The State Key Program of National Natural Sci- ence of China, Grant/Award Number661533118. National Natural Science Foundation of China, Grant/Award Number: 61402220; The Philos- ophy and Social Science Foundation of Hunan
COP
70.7
24
w. SNee
70.5
32
10
Table 5: The experiment results on the SciERC. The base model is PL-Marker. COP means covariance op- timizing; Time (h) is the time consumption model training. The text with bold and italics indicates the best performance, and that with underlining indicates the second best.
Province, Grant/Award Number: 16YBA323; Natural Science Foundation of Hunan Province, Grant/Award Number: 2020JJ452,20022J3000
Scientific Research Fund of Hunan Provincial Education Department, Grant/Award Number:
18B279, 198299,19A43992220336
improvements on strong baselines and reduces the influence of data unbalanced bias. In the future, we will pay more attention to (I) how to further reduce resource consumption on COP and (2) how can let the weight of VOP adjusts itself automati- cally adapt to the different datasets.
References
Elisa Bassignana and Barbara Plank. 2022. Crossre: A cross-domain dataset for relation extraction. Findings of the Association for Computational Lin- guistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 3592-3600 Association for Computational Linguisties.
Limitations
Iz Beltagy, Kyle Lo. and Arman Cohan. 2019. Scib- ert: A pretrained language model for scientific text. In Proceedings of the 2019 Conference Empiri- cal Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, FMNIP--JCNN 2019, Hong Kong, China, November 3-7, 2019, pages 3613-3615 Association for Computational Linguistics.
OVO novel training framework for optimiz- ing feature representations directly using causal technology. The limitations of OVO are as follows:
COP is an additional training process in each batch that will improve resource consumption. Larger models and more previous informa- tion used in COP will cost more additional resourees.
Razvan C. Bunescu and Raymond J. Mooney. 2005. A shortest path dependency kernel for relation ex- traction. In HLT/EMNLL 2005, Human Language Technology Conference and Conference on Empirical Methods Natural Language Processing, Proceed- ings of the Conference, 6-8 October 2005, Vancouver British Columbia, Canada, pages 724-731. The As- sociation for Computational Linguistics.
To adapt OVO to more models, there some hyperparameters control the degree of effect of COP and VOP during training. These hyperparameters are sensitive differ- ent models and datasets, so takes time to experiment to find the optimal values them.
Huajun Chen, Ning Hu, Guilin Qi, Haofen Wang. Zhen Bi, Jie Li, and Fan Yang. 2021. Openkg chain: A blockchain infrastructure for open knowledge graphs. Data Intell., 3(2):205-227.

---

David A. Cohn, Zoubin Ghahramani, and Michael I. Jordan. 1996. Active learning with statistical models. J. Artif. Intell. Res., 4: 4:129-145.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for un- supervised visual representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Pat- lern Recognition (CVPR), pages 9726-9735.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, NAACL-HLL 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers). pages 4171-4186. Association for Computational Linguistics.
Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Mart- tinen, and Philip S. Yu. 2022. A survey on knowl- edge graphs: Representation. acquisition, and appli- cations. IEEE Trans. Neural Networks Learn. Syst., 33(2):494-55
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut, 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman, 2018. Measuring and mitigat- ing unintended bias in text classification. In Proceed- ings of the 2018 AAAJ/ACM Conference AI, Ethics, and Society, AIES 18, page 67-73, New York, NY, USA. Association for Computing Machinery.
Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. 2020. A joint neural model for information extraction with global features.
Carsten F. Dormann, Jane Elith, Sven Bacher, Carsten Buchmann, Gudrun Carl, Gabriel Carré, Jaime R. Garefa Marquéz, Bernd Gruber, Bruno Lafourcade, Pedro J. Leitão, Tamara Münkemüller, Colin Mc- Clean, Patrick E. Osborne, Björ Reineking, Boris Schröder, Andrew K. Skidmore, Damaris Zurell, and Sven Lautenbach. 2013. Collinearity: a review of methods to deal with it and simulation study evalu- ating their performance. 36())
Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guisties, pages 7999-8009, Online. Association for Computational Linguistics.
Zihan Lin, Changxin Tian, Yupeng Hou, and Wayne Xin Zhao. 2022. Improving graph collaborative filtering with neighborhood- -enrichee contrastive learning. In WWW '22: The A ACM Web Conference 2022, Virtual Event, Lyon, France, April 25 29, 2022, pages 2320- 2329. ACM.
36(1):27-46,
Huifang Du Zhongwen Le. Haofen Wang, Yunwen Chen, and Jing Yu. 2022a. COKG-QA: multi- hop question answering over COVID-19 knowledge graphs. Data Iniell. 4(3),47143377
Yi Luan, Luheng He, Mari Ostendorf. and Hannaneh Hajishirzi. 2018 Multi-task identification of enti- ties, relations, and coreference for scientific knowl- edge graph construction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, Brussels, Belgium, October 31 November 2018 pages 3219-3232. Association for Computational Linguistics.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022b. GLM: general language model pretraining with au- toregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Treland, May 22-27, 2022, pages 320- 335. Association for Computational Linguistics.
Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannanch Hajishirzi. 2019. A general framework for information extraction using dynamic span graphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 3036-3046, Minneapolis, Minnesota. Association for Computational Linguistics.
Markus Eberts and Adrian Ulges. 2020. Span-based joint entity and relation extraction with transformer pre-training In ECAI 2020- 24th European Confer- ence on Artificial Intelligence, 29 August-8 Septem- ber 2020, Santiago de Compostela. Spain, August 29 September 8, 2020- Including 10th Conference on Prestigious Applications of Artificial Intelligence (PAIS 2020), volume 325 of Frontiers in Artificial In- lelligence and Applications, pages 2006-2015 IOS Press.
Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using Istms on sequences and tree structures. In Proceedings of the 54th Annual Meet- ing of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Vol- ume 1: Long Papers. The Association for Computer Linguistics.
Jiannan Guo, Yangyang Kang, Yu Duan, Xiaozhong Liu, Siliang Tang, Wenqiao Zhang, Kun Kuang. Chang- long Sun, and Fei Wu. 2022. Collaborative intel- fusion
ligence orchestration: Inconsistency-base of superviied learning and active learning. In KDD '22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14- 18, 2022, pages 2935-2945 ACM.
Subhabrata Mukherjee and Ahmed Hassan Awadallah. 2020. Uncertainty- aware self-training for text classi- fication with few labels. CoRR, abs/2006.155 15315.

---

Guoshun Nan, Jiaqi Zeng, Rui Qiao. Zhijiang Guo, and Wei Lu. 2021. Uncovering main causalities for long- tailed information extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 9683-9695. Association for Computa- tional Linguistics.
David Wadden, Ulme Wennberg, Yi Luan, and Han- naneh Hajishirzi. 2019. Entity, relation, and event extraction with contextualized span representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP--JCCNPP) pages 5784- 5789, Hong Kong, China. Association for Computa- tional Linguisties.
Judea Pearl. 1995. Causal diagrams for empirical re- search. Biometrika, 82(4))669-688
Yijun Wang. Changzhi Sun, Yuanbin Wu, Hao Zhou, Lei Li, and Junchi Yan. 2021. UniRE: A unified label space for entity relation extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 220-231, Online. Association for Computational Linguisties.
Judea Pearl, Madelyn Glymour, and Nicholas P Jewell. 2016. Causal inference in statistics: A primer. John Wiley & Sons.
Lev-Arie Ratinov and Dan Roth. 2009. Design chal- lenges and misconceptions in named entity recogni- tion. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning, CoNLL 2009, Boulder, Colorado, USA, June 4-5, 2009, pages 147-155, ACL.
Yiwei Wang, Muhao Chen, Wenxuan Zhou, Yujun Cai, Yuxuan Liang, Dayiheng Liu, Baosong Yang. Juncheng Liu, and Bryan Hooi. 2022. Should we rely on entity mentions for relation extraction? debi- asing relation extraction with counterfactual analysis. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 3071-3081. Association for Computa- tional Linguisties.
Farshid Rayhan, Sajid Ahmed, Asif Mahbub, Md. Raf- san Jani, Swakkhar Shatabda, and Dewan Md. Farid. 2017. Cusboost: Cluster-based samplisa with boosting for imbalanced classification. CoRR,
abs/1712.00 111.04456..
Santosh Tokala Yaswanth Sri Sai, Prantika Chakraborty, Sudakshina Dutta, Debarshi Kumar Sanyal, and Partha Pratim Das. 2021. Joint entity and relation extraction from scientific documents: Role of lin- guistic information and entity types. In Proceed- ings of the 2nd Workshop on Extraction and Evalua- tion of Knowledge Entities from Scientific Documents (EEKE 2021) co-located with JCDL 2021, Virtual Event, September 30th, 2021, volume 3004 of CEUR Workshop Proceedings, pages 15- CEUR-WS. .or.g.
Deming Ye, Yankui Lin, Peng Li, and Maosong Sun. 2022. Packed levitated marker for entity and relation extraction. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics ( (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 4904-4917. Association for Computational Linguistics.
Xingxuan Zhang, Peng Cui, Renzhe Xu, Linjun Zhou, Yue Hc, and Zheyan Shen. 2021. Deep stable learn- ing for out-of-distribution generalization. In IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR 2021, virtual, June 19-25, 2021, pages Computer Vision Foundation/ / IEEE.
Yuming Shang. Heyan Huang, and Xianling Mao. 2022. Onerel: Joint entity and relation extraction with one module in one step. In Whirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty- Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intellgence, EAAI 2022 Virtual Event, February 22- March 1, 2022, pages 11855-111 1 1293. AAAI Press.
Zexuan Zhong and Danqi Chen. 2021. A frustratingly easy approach for entity and relation extraction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 50-61. Association for Computational Linguistics.
Zheyan Shen, Peng Cui, Kun Kuang, Bo Li, and Peix- uan Chen. 2018. Causally regularized learning with agnostic data selection bias. In 2018 ACM Multi- media Conference on Multimedia Conference, MM 2018, Seoul, Republic of Korea, October 22-26, 2018, pages 411-419. ACM.
A Causal Graph
From Fig. 2, L + F denotes that real labels in datasets have an effect on the feature vectors learned by model training, and L + P denotes that real labels have an effect on model predictions be- cause the loss in training is calculated by labels and predictions. So the backdoor path F < I. >> P makes models give relatively high scores to the labels with numerous samples.
Zheyan Shen, Peng Cui. Tong Zhang, and Kun Kuang. 2020. Stable learning via sample reweighting. In The Thirty-Found AAAI Conference on Artificial Intelli- gence, AAAI 2020, The Thirty- Secron- Innovative Ap- plications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 5692- 5699. AAAI Press.

---

B Derivation Process of Backdoor Adjustment
Algorithm 1 The training procedure of the OVO
Input: Initial pre-trained BERT encoder BE(") and classi- fier C training dataset DD training epochs for model training 7 Tixt, training epochs for COP Teo
In this section, we will give an explanation for each step of backdoor adjustment. To make it easier to understand, we draw equation follows:
Output: Trained model (BE(Tme)) CC(mm))
1: for time to Tm do
Fpre Clear the previous feature vectors;
3: W Clear the previous weights; 'D do
for (Sents,,pana
p(Pld((F)) L)((L((
5: H BB((me)) (Sents)
6:
F Four GelSponnFeaturese
(
2
((PIdo(F),
W eur Initial
concal( pro: Feur)
10:
= concal pres Weur)
3
(P(LFF)
for teop =
Teop do Lw = Eies cou WW,, WF,)
13:
W <- Optim(Lw))
Ep(PIF,L)p(U
14:
end for
15:
(F pro: W pre) + update and fuse Lorig Crosspintropu Spans)
16:
(13)
17:
LCOP W WourCorig
Where the meaning of P. F, and L are the same Fig. 2; do(-) means do-operator. The underline means the main differences from the previous step.
18:
Lvop = Σ/ In var(F (1.o))
19:
C LOOp LvOp
20:
BB C Optim(C)
21:
end for
22:
end for
(1) using the total probability theorem where do(F) can be seen as normal random vari- ables event like P and L.
74.5
PL. Marker
(2) from the causal graph in Fig. 2, the path L, + F is direct causal path without back- door paths, so p(L]do(F)) and F) are equivalent.
Relation F1 (%)
74.0
73.5
(3) because of conditioning L, the association in the path F < I. P is blocked. So in
73.0
this case, the association in the path F - P is causality which can remove the do-operator without any influence.
72.5
2"5
2n6
2^7
2^8
2n9
inf
Saving Batch Number
(4) due the direction of the causal path between L and being +- L, there is no association from to which causes the L inde- pendent with F meaning p( (F) = p(L).
Figure 5: The impact of different Saving Batch Num- bers on the performance of COP is evaluated using the SciERC test set. The inf notation denotes the scenario where all feature vectors are used for COP without any deletions. Relation F1 means evaluating with gold en- tities in Rel+; for 8, is equal to 0.9 in the experi- ments.
C Overall Algorithm of OVO
The algorithm of OVO is shown in the algorithm. 1.
D Analysis of COP
different saving batch numbers on PL-Marker (Ye et al., 2022). As shown in Fig. 5, as the saving batch number rises. the overall trend of Fl also rises and reaches the best performance in. (around 210 in our training setting). It can be concluded that using more vectors for COP can reduce spurious correlation better and gain more improvement, but require more computing resources.
D.1 Saving Batch Number
When the dataset is large enough, is hard to put all feature vectors into COP and optimize them once. So, by using eq. 6, we can use part of the vectors to relieve the pressure on computing resources. To research the effect of COP when just using partial vectors, we conduct comparative experiments with

---

41.0
SPORT.
40.5
SciERC REL+ (%)
70
40.0
Ent(%)
69
39.5
39.0
PL-Marke
67
SPERT.PL
38.5
0.0
0.5 alpha
1.0
38.0
0.0
0.2
0.4
0.6
0.8
1.0
COP rate
P1.Mader STERTTTT
Figure 7: Effect of the different weights of VOP. F1 is Rel on datasets.
54
Rel (%)
0.4 on ACE2005 and SciERC respectively. The de- tailed results for the COP rate are shown in Fig. 7 which only uses COP in experiments.
52
50
E Analysis of VOP
0.0
1.0
alpha
E.1 Tradcoff on Variance Optimizing
For better adaptation to different tasks, it is impor- tant to give Lossyop weight, preventing it influence the model training and hurt the perfor- mance. We compute the final loss with weighed Lossyop OP as follows:
44
Rel+ (%)
42
Loss final LossCoP + ALossvop (14)
Where À is hyperparameter regarding the weight of Lossyop- From the results in Fig. 8, differ- ent datasets get the best performance in different weights of VOP which X = 0.2 is the optimal value for ACE2005 and X = 0.1 is that for SciERC, and the performance is particularly sensitive to the weight which changes drastically using different X.
38
0.0
0.5
1.0
alpha
Figure 6: Effect of different of COP.
D.2 Effeet of alpha
To explore the effect of fusing current information, we conduct experiments for fusing hyperparame- ter a of eq. 8. We fix the saving batch number to infinite and change the a in the same training configuration. Fig. shows the detailed result of the experiments.
D.3 COP rate
We use TCOP in PURE (Zhong and Chen, 2021) and PL-Marker (Ye et al., 2022). For SpERT.PL, we set the COP rate equal to 0.1 and

---

FL-Muker SPIRT PURE
ACE2005 Rel (%)
0.1
0.2
0.3
0.4
0.5
lambda
PLMazke
56
SciERC Rel (%6)
52
50
0.0
0.1
0.2
0.3 lambda
0.4
0.5
Figure 8: Effect of the different weights of VOP. F1 Rel on datasets.


Reviews:
[{"review_id": "c7jjfkdIhF", "reviewer": "Reviewer_J2YN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful review of our paper. Your insightful comments and valuable feedback have significantly improved the quality of our work. In this response, we provide a detailed response to each of your points.\n\n1. In the context of variance optimization, our primary objective is to address the bias stemming from the presence of long-tail distributions, which is induced by an imbalance in labels within datasets. We shall provide a comprehensive exposition of the causal graph as delineated below:\n\n     To simplify the description, we use the same abbreviations in the paper: \n\n     L : Real Label\n\n     F : Feature vector\n\n     P : Prediction\n\n    In our causal graph, P represents the outputs of the model, while L represents the real labels of samples. Throughout the training process, our primary objective is to minimize the disparity between P and L, a task typically quantified using a loss function, such as cross-entropy. Consequently, to capture and depict this relationship between L and P, we introduce an edge **L -> P** within the causal graph. \n\n   Moreover, within datasets characterized by a long-tail distribution of labels, it is observed that labels associated with a higher frequency of occurrence tend to receive more frequent predictions from the model. This observation is a direct outcome of the model's learning process, wherein optimization is primarily directed toward enhancing accuracy with respect to the predominant class. Consequently, to represent the impact of label distribution on the feature vectors of data samples, we introduce an edge, symbolized as **L -> F**, into our causal graph. \n\n   In order to ameliorate the aforementioned phenomenon, we propose the incorporation of a novel optimization target during the training phase. By minimizing the variance in the feature vectors of each label, as shown in Table 4, the performances of the labels with a fewer number of samples gain significant improvement.\n\n  \n\n2. Collinearity and correlation phenomena manifest as a consequence of interrelationships between variables. In the context of covariance optimization, we employ the covariance metric to quantitatively assess the degree of correlation existing between variables. In COP, our primary aim is to mitigate this correlation. We posit that a reduction in covariance signifies a diminishment of shared associations among variables, consequently leading to a decrease in both collinearity and correlation among these variables.\n\n   \n\n3. We appreciate your suggestion and intend to conduct a comprehensive review of the pertinent literature pertaining to causal inference.\n\n   \n\n4. The concern in question was discussed within our paper. Specifically, documented in Appendix D.1, we evaluate the impact of saving batch numbers.\n\n   \n\n5. Thank you for your valuable feedback. We appreciate your suggestion to analyze the proposed method from a more general perspective and to explore its effectiveness in the context of causal representation learning. While we acknowledge the importance of these directions, our current research is focused on joint entity and relation extraction tasks. It's indeed challenging to cover all possible applications and theoretical aspects in a single study. \n\n**[5 continued]**\n\nTo further evaluate the applicability of OVO, we conduct supplement experiments during rebuttal. \n\n- 1) We apply OVO to the baseline of CrossRE [1]. The task is Relation Classification which needs to predict the relation type of entity pairs having a relation. The results is reported in (Supplement Table A1).\n\n- 2) To evaluate the effectiveness of the out-domain of OVO, we conduct out-domain experiments using CrossRE.\n  - Firstly, we evaluate in overall out-domain setting which uses the train set of one domain during training, using the test sets of the other all domains during testing. \n  - Secondly, we evaluate in fine-grained out-domain setting which uses the train set of one domain during training, using the test set of another different domain during testing. \n\nThe results of the overall out-domain setting and the fine-grained out-domain setting are shown in (Supplement Table A2) and (Supplement Table A3) respectively.\n\t\n(Supplement Table A1): Results in CrossRE baseline. $w.$ and $w/o.$ mean using OVO or not.\n\n| MICRO-F1(%) | news | politics | science | music | literature | ai   | avg  |\n| ----------- | ---- | -------- | ------- | ----- | ---------- | ---- | ---- |\n| w.o.        | 46.4 | 58.3     | 40.1    | 76.0  | 67.7       | 45.4 | 55.7 |\n| w.          | 48.0 | 58.8     | 40.5    | 77.1  | 68.5       | 46.2 | 56.5 |\n\n(Supplement Table A2): Results in overall out-domain setting. $w.$ and $w/o.$ mean using OVO or not.\n\n| Rel(%) | news | science | politics | literature | ai   | music | avg  |\n| ------ | ---- | ------- | -------- | ---------- | ---- | ----- | ---- |\n| w/o.   | 9.7  | 13.0    | 14.4     | 14.8       | 11.6 | 14.1  | 12.9 |\n| w.     | 10.3 | 14.3    | 15.4     | 16.4       | 14.0 | 16.4  | 14.5 |\n\n(Supplement Table A3): Results in fine-grained out-domain setting. All values in the table are Rel(%). $w.$ and $w/o.$ mean using OVO or not; | means the same as above.\n\n| train | test       | w.o  | w.   | train   | test       | w.o  | w.   |\n| ----- | ---------- | ---- | ---- | ------- | ---------- | ---- | ---- |\n| news  | ai         | 10.1 | 10.3 | science | news       | 22.9 | 24.6 |\n| \\|    | literature | 7.5  | 8.8  | \\|      | ai         | 15.6 | 22.3 |\n| \\|    | music      | 6.2  | 7.1  | \\|      | literature | 10.2 | 11.3 |\n| \\|    | science    | 9.6  | 11.8 | \\|      | music      | 9.7  | 11.4 |\n| \\|    | polotics   | 14.2 | 16.7 | \\|      | polotics   | 16.6 | 18.2 |\n\n[references]\n\n[1] Elisa Bassignana, Barbara Plank. CrossRE: A Cross-Domain Dataset for Relation Extraction[J], EMNLP (Findings), 2022: 3592-3604.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The experiments are well-designed and are, in my opinion, sufficient to support the points.\n2. The proposed method achieves significant improvements in several datasets of joint relation and entity extraction task.", "reasons_to_reject": "1. In variance optimizing, the proposed SCM is not detailed enough. Why Real Label -> Prediction and why Real Label -> Feature vector? In Line 331-335 and Appendix A, the authors roughly state that it is because of model training, lacking in-depth discussion. This does not convince me that SCM is correct. It is worth noting that [1] attempts to address the long-tail problem from the perspective of optimizer momentum. I think the authors also need to provide an in-depth theoretical analysis of these causes and effects reflected in model training.\n2. In the covariance optimizing, the authors did not make it clear why the proposed method can reduce the effect of collinearity and correlation. Like above, more analysis is needed.\n3. I think some related work on causal inference needs to be reviewed. It is expected that more works related to the paper would be discussed, e.g., [1] [2] [3]. \n4.  lack of analysis of different batch sizes, which is important for MoCo training.\n5. The proposed method should not only be applied to information extraction, I think the authors need to analyze the theory behind the method from a more general perspective. And it is suggested to study the effectiveness of the method from a previous setting of causal representation learning.\n\n[1] Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect\n\n[2] Uncovering Main Causalities for Long-tailed Information Extraction\n\n[3] De-biasing Distantly Supervised Named Entity Recognition via Causal Intervention", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper tried to solve the problem of distribution bias in joint entity and relation extraction, such as long-tail and out-of-distribution, from a causal perspective. One method of debiasing (covariance optimizing) optimizes the feature covariance and adjusts the sample weights to eliminate collinearity and correlation; the other method (variance optimizing) is based on backdoor adjustment to minimize the within-class variance. The authors conducted experiments on joint entity and relation extraction task and achieved significant improvement in several datasets.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "trEBJ2qpyw", "reviewer": "Reviewer_debR", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our gratitude for dedicating your time to the examination of our scholarly contribution. In light of your comprehensive review, we hereby express our intention to meticulously address each of the points you have raised, providing a thorough and detailed response.\n\n**[The causal analysis has flaws.]**\n\nWe apologize for the slight discussion of the causal analysis. We will detailly describe the causal analysis as follows:\n\nTo simplify the description, we use the same abbreviations in the paper: \n\nL : Real Label\nF : Feature vector\nP : Prediction\n\nIn our causal graph, P represents the outputs of the model, while L represents the real labels of samples. Throughout the training process, our primary objective is to minimize the disparity between P and L, a task typically quantified using a loss function, such as cross-entropy. Consequently, to capture and depict this relationship between L and P, we introduce an edge L -> P within the causal graph. \n\nMoreover, within datasets characterized by a long-tail distribution of labels, it is observed that labels associated with a higher frequency of occurrence tend to receive more frequent predictions from the model. This observation is a direct outcome of the model's learning process, wherein optimization is primarily directed toward enhancing accuracy with respect to the predominant class. Consequently, to represent the impact of label distribution on the feature vectors of data samples, we introduce an edge, symbolized as L -> F, into our causal graph. \n\nWe will answer the questions below this comment as follows:\n\n1. Is the SCM for the forward or backward pass?\n\n   The proposed SCM is for backward pass. Our initial focus is on the phenomenon that the long-tail distribution within datasets impacts the performance of labels associated with a small number of samples. Subsequently, we undertake a causal analysis to mitigate this influence.\n\n2. Is each node in the SCM a distribution or a single instance?\n\n   The nodes in the SCM are distributions. Our objective is to analyze the influence of labels on the model training procedure.\n\n3. Is L in the SCM a single label or all the same labels?\n\n   L represents all labels, encompassing various labels rather than a single label or identical labels. The grouping of samples based on labels is our strategy to alleviate the impact of label distribution, and it is not directly related to SCM.\n\n**[The authors do not provide convincing evidence to show that the proposed method can be explained by causal theory. ]**\n\nTo our best knowledge, backdoor adjustments do not necessarily require a change in the value of the data, and statistical methods can be used to control for or account for the effects of confounders by appropriately modeling their influence, without requiring active changes to the value or distribution of the confounders. For example, in Simpson's Paradox, backdoor adjustment is implemented by statistical methods.\n\nMoreover, there are many recent works to introduce causal technology into the deep learning field using different implementation methods. And part of them does not change the value/distribution of the confounder. For example, [1] does not change the dataset, but designs counterfactual decoders for backdoor adjustment; [2] proposes a method to approximate\nthe causal intervention for reducing bias influence, and utilize it during training.\n\nVOP utilizes minimizing variance between samples of the same label to equalize the significance of different labels. To do so, the long-tail distribution of labels has reduced the impact on feature vectors. Thus the effect of **L -> F** is blocked which approximates to do-operator. As shown in Table 4 in the paper, the performances of the labels with a fewer number of samples gain significant improvement. \n\n**[The evaluation is not sound. No baseline for bias mitigation is compared. The method is only evaluated in an in-domain setting.]**\n\nFor other bias mitigation methods, we have undertaken a comparative analysis between our proposed approach and StableNet which is similar to COP. The outcomes of these experimental comparisons have been documented in Tables 3 and 5. The findings therein affirm that our method outperforms StableNet in terms of both performance metrics and resource consumption.\n\nFor out-domain settings, we apologize for missing the corresponding experiments due to the focus of our paper being the conventional joint entity and relation extraction. To supplement the out-domain experiments, we conduct experiments in the dataset CrossRE [3] which is a cross-domain dataset for relation extraction. The dataset is available in [4]. There are six domains in CrossRE, namely news, politics, science, music, literature, and ai. \n\nFirstly, we apply OVO to the baseline in [3] to evaluate the effectiveness of our proposed method. The task of baseline is in-domain Relation Classification which needs to predict the relation type of entity pairs having a relation. As shown in (Supplement Table B1), OVO gains improvements in all domains of CrossRE with an average improvement of 0.8% and a best improvement of 1.6% in the news domain.\n\n(Supplement Table B1): Results in CrossRE baseline. $w.$ and $w/o.$ mean using OVO or not.\n| MICRO-F1(%)| news | politics | science | music | literature | ai   | avg  |\n| -------- | ---- | -------- | ------- | ----- | ---------- | ---- | ---- |\n| w.o.     | 46.4 | 58.3     | 40.1    | 76.0  | 67.7       | 45.4 | 55.7 |\n| w.       | 48.0 | 58.8     | 40.5    | 77.1  | 68.5       | 46.2 | 56.5 |\n\nThen, due to the missing Relation Extraction results in [3], we conduct experiments in CrossRE dataset using PL-Marker [5] in two out-domain settings: \n\t1) Overall out-domain setting: using the train set of one domain during training, using the test sets of the other five domains during testing. \n\t2) Fine-grained out-domain setting: using the train set of one domain during training, using the test set of another different domain during testing. \n\nIn the overall out-domain setting, as shown in (Supplement Table B2), using OVO gains improvements in all cases ith an average improvement of 1.5% and a best improvement of 2.4% in the ai domain.\n\n(Supplement Table B2): Results in overall out-domain setting. $w.$ and $w/o.$ mean using OVO or not.\n|  Rel(%)  | news | science | politics | literature | ai   | music | avg  |\n| ---- | ---- | ------- | -------- | ---------- | ---- | ----- | ---- |\n| w/o. | 9.7  | 13.0    | 14.4     | 14.8       | 11.6 | 14.1  | 12.9 |\n| w.   | 10.3 | 14.3    | 15.4     | 16.4       | 14.0 | 16.4  | 14.5 |\n\nIn the fine-grained out-domain setting, as shown in (Supplement Table B3), due to the limitation of time, we only conduct experiments in the news and science domains. The results show the out-domain effectiveness of OVO.\n\n(Supplement Table B3): Results in fine-grained out-domain setting. All values in the table are Rel(%). $w.$ and $w/o.$ mean using OVO or not; | means the same as above.\n| train | test       | w.o  | w.   | train   | test       | w.o  | w.   |\n| ----- | ---------- | ---- | ---- | ------- | ---------- | ---- | ---- |\n| news  | ai         | 10.1 | 10.3 | science | news       | 22.9 | 24.6 |\n| \\|    | literature | 7.5  | 8.8  | \\|      | ai         | 15.6 | 22.3 |\n| \\|    | music      | 6.2  | 7.1  | \\|      | literature | 10.2 | 11.3 |\n| \\|    | science    | 9.6  | 11.8 | \\|      | music      | 9.7  | 11.4 |\n| \\|    | polotics   | 14.2 | 16.7 | \\|      | polotics   | 16.6 | 18.2 |\n\nThe results in (Supplement Table B2) and (Supplement Table B3) show the out-domain effectiveness of OVO. We will provide supplement experiments if we have a chance to upload the next version of our paper. Your feedback is greatly appreciated, and we are committed to enhancing the quality of our research.\n\n**[references]**\n\n[1] Yiquan W, Kun K, Yating Z, Xiaozhong L, Changlong S, Jun X, Yueting Z, Luo S, Fei W, et al. De-Biased Court’s View Generation with Causality[J], Computer Science, 2020, 2020.emnlp-main: 763-780.\n\n[2] Xiaobao Wu, Chunping Li, Yishu Miao. Discovering Topics in Long-tailed Corpora with Causal Intervention.[C], Annual Meeting of the Association for Computational Linguistics, 2021, 2021.findings-acl: 175-185.\n\n[3] Elisa Bassignana, Barbara Plank. CrossRE: A Cross-Domain Dataset for Relation Extraction[J], EMNLP (Findings), 2022: 3592-3604.\n\n[4] https://github.com/mainlp/CrossRE\n\n[5] Deming Y, Yankai L, Peng L, Maosong S, et al. Packed Levitated Marker for Entity and Relation Extraction[C], Annual Meeting of the Association for Computational Linguistics, 2022, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers): 4904–4917.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The results show that covariance optimization helps to improve model performance on entity and relation extraction.", "reasons_to_reject": "* The causal analysis has flaws. The authors confuse several concepts in the proposed SCM. Is the SCM for the forward or backward pass?\nIs each node in the SCM a distribution or a single instance? Is L in the SCM a single label or all the same labels? The authors use these contradicting meanings to explain the SCM in different parts of the paper. \n* The authors do not provide convincing evidence to show that the proposed method can be explained by causal theory. To be more specific, equation 11 is not a reasonable realization of equation 10. To perform backdoor adjustment, it is essential to change the value/distribution of the confounder proactively. Minimizing \"the variance between the samples of the same label\" (line 360) is not a causal intervention based on the proposed SCM.\n* The evaluation is not sound. No baseline for bias mitigation is compared. The method is only evaluated in an in-domain setting.\n\n*After several rounds of discussion with the authors, I believe their causal analysis has significant flaws. First, the proposed SCM and follow-up analysis are not coherent. Some of the nodes, links, and terms in the equations do not have concrete meanings, and the authors change the definition of variables in different steps of the proof. For example, the definition of $L$ is inconsistent in the analysis and this results in a weird term $p(P|F,L)$. Second, the most important step that connects the causal analysis and the proposed covariance minimization method is based on a **wild assumption** that a strong correlation between $F$ and $L$ will lead to $p(P|do(F)) = p(P|F) = p(P|F,L)$. This assumption challenges the basic of causal intervention and is unrealistic in nature.*", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper analyzes the spurious correlations in entity and relation extraction. The authors propose a covariance optimization method to improve the model performance.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "\n"}, "extra_text": {}}, {"review_id": "YsR7FYwTVT", "reviewer": "Reviewer_qNHL", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. We appreciate your insightful comments and valuable feedback, which have undoubtedly improved the quality of our work. In response to your review, we would like to address each of your points in detail.\n\n**[include more details such as training time and GPU memory usage]**\n\nThank you for bringing this to our attention. We wholeheartedly agree that providing more comprehensive information about resource consumption is indeed valuable for a better understanding of our proposed method. As illustrated in (Supplement Table C1), we have included a breakdown of resource consumption specifically for the SciERC dataset. We will provide a comprehensive overview of overall resource consumption in the next version of our paper. Your feedback is greatly appreciated, and we are committed to enhancing the quality of our research.\n\n(Supplement Table C1):  Resource consumption of OVO. $w.$ and $w/o.$ mean using OVO or not.\n\n| Name       | Task    | OVO | GPU(GB) | Time(h) |\n|------------|---------|-----|---------|---------|\n| PL-Marker  | NER     | w/o. | 19     | 2.0     |\n|            |         | w.  | 24     | 4.0     |\n|            | RE      | w/o. | 4      | 0.3     |\n|            |         | w.  | 6      | 0.5     |\n| PURE       | NER     | w/o. | 6      | 1.0     |\n|            |         | w.  | 19     | 1.2     |\n|            | RE      | w/o. | 8      | 0.9     |\n|            |         | w.  | 12     | 1.1     |\n| SPERT.PL   | NER+RE  | w/o. | 11     | 0.2     |\n|            |         | w.  | 18     | 0.5     |\n\n**[include analysis for other data biases besides the long-tail distribution bias.]**\n\nWe agree with this about analyzing other biases. During rebuttal, we conduct experiments to evaluate the effectiveness of OVO in out-of-distribution. \n\nWe conduct experiments in CrossRE[1] dataset using PL-Marker [2] in two out-domain settings: \n- 1) Overall out-domain setting: using the train set of one domain during training, using the test sets of the other five domains during testing. \n- 2) Fine-grained out-domain setting: using the train set of one domain during training, using the test set of another different domain during testing. \n\nIn the overall out-domain setting, as shown in (Supplement Table C2), using OVO gains improvements in all cases ith an average improvement of 1.5% and a best improvement of 2.4% in the ai domain.\n\n(Supplement Table C2): Results in overall out-domain setting. $w.$ and $w/o.$ mean using OVO or not.\n|  Rel(%)  | news | science | politics | literature | ai   | music | avg  |\n| ---- | ---- | ------- | -------- | ---------- | ---- | ----- | ---- |\n| w/o. | 9.7  | 13.0    | 14.4     | 14.8       | 11.6 | 14.1  | 12.9 |\n| w.   | 10.3 | 14.3    | 15.4     | 16.4       | 14.0 | 16.4  | 14.5 |\n\nIn the fine-grained out-domain setting, as shown in (Supplement Table C3), due to the limitation of time, we only conduct experiments in the news and science domains. The results show the out-domain effectiveness of OVO.\n\n(Supplement Table C3): Results in fine-grained out-domain setting. All values in the table are Rel(%). $w.$ and $w/o.$ mean using OVO or not; | means the same as above.\n| train | test       | w.o  | w.   | train   | test       | w.o  | w.   |\n| ----- | ---------- | ---- | ---- | ------- | ---------- | ---- | ---- |\n| news  | ai         | 10.1 | 10.3 | science | news       | 22.9 | 24.6 |\n| \\|    | literature | 7.5  | 8.8  | \\|      | ai         | 15.6 | 22.3 |\n| \\|    | music      | 6.2  | 7.1  | \\|      | literature | 10.2 | 11.3 |\n| \\|    | science    | 9.6  | 11.8 | \\|      | music      | 9.7  | 11.4 |\n| \\|    | polotics   | 14.2 | 16.7 | \\|      | polotics   | 16.6 | 18.2 |\n\nThe results in (Supplement Table C2) and (Supplement Table C3) show the out-of-distribution effectiveness of OVO. \n\n**[references]**\n\n[1] Elisa Bassignana, Barbara Plank. CrossRE: A Cross-Domain Dataset for Relation Extraction[J], EMNLP (Findings), 2022: 3592-3604.\n\n[2] Deming Y, Yankai L, Peng L, Maosong S, et al. Packed Levitated Marker for Entity and Relation Extraction[C], Annual Meeting of the Association for Computational Linguistics, 2022, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers): 4904–4917.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed framework alleviates data biases and enhances feature representation.\n- It achieves noticeable improvement when applying to 3 strong baselines for joint entity and relation extraction task", "reasons_to_reject": "- The framework involves two optimization problems, which could be slow and expensive. It would be beneficial to include more details such as training time and GPU memory usage.\n- It would be better to include analysis for other data biases besides the long-tail distribution bias.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces a covariance and variance optimization framework (OVO) to optimize feature representations from a causality perspective for joint entity and relation extraction tasks. Inspired by stable learning, they propose covariance optimizing (COP) to reduce collinearity of each feature, get the causal effects, alleviate the data selection and distribution bias. In addition, they propose variance optimizing (VOP) to minimize the variance of each dimension in the feature vectors of the same class label, mitigating the long-tail distribution bias further. This framework achieves improvement over 3 strong baselines.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ODWAwdo2J4", "reviewer": "Reviewer_5RrX", "summary": "", "questions": "1. In the causal graph, why B leads to L? Does it mean the real label is only based on bias? Also, does it mean there is no bias in features?\n2. In the equation in line 245, are there constraints on W? At least they should be nonnegative and sum up to 1?\n3. How is the solution of W calculated?\n4. In line 297, adding two features or weights of different samples does not make sense.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. We appreciate your insightful comments and valuable feedback, which have undoubtedly improved the quality of our work. In response to your review, we would like to address each of your points in detail.\n\n1. In the context of out-of-distribution settings, we extend our apologies for the omission of experiments. This omission stems from the focus of our research on the conventional task of joint entity and relation extraction. To supplement the corresponding experiments, we conduct experiments in dataset CrossRE [1] which is a cross-domain dataset for relation extraction. There are six domains in CrossRE, namely news, politics, science, music, literature, and ai. \n\n   Firstly, we apply OVO to the baseline in [1] to evaluate the effectiveness of our proposed method. The task of baseline is in-domain Relation Classification which needs to predict the relation type of entity pairs having a relation. As shown in (Supplement Table D1), OVO gains improvements in all domains of CrossRE with an average improvement of 0.8% and a best improvement of 1.6% in the news domain.\n\n(Supplement Table D1): Results in CrossRE baseline. $w.$ and $w/o.$ mean using OVO or not.\n\n| MICRO-F1(%) | news | politics | science | music | literature | ai   | avg  |\n| ----------- | ---- | -------- | ------- | ----- | ---------- | ---- | ---- |\n| w.o.        | 46.4 | 58.3     | 40.1    | 76.0  | 67.7       | 45.4 | 55.7 |\n| w.          | 48.0 | 58.8     | 40.5    | 77.1  | 68.5       | 46.2 | 56.5 |\n\n\n​    Then, due to the missing Relation Extraction results in [1], we conduct experiments in CrossRE dataset using PL-Marker [2] in two out-domain settings: \n\n- 1) Overall out-domain setting: using the train set of one domain during training, using the test sets of the other five domains during testing. \n\n- 2) Fine-grained out-domain setting: using the train set of one domain during training, using the test set of another different domain during testing. \n\n  In the overall out-domain setting, as shown in (Supplement Table D2), using OVO gains improvements in all cases ith an average improvement of 1.5% and a best improvement of 2.4% in the ai domain.\n\n(Supplement Table D2): Results in overall out-domain setting. $w.$ and $w/o.$ mean using OVO or not.\n\n| Rel(%) | news | science | politics | literature | ai   | music | avg  |\n| ------ | ---- | ------- | -------- | ---------- | ---- | ----- | ---- |\n| w/o.   | 9.7  | 13.0    | 14.4     | 14.8       | 11.6 | 14.1  | 12.9 |\n| w.     | 10.3 | 14.3    | 15.4     | 16.4       | 14.0 | 16.4  | 14.5 |\n\nIn the fine-grained out-domain setting, as shown in (Supplement Table D3), due to the limitation of time, we only conduct experiments in the news and science domains. The results show the out-domain effectiveness of OVO.\n\n(Supplement Table D3): Results in fine-grained out-domain setting. All values in the table are Rel(%). $w.$ and $w/o.$ mean using OVO or not; | means the same as above.\n\n| train | test       | w.o  | w.   | train   | test       | w.o  | w.   |\n| ----- | ---------- | ---- | ---- | ------- | ---------- | ---- | ---- |\n| news  | ai         | 10.1 | 10.3 | science | news       | 22.9 | 24.6 |\n| \\|    | literature | 7.5  | 8.8  | \\|      | ai         | 15.6 | 22.3 |\n| \\|    | music      | 6.2  | 7.1  | \\|      | literature | 10.2 | 11.3 |\n| \\|    | science    | 9.6  | 11.8 | \\|      | music      | 9.7  | 11.4 |\n| \\|    | polotics   | 14.2 | 16.7 | \\|      | polotics   | 16.6 | 18.2 |\n\nThe results in (Supplement Table D2) and (Supplement Table D3) show the out-of-distribution effectiveness of OVO. \n\n2. In datasets with a long-tail distribution of labels, where certain labels are more common and others are rarer, machine learning models tend to favor predicting the more abundant labels to optimize accuracy. To capture this phenomenon in our causal graph, we introduce an edge labeled L -> F to represent how the label distribution influences the feature vectors of data samples.\n\n   Our VOP approach addresses this issue by minimizing variance among samples of the same label, effectively reducing the impact of the long-tail label distribution on feature vectors. As a result, the effect denoted as L -> F is blocked, approximating a do-operator. The outcomes of this approach, as demonstrated in Table 4 of our paper, show significant performance improvements for labels that are less common in the dataset.\n\n3. We appreciate your feedback regarding the need for proper symbol definitions and consistent usage throughout the paper. We will thoroughly review the paper to identify any undefined symbols and ensure that all symbols are used consistently and appropriately. \n\n**[Questions]**\n\n1. In the causal graph, why B leads to L? Does it mean the real label is only based on bias? Also, does it mean there is no bias in features?\n\n   B -> L means the bias affects the models through labels. The bias of long-tail distribution is caused by data collection, which produces large differences in sample sizes under different labels. This phenomenon will influence the model during training, which leads to labels with a greater abundance of samples tending to be predicted more often by the model.\n\n   It does not mean the real label is only based on bias, but bias in the data collection process contributes to the long-tail distribution of labels. The features are initially without bias due to the random initialization, but the bias will further affect features during training.\n\n2. In the equation in line 245, are there constraints on W? At least they should be nonnegative and sum up to 1?\n\n   We use the softmax function to constrain the values of weights, which seem missing in the paper. We apologize for this and will fix it in the paper.\n\n3. How is the solution of W calculated?\n\n   We use an extra training phase to acquire the final weight using gradient descent. Appendix C provides an overall algorithm of OVO and lines 11-14 show how to acquire the final weights.\n\n4. In line 297, adding two features or weights of different samples does not make sense.\n\n   Due to the continuous updating of features during training, feature inconsistency may arise. Line 297 combines the features from the current batch with those saved from previous batches to mitigate the impact of previous features and ensure feature consistency. Therefore, in this context, adding the features or weights of different samples is meaningful as it contributes to maintaining feature consistency and improving the overall performance of the model. Additionally, Appendix D.2 illustrates the effect of the fusion operation.\n\n**[references]**\n\n[1] Elisa Bassignana, Barbara Plank. CrossRE: A Cross-Domain Dataset for Relation Extraction[J], EMNLP (Findings), 2022: 3592-3604.\n\n[2] Deming Y, Yankai L, Peng L, Maosong S, et al. Packed Levitated Marker for Entity and Relation Extraction[C], Annual Meeting of the Association for Computational Linguistics, 2022, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers): 4904–4917.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Mitigating spurious correlations in information extraction models is an important problem, especially for real-world applications where the data is often out-of-distribution. However, this problem is not sufficiently addressed in the literature, and this paper makes valuable contributions to this topic.\n2. Previous methods for mitigating spurious correlations for IE models are mainly conducted in the inference phase, this paper introduces innovative techniques to tackle this problem during the training phase.", "reasons_to_reject": "1. The main claim of reducing dataset bias is not sufficiently supported in experiments. Current experiments only include results on in-distribution test data, long-tailed classes, and some analysis of computational efficiency. However, I think a more important experiment to show the results on out-of-distribution data, which can be another domain of text (with the same entity/relation types), unseen entities, noisy labels, etc.\n2. The motivation behind the variance optimization method needs to be clarified. I do not understand why reducing variance is related to backdoor adjustment.\n3. Properly defining and consistently using symbols is crucial. Some symbols in the paper are not defined, and some are used more than one time.\n\nEdit after rebuttal: As the authors have added results on OOD examples, I raised the soundness score.", "questions_to_authors": "1. In the causal graph, why B leads to L? Does it mean the real label is only based on bias? Also, does it mean there is no bias in features?\n2. In the equation in line 245, are there constraints on W? At least they should be nonnegative and sum up to 1?\n3. How is the solution of W calculated?\n4. In line 297, adding two features or weights of different samples does not make sense.", "paper_topic_and_main_contributions": "Entity and relation extraction models often suffer from a variety of dataset biases, leading to reduced robustness and generalization. To tackle this problem, this paper presents a novel causal framework featuring two techniques, covariance optimization and variance optimization, both aiming to mitigate spurious correlations. Experiments on two benchmarks show that the proposed framework show improved results when integrated into various backbone models.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Since W is a vector in Eq. 4, better to use a bold lowercase w.\n2. For each equation, remember to add a comma after each line, and make the \"Where\" after the equations lowercase.\n3. The N and L are used multiple times with different meanings. Try to use different letters."}}]

Author rebuttal:

Meta-review: This work introduces a covariance and variance optimization framework (OVO) to debias NER and RE tasks that is based on structural causal inference like a few other debiasing methods, but leads to more fine-grained adjustment than most of those. The technical novelty of this method is generally acknowledged by the reviewers. The paper originally missed important results on out-of-distribution evaluation, which is necessary for debiasing methods. There have also been a series of important comparative methods there were originally missed. The authors have provided some of those in the discussion phase, and the AC believe that these are essential to be included into the draft should the paper be accepted. In addition, there seems to be flaws identified regarding the SCM used behind the method, for which the discussion didn't reach consensus among the authors and the reviewers.
Decision: Accept-Findings