Constrained Tuple Extraction with Interaction-Aware Network
Year: 2023

Abstract:
Tuples extraction is a fundamental task for information extraction and knowledge graph construction. The extracted tuples are usually represented as knowledge triples consisting of subject, relation, and object. In practice, however, the validity of knowledge triples is associated with and changes with the spatial, temporal, or other kinds of constraints. Motivated by this observation, this paper proposes a constrained tuple extraction (CTE) task to guarantee the validity of knowledge tuples. Formally, the CTE task is to extract constrained tuples from unstructured text, which adds constraints to conventional triples. To this end, we propose an interaction-aware network. Combinatorial interactions among context-specific external features and distinct-granularity internal features are exploited to effectively mine the potential constraints. Moreover, we have built a new dataset containing totally 1,748,826 constrained tuples for training and 3656 ones for evaluation. Experiments on our dataset and the public CaRB dataset demonstrate the superiority of the proposed model. The constructed dataset and the codes are publicly available.1

heading: 1 Introduction
text: Tuples extraction task aims to extract knowledge tuples from unstructured texts, which is a fundamental task for information extraction, knowledge graph construction, and so on (Cui et al., 2018; Jiang et al., 2019; Banerjee and Baral, 2020; Li et al., 2022). The extracted knowledge tuples are mainly represented in the form of (subject, relation, object) (Jiang et al., 2019), and are usually acquired through named entity recognition (NER), relation extraction (RE) (Zhong and Chen, 2021; Li et al., 2020; Yahya et al., 2014), and open information extraction (Open IE) (Jiang et al., 2019; Wang et al., 2022). With the rise of large-scale pretraining methods (e.g. BERT (Devlin et al., 2019)),
∗Corresponding author. 1https://github.com/ckgconstruction/ckg.
the quality of knowledge triples has improved significantly.
However, current knowledge triples lack constraints for their authenticity. In practice, constraints are ubiquitous in numerous domains, such as spatial, temporal, conditional and environmental ones. Constraints are essential supplements to knowledge triples, and play an instructional role. Our deep investigation on literature indicates that this topic has not gained enough attention.
Some works utilize temporal knowledge tuples to reflect temporal dynamics (Gracious et al., 2021; Jung et al., 2021). However, only temporal constraints are not sufficient and general enough to guarantee the validity of knowledge triples. For instance, let us consider the following sentence:
“Consuming the same power, the performance of ARM CPU is better than that of Intel CPU.”, only if the conditional constraint “consuming the same power” is satisfied, the knowledge triple “(ARM CPU, better than, Intel CPU)” is true. The above observations motivate us to add constraints to conventional knowledge triples for the validity of knowledge tuples. Thereby, we propose a novel task called constrained tuple extraction (CTE), which aims to provide knowledge tuples with temporal, spatial, conditional constraints in general domain. CTE represents knowledge tuples in the form of (subject, relation, object, constraint). Constraints are some phrase descriptions that guarantee the validity of knowledge tuples, so that the knowledge tuples can be effectively utilized.
The constrained tuples are extracted via Open IE approaches, similar to the works in Jiang et al. (2019); Wang et al. (2022). This extraction approach is more general and doesn’t require predefined entity and relation types. Table 1 shows some examples for CTE task. It is worth noting that the constrained tuple extraction task aims at providing an explicit and uniform information extraction technique to guarantee the validity of knowledge
11430
(or information) tuples, rather than a new form of knowledge representation.
In addition, an interaction-aware network is developed to fulfill CTE task from unstructured texts. We argue that constraints in knowledge tuples affect the validity and are contained in deep-level conceptual semantics rather than shallow contexts. Interactions among multiple textual features can further mine implicit semantic information beyond context, which is beneficial to discovering potential constraints.
Architecturally, the interaction-aware network consists of three modules: context-specific enhancement module, distinct-granularity feature extraction module and interaction-aware module. Specifically, the first module is designed to learn the context-specific external features by leveraging multi-view graphs. The second module is to sufficiently extract the distinct-granularity (e.g. phraselevel, word-level, and global-level) internal context features. The third module is developed to achieve the goal of interaction between the external and internal features in a combination way utilizing distribution-sparse multi-head attention. The combinatorial interactions occur between any two external or internal features. Thus, this trick yields a mechanism in that the deep-level conceptual semantic can be explored to help mine the potential constraints existing in knowledge.
The contributions of this paper are summarized as follows:
• We propose a novel task called constrained tuple extraction (CTE), whose mission is to extract knowledge tuples with temporal, spatial, conditional constraints. A new manually annotated Constrained Tuple Extraction Benchmark (CTEB) dataset for the CTE task is built and publicly available.
• Interaction-aware network (IAN) is proposed
to fulfill CTE task, which facilitates the combinatorial interactions among the contextspecific external features and the distinctgranularity internal features to effectively mine the potential constraints in knowledge.
• Distribution-sparse multi-head attention is designed not only to select the dominating attentions but also to facilitate efficient interactions.

heading: 2 Related Work
text: The goal of the tuples extraction task is to extract knowledge tuples from unstructured texts (Jiang et al., 2019; Banerjee and Baral, 2020; Li et al., 2022). The extracted knowledge tuples are mainly acquired through relational triple extraction (Zhong and Chen, 2021; Li et al., 2020; Yahya et al., 2014) and open information extraction (Jiang et al., 2019; Wang et al., 2022).

heading: 2.1 Open Information Extraction
text: Open information extraction (Open IE) aims to extract predicates and corresponding arguments from unstructured texts in open domain, without predefining entity and relation types. The extracted predicates and corresponding arguments can constitute knowledge tuples. Open IE methods mainly include rule-based ones (Fader et al., 2011; Corro and Gemulla, 2013; Angeli et al., 2015) and neural network based ones (Stanovsky et al.; Cui et al., 2018; Wang et al., 2022). The neural network models are further divided into sequence labeling ones (Stanovsky et al.; Roy et al., 2019; Jiang et al., 2020), sequence generation ones (Cui et al., 2018; Sun et al., 2018; Kolluru et al., 2020b) and spanbased ones (Zhan and Zhao, 2020).
Specifically, Ro et al. (2020) proposed Multi2OIE to utilize BERT to extract predicates.
Then the BERT hidden feature, position embedding and predicate average feature were input into multi-head attention blocks to extract arguments. Solawetz and Larson (2021) proposed SRL_BERT to improved RnnOIE (Stanovsky et al., 2018) by replacing the bidirectional encoder with BERT and the predicate index embedding with sentence embedding. Jiang et al. (2019) proposed a three-layer structure for scientific tuple extraction using Open IE methods. The tuples were divided into fact tuples and condition tuples. In addition, OpenIE6 (Kolluru et al., 2020a) utilized two-dimensional grid labeling to improve the extraction efficiency for Open IE task. Later, DetIE (Vasilkovsky et al., 2022) regarded the tuples as three-dimensional anchor boxes, and improved the extraction speed by that single-shot approach. Tuples extraction based on Open IE has better generality, because it does not require predefined entity and relation types. Open IE task focuses on the structure of predicates and corresponding arguments, while CTE task focuses on providing a more unified form to ensure the validity of knowledge tuples.

heading: 2.2 Relational Triple Extraction
text: Relational triple extraction is mainly accomplished by named entity recognition (NER) and relation extraction (RE) (Fu et al., 2020; Zhong and Chen, 2021).
Typically, Li et al. (2020) extracted relational triples from free texts in the e-commerce field based on NER and RE. Moreover, Zhong and Chen (2021) explicitly injected positions and categories information of entities into the input sentences for relation extraction, so that different contextual representations were learned for entities and relations. Nevertheless, relational triple extraction based on NER and RE is usually limited to the domain portability.

heading: 3 Methodology

heading: 3.1 Problem Definition
text: Given a piece of text m, the constrained tuple extraction task is to extract nt (>=1) constrained knowledge tuples formatted as t = (subject, relation, object, constraint) from each sentence in text m. The constraints can be formatted as temporal expressions, spatial descriptions, and conditional forms. Formally, the constrained knowledge tuples t can be formulated as:
t = (s, r, o, c), (1)
where s, r, o, c represent the subject, relation, object and constraint, respectively.

heading: 3.2 Overview of the Proposed Model
text: Figure 1 shows the architecture of the proposed interaction-aware network (IAN) for CTE task. IAN consists of three modules: context-specific enhancement module, distinct-granularity feature extraction module, and interaction-aware module. First, context-specific enhancement module is developed to leverage multi-view graphs and learn context-specific external features for input text. Second, to sufficiently exploit the inherent features of the text, distinct-granularity internal features are explicitly extracted from the raw text. Finally, interaction-aware module is designed to facilitate the combinatorial interactions between any two features in the context-specific external and the distinct-granularity internal features. Meanwhile, distribution-sparse multi-head attention is proposed to select the dominating attentions and alleviate the interaction deficiency problem.

heading: 3.3 Context-Specific Enhancement Module
text: External auxiliary features can provide additional semantic information for constrained tuple extraction. The existing methods usually introduce entityspecific external knowledge. The introduced external features only consider the entity itself rather than the context of the entity. It results in that the introduced external features are indistinguishable for the same entities in different texts (Li et al., 2020). In this paper, different from them, context-specific external features are introduced for spans in the text according to the contexts of spans. Thus, the introduced external features can provide contextspecific auxiliary information and enrich raw texts.
In this work, Wikidata2 is used to leverage and generate preliminary auxiliary information. First, the candidate spans in the input text are matched and aligned with the data in Wikidata. Thus, the potential entity nodes in Wikidata corresponding to the candidate spans can be obtained. Then, centered on the potential entity nodes, their two-hop neighbors and edges in Wikidata are regarded as preliminary auxiliary information. Therefore, the two-hop graph structures G are generated according to the aligned spans and Wikidata.
To generate context-specific external features, multi-view graphs are generated from G for each
2https://www.wikidata.org/wiki/Wikidata
aligned span. Inspired by the works in Nathani et al. (2019) and Xue et al. (2021), the auxiliary information is represented in the form of graph, which contains nodes (potential entities) and edges (attributes or relations) in Wikidata related to the aligned spans. Specifically, both sentence and the contents of the nodes in the related two-hop graph structure are input into BERT (Devlin et al., 2019). For each aligned span in each sentence, the initial external auxiliary information is defined as V0 = {v01, v02, ..., v0T+1}. Each element in V0 corresponds to a node in G. Gaussian graph generator is used to generate the potential multi-view graphs. The contextual feature is captured when encoding each node into Gaussian distributions:
{μ1i , μ2i , ..., μNi } = fθ(v0i , hCLS), (2)
{σ1i , σ2i , ..., σNi } = φ(f ′ θ(v 0 i , hCLS)), (3)
where θ denotes SoftPlus activation function, hCLS is the representation of CLS token, fθ and f ′ θ represent two learnable neural networks, N is the number of views. Then a series of Gaussian distributions {δn1 , δn2 , ..., δnT+1} are obtained for the n-th view in the multi-view graph.
Next, Kullback-Leibler (KL) divergence between the Gaussian distributions of two nodes is used to generate weights for edges. In addition, due to the asymmetry of KL divergence, the obtained multi-view graph is a directed graph. The edge weight between two nodes in the n-th view is
calculated as:
enij = KL(δ n i (μ n i , σ
n2 i )||δnj (μnj , σn 2 j )). (4)
Totally, N adjacency matrices M1,M2, ...,MN
are acquired after getting the weights of edges between nodes. Hence, we can get the multi-view graphs Gm = {V0,M1,M2, ...,MN}.
Next, the embedding generation method KBGAT (Nathani et al., 2019) is used to generate embeddings for the nodes in Gm (i.e. the aligned spans in text). The KBGAT method considers both nodes and edges features to generate embeddings for graphs, and uses hinge-loss as the training objective. The inputs of KBGAT are node embeddings matrix E ∈ Rle×de and edge embeddings matrix R ∈ Rlr×dr . Here, le and lr represent the total numbers of nodes and edges respectively, de denotes the dimension of each node embedding, dr indicates the dimension of each edge embedding. The adjacency matrices in KBGAT are replaced by Gm to generate embeddings for the nodes in Gm. Then, context-specific external features Fk can be obtained for aligned spans in text. More details about Fk generation are shown in Appendix A.3.

heading: 3.4 Distinct-Granularity Feature Extraction Module
text: Aiming to sufficiently exploit the inherent textual features, distinct-granularity internal features are extracted from the raw text.
Since subjects, objects, relations and constraints usually appear in form of phrases in real world, phrase-level features play an important role in constrained tuple extraction. AutoPhrase (Shang et al., 2018) is employed to explicitly mine the phrases in the text. To generate contextualized phrase features, we insert phrase start tag < PHRASE > and phrase end tag < /PHRASE > into sentences. Then, the BERT (Devlin et al., 2019) embeddings of the phrase tags and the words in phrases are averaged to obtain the phrase-level feature Fp.
Fp = BERT(AutoPhrase(m)). (5)
Then, BERT is utilized to generate embeddings for words and sentences. The hidden states of last four layers of BERT are added to generate wordlevel feature Fw. The sentence-level feature Fs is represented by the [CLS] representation in the hidden states of last layer of BERT. The representations in the last layer of BERT are denoted as BERT hidden Fr.
In order to enable the internal and external features to interact directly with the text global feature, the phrase-level, sentence-level, word-level and external features are fused to obtain the textual heterogeneous fused feature Fg. By doing so, Fg is taken as a separate feature and can be utilized to mine the combinatorial interactions in the interaction-aware module. By using the multimodal fusion method EMFH (Xue et al., 2023; Yu et al., 2018), different from them, residual connections are added among the multiple EMFB blocks to fuse Fp,Fs,Fw and the context-specific external feature Fk. We denote the improved fusion method as ResEMFH. The EMFB block is calculated as:
zie = DP (ϕ(Ũ T kFk ◦ ŨTwFw ◦ ŨTs Fs ◦ ŨTpFp)),
(6) ziq = Norm(SumPool(ze)), (7)
where ϕ denotes the tanh activation function, DP represents dropout operation, zie and z i q stand for the outputs of expand stage and squeeze stage in the i-th EMFB block, respectively.
Next, Lg EMFB blocks are then cascaded via residual connections. Finally, the outputs of Lg EMFB blocks are averaged to acquire textual heterogeneous fused feature Fg:
zi+1q = z i q + EMFB(z i e,Fk,Fw,Fs,Fp), (8)
Fg = Mean(z 1 q , z 2 q , ..., z Lg q ). (9)
To provide predicate mentions for constrained tuple extraction, we utilize predicate feature and position embedding used in the work of Ro et al. (2020). Here, predicate feature is obtained by averaging and duplicating the hidden states of the predicates. Position embedding uses binary values to represent the positions of predicate spans. Finally, word-level, sentence-level, phrase-level textual features Fw, Fs, Fp, BERT hidden Fr, heterogeneous fused feature Fg, predicate feature and position embedding together constitute distinct-granularity internal features.

heading: 3.5 Interaction-Aware Module
text: The purpose of the interaction-aware module is to mine the combinatorial interactions between any two features in context-specific external features and distinct-granularity internal features. These combinatorial interactions contain implicit semantics and deep-level correlations besides the contextual information. Previous works usually use multi-head attention for information extraction (Ro et al., 2020; Vaswani et al., 2017). However, we observe that the self-attention used in multi-head attention suffers from interaction deficiency and attention sparsity problems. That is, interactions occur only among minority specific features, rather than among all external and internal features, as illustrated in Section 4.4. Meanwhile, a part of query key pairs dominate the main attention weights, and there are many non-key attention weights.
For above purposes, we propose distributionsparse multi-head attention mechanism, which selects the dominating attentions in each head, and facilitates the interactions among context-specific external features and distinct-granularity internal features. It is worth noting that only the selected dominating query-key pairs are calculated.
The general self-attention takes query Q, key K and value V as inputs. In this module, contextspecific external and distinct-granularity internal features are concatenated to form the feature Farg, which is utilized to generate key-value pairs and query. Farg itself is regarded as a query. Key-value pairs are subsets of Farg derived from the predicate positions. Let qi, ki, vi denote the i-th row in Q, K, V, respectively. LQ, LK , LV represent the numbers of rows in Q, K, V. Following previous works (Tsai et al., 2019; Zhou et al., 2021), the
attention of the i-th query is calculated as:
Attention(qi,K,V) = ∑
j
k(qi, kj)∑ l k(qi, kl) vj ,
(10)
p(kj |qi) = k(qi, kj)∑ l k(qi, kl) , (11)
where k(qi, kj) selects the exponential kernel exp(qik j / √ d), d represents the input dimension. The quadratic dot-product computation is required in self-attention, and the memory usage scales in O(LQLK). This is a bottleneck to enhance the information extraction capacity.
It can be observed from Eq.(10) and Eq.(11) that the attention weight of the i-th query is obtained by calculating the compatibility p(kj |qi). The output is a combination of attention weights and values v. The dominating query-key pairs encourage the attention distribution away from the mean distribution q(kj |qi) = 1/LK . Inspired by Zhou et al. (2021), by measuring the “difference” between distribution p and distribution q, important query-key pairs can be distinguished. Using dominating query-key pairs can filter out redundant interactions and allow the model to focus on effective combinatorial interactions. Meanwhile, the attention sparsity problem can be alleviated, and the computation and memory usage can be reduced. For this purpose, Kullback-Leibler divergence is utilized to measure the “difference”. The sparsity measurement for the i-th query is formulated as:
S(qi,K) = ln
LK∑
j=1
e qik j√ d − 1
LK
LK∑
j=1
qik j√ d , (12)
where the two terms in Eq.(12) are the Log-SumExp and the arithmetic mean of qi on all the keys, respectively. The distribution probability p is more diverse and more likely to include important querykey pairs when S(qi,K) for the i-th query is larger. To further reduce the calculation for traversal of queries when computing the sparsity measurement, according to Calafiore et al. (2020), the sparsity measurement can be empirically approximated as:
S̃(qi,K) = max j
{ qik T j√ d } − 1 LK
LK∑
j=1
qik T j√ d . (13)
Subsequently, only LK lnLQ dot-product pairs are randomly sampled to calculate the S̃(qi,K). The other pairs are filled with zero. According
to the above sparsity measurement, distributionsparse attention is designed to make each key focus only on Top-h dominating queries:
Attention(Q,K,V) = softmax( Q̃K √
d )V,
(14) where Q̃ is a sparse matrix and contains only the sparse Top-h queries. A constant sampling factor c is utilized to control h = c ∗ lnLQ. Thus, for each query-key lookup, the distribution-sparse selfattention calculates O(lnLQ) dot-product. Meanwhile, the memory usage is O(LK lnLQ). Different sparse query-key pairs are generated and calculated in each head. Then sparse Top-h from them are selected. Here, queries and keys has the same shape, thus time complexity and space complexity of the distribution-sparse self-attention are reduced to O(NlnN).
The CTE task is regarded as a sequence labeling task. Similar to the previous work Multi2OIE(Ro et al., 2020), two-stage extraction strategy is adopted. Predicates are first identified, and then subjects, objects, and constraints are extracted. In the first stage, BERT hidden Fr is input into a fully connected (FC) layer to classify predicates. In the second stage, the context-specific external features Fk and the distinct-granularity internal features are concatenated to form feature Farg. Then, Farg is fed into the distribution-sparse multi-head attention followed by another FC layer to obtain the final constrained knowledge tuples. Cross-entropy loss is used to train the IAN model, same as that in Multi2OIE.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Datasets: To verify the constrained tuple extraction task and the interaction-aware network, two public datasets are utilized in this section. The first dataset is a new manually annotated dataset called Constraint Tuples Extraction Benchmark (CTEB) built in this paper. We manually reannotate the sentences from validation set and test set of CaRB3 (Bhardwaj et al., 2019) and a part of sentences of LSOIE4 (Solawetz and Larson, 2021) dataset. BIO annotation is used to label subjects, relations, objects, and constraints in sentences from CTEB dataset, as described in Section 1 and Sec-
3https://github.com/dair-iitd/CaRB 4https://github.com/Jacobsolawetz/large-scale-oie
tion 3.1. A total of 2174 sentences with 3656 constrained knowledge tuples are annotated for the CTE task. We detail the annotation procedure in Appendix A.2. The dataset is divided into training set, validation set and test set using the split of 6:2:2. The second dataset is the commonly used CaRB dataset for Open IE task, which contains predicates and the corresponding arguments from 1282 sentences.
Transfer learning is utilized to train and evaluate the CTE task. The bootstrapped OpenIE4 dataset5 is used as the training set in the first training stage. The model is first trained on a subset of OpenIE4 training set, which can improve the training speed. Then the model is trained and evaluated on CTEB dataset using transfer learning. Moreover, 1,748,826 constrained knowledge tuples are extracted from the 1,109,411 sentences in OpenIE4 dataset by utilizing the proposed IAN model, providing a training set for future works. The number N of adjacency matrices maintained for each matched span is set to 3. Additional experimental details are listed in Appendix A.1. Evaluation metrics: P(Precision), R (Recall), F1 (F1-score), and AUC (the area under the curve) are used to evaluate the performance of different models. Tuple match6 (Bhardwaj et al., 2019) is used as the matching function.

heading: 4.2 Experimental Results
text: The proposed IAN model is compared with the following models: OpenIE4 (Mausam, 2016), SpanOIE (Zhan and Zhao, 2020), BIO (Zhan
5https://github.com/zhanjunlang/Span_OIE 6https://github.com/dair-iitd/CaRB
and Zhao, 2020), Multi2OIE (Ro et al., 2020), BERT+BiLSTM (Ro et al., 2020), SRL_BERT (Solawetz and Larson, 2021), OIE@OIA (Wang et al., 2022) and DetIE (Vasilkovsky et al., 2022). OpenIE4 is a traditional rule-based Open IE method, and the other models extract tuples based on neural networks.
Table 2 shows the performances of different models on CTEB and CaRB datasets. Those two datasets are used to evaluate CTE task and Open IE task, respectively. From the experimental results in Table 2, we can observe that:
1) By exploiting the combinatorial interactions among context-specific external features and distinct-granularity internal features, the proposed IAN model outperforms other models on both CTE and Open IE tasks. In terms of F1, compared with the state-of-the-art methods DetIE and Multi2OIE, our IAN achieves the best performance at 63.8% and 54.6% with increases of 2.5% and 2.3% on CTE and Open IE tasks. The superiorities of IAN mainly include: a) it introduces context-specific external features; b) distinct-granularity internal features are extracted from the raw text to sufficiently mine the inherent textual features; c) the combinatorial interactions between any two features in above external and internal features are effectively mined.
2) In both CTE and Open IE tasks, there is a gap between the metrics R (recall) and P (precision). It indicates that the number of tuples extracted from raw text is usually insufficient compared to the gold tuples.
3) Compared with the Multi2OIE model, the main differences are that our IAN model leverages
external and internal features and alleviates the interaction deficiency and the attention sparsity problems. The superiority of the IAN model shows the effectiveness of rich semantic features, combinatorial interactions and dominating attentions selection in information extraction tasks.

heading: 4.3 Ablation Study
text: To illustrate the contributions of different modules in the IAN model, we design ablation experiments on the IAN model for CTE task. The ablation study results are shown in Table 3. “w/o external” means removing the context-specific external features. “w/o interact” model uses general multi-head attention rather than the distributionsparse multi-head attention. “w/o internal” model removes phrase-level, sentence-level, word-level and heterogeneous fused features. “w/o fuse” denotes removing the heterogeneous fused feature. “IAN-DBPedia” and “IAN-YAGO” represent using external knowledge graphs DBPedia and YAGO instead of Wikidata when generating external features.
From Table 3, it can be seen that the performance of the IAN model degrades significantly when removing the distribution-sparse multi-head attention, indicating that combinatorial interactions and attention sparsity affects the performance of constrained tuple extraction. The “w/o external” model performs worse than the “w/o internal” model, which illustrates that the contribution of external features is larger than that of internal features. Additionally, the IAN model is not sensitive to the choices of different external knowledge bases.

heading: 4.4 Visualization
text: The combinatorial interactions among the contextspecific external features and the distinctgranularity internal features which utilize general multi-head self-attention and distribution-sparse multi-head attention in IAN model are visualized in Figure 2(a) and Figure 2(b), respectively.
It can be observed that interactions occur only among minority specific features in general multihead attention, as shown in Figure 2(a). It is difficult for external and internal features to interact with each other. In contrast, for the distributionsparse multi-head attention in Figure 2(b), interactions conduct among multiple features, and only the selected dominating query-key pairs are calculated. The distribution-sparse multi-head attention can effectively facilitate effective interactions and reduce the computation and memory usage.
(a) General Multi-head Attention (b) Distribution-sparse MultiHead Attention
Figure 2: Attention Visualization.

heading: 5 Conclusion and Future Work
text: In this paper, a novel task called Constrained Tuple Extraction (CTE) has been proposed, which aims to guarantee the validity of knowledge tuples and represent the constrained information in the real world more accurately. Interaction-aware network is designed to fulfill CTE task, which can effectively extract constrained knowledge tuples from raw texts. The proposed network can introduce the contextspecific external features, sufficiently extract the distinct-granularity internal features from texts, and effectively exploit the combinatorial interactions among external and internal features. Meanwhile, distribution-sparse multi-head attention is developed to facilitate combinatorial interactions and alleviate the interaction deficiency and attention sparsity problems. Extensive experiments demonstrate that the proposed IAN model outperforms present methods. In the future, we will address
the issues about the vertical domain-oriented constrained tuple extraction.
Limitations
The open information extraction methods may amplify the bias of the corpus by extracting any relation occurring in the data. The models with deep learning may learn the relation bias from the training corpus and extract those biased statements. To mitigate the effect of data bias, we try to balance the relations in constrained tuples and the ratio of constraints when constructing the CTEB dataset. In addition, the utilization of external auxiliary information increases additional computation time. Our IAN model has still achieved superior performance when the external auxiliary information is removed.

heading: Acknowledgements
text: The work is supported by the National Key Research and Development Program of China (2020AAA0104903) and the National Natural Science Foundation of China (62072039). We thank the anonymous reviewers for their valuable comments and feedback.

heading: A Appendix
text: A.1 Experimental Details Datasates and Training. For the CTEB dataset, we reproduce the baselines. For the CaRB dataset, we report the experimental results from the papers of the baselines. To enhance the training process, transfer learning is used to train the proposed interaction-aware network (IAN). First, a subset containing 20,000 sentences from OpenIE4 dataset is used to train the IAN model. Then, the output layer of the IAN model is modified to the form required by the constrained tuple extraction (CTE) task. Next, the IAN model is trained, validated and tested on the newly annotated Constraint Tuples Extraction Benchmark (CTEB) dataset. For the CaRB dataset, all the models (including our IAN model) are trained the same way as the existing Open IE works, and are not trained on the CTEB dataset. For the CTEB dataset, all the deep learning models (including our IAN model) are trained on the OpenIE4 training set and the training set of the CTEB dataset, and then the results are reported on the CTEB test set. We ensure that our model and the baseline models are trained and evaluated in the same way to guarantee fair comparisons.
The statistics of the datasets used in this paper are shown in Table 4. Parameter Settings. AdamW optimizer (Loshchilov and Hutter, 2019) is used to train the IAN model. The initial learning rate is set to 3e-5. The steps of warmup is set to one-tenth of the total training steps. The epochs for the
first and second training stages are set to 20 and 100, respectively. The IAN model includes 4 distribution-sparse multi-head attention blocks with 8 attention heads. The dropout rates for the attention blocks and classification layer are set to 0.05 and 0.2, respectively. The constant sampling factor c in distribution-sparse multi-head attention is set to 5. Pytorch is used to implement the IAN model. We have conducted all experiments on the Ubuntu system with 3090 GPU. Data Bias Mitigation. To mitigate the effect of data bias, we try to balance the relations in constrained tuples and the ratio of constraints when constructing the CTEB dataset. The ratio of temporal, spatial, and conditional constrained tuples is 845:465:246. During evaluation, the CTEB dataset is randomly divided into training, validation, and test set to alleviate the data bias. During training, we first utilize transfer learning to perform warm up training on the large Open IE dataset OpenIE4, and then train the model on the CTEB training set, which can also alleviate the bias problem. Other Details. As for the implementation details of the competing models, the inputs have not been changed, and only the output layer is modified. The output layer is changed from the original classification of predicate and arguments to the classification of subject, relation, object, and three constraints. The rest of the model details have not been changed. Experimental results reported in this paper are averaged over three different seed settings. For the ablation experiments, the hyper-parameter settings are the same as those of the final model, except for the removal of specific modules.
In this work, AUC (Area Under Curve) is calculated from a plot of the precision and recall values for all potential cutoffs. Matching a tuple accurately is challenging. The reason lies in that both constrained tuple extraction and Open IE tasks re-
quire how to match different spans containing multiple words in a sentence.
From the experimental results, we can observe that the low AUCs is due to the low recall values. For the proposed IAN model, the precision achieves 0.719 and 0.635 on the CTEB dataset and the CaRB dataset respectively, while the recalls are 0.574 and 0.479. This fact indicates that our model predicts most correct tuples in the predicted results, however, it does not cover the ground truth tuples well. Th reason lies in that there are multiple true tuples in each sample, and each tuple contains spans of different elements including subjects, relations, objects and constraints. Therefore, there are many complex elements to be predicted in each sentence. However, our IAN model has outperformed the state-of-the-art methods.
In the model design, the constraint identification is taken as a process of multi-class classification. Architecturally, in the head of our designed IAN model, there is a layer to predict the category of the constraints. That is, the three kinds of constraints are regarded as different classes, and we employed a fully connected layer in our IAN model to achieve this goal. Therefore, our proposed model is applicable to other constraints, and only needs to modify the classes of the constraints in the output layer. In this way, our IAN model renders extensibility in usage.
A.2 Annotation Procedure
The sentences in the CTEB dataset are selected from the CaRB dataset and the LSOIE dataset. When selecting sentences, we try to pick sentences that contain constraints as much as possible, and do not make any changes to the sentences for subsequent annotation. Each sample was annotated by three graduate students, one of whom did the preliminary annotation, and the other two checked and corrected the annotations.
We use brat annotation tool7 to annotate as many constraints as possible in the selected sentences. Three steps are performed when performing annotation task: 1) identifying subjects, relations, and objects; 2) identifying constraints, including temporal, spatial, and conditional constraints; 3) combining subjects, relations, objects, and constraints to acquire the constraint tuples.
We abide by the following principles when annotating sentences: 1) completeness: all subjects,
7http://brat.nlplab.org/
relations, objects and constraints in the sentences need to be annotated; 2) assertedness: all constraint tuples are implied by the original sentences; 3) atomicity: the quadruples are used as the atomic tuples.
When there are multiple constraints or multiple (subject, relation, object) triples or both in a sentence, one related constraint is added to each (subject, relation, object) triple. When (subject, relation, object) triples and constraints exist in a manyto-many situation, the copy operation for triples and constraints is performed so that the quadruples are still atomic tuples.
A.3 External Feature
To utilize external features, additional external knowledge related to the input text needs to be acquired. The purpose of choosing Wikidata is that Wikidata can provide the external knowledge related to the spans in the sentence. The external knowledge provided by Wikidata includes the properties and entities related to the spans.
To obtain external auxiliary information, we enumerate the spans in the sentence setting a maximum length (5 words in this work), and then call the query interface of Wikidata by inputting the enumerated spans. Next, the spans that can receive a response containing contents from the interface are regarded as the aligned spans. In other words, aligned spans refer to the spans that can be queried as nodes in Wikidata.
In theory, the complexity of maintaining N adjacency matrices is O(n3). In fact, since the amount of the nodes in external auxiliary information is not large, the calculation consumption is acceptable. The average number of matched spans in each sample is about 1.5. The number of nodes contained in the external information for each matched span is 20 (5 nodes for the first hop, 15 nodes for the second hop). N represents the number of adjacency matrices maintained for each matched span, and is set to 3 in the experiments. Thus, the calculation of maintaining N adjacency matrices for each sample is about 1.5 ∗ (3 + 1) ∗ (20 + 1) ∗ (20 + 1).
“N multi-views” represents N potential external knowledge graph structures. Previous related works built a fixed external knowledge graph structure for input text. Comparatively, we generate N distinguishable external knowledge graph structures for spans in text via N multi-views. Typically, each view selects the contributed edges and
nodes from the original two-hop external knowledge graph structure, which is accomplished by utilizing sentence contextual information to generate Gaussian distributions and weighting the edges based on the KL divergence. Thereby, “N multiviews” can generate non-fixed context-specific external knowledge graph structures for spans.
In this way, external auxiliary information can be used more effectively and flexibly by selecting sentence context-appropriate external knowledge. In addition, too many multi-views could induce redundant information, and the generation of N multi-views requires certain computing resources. After balancing model performance and computation consumption, the number of generated multiviews is set to 3.
The embedding generation method KBGAT (Nathani et al., 2019) can generate embeddings for nodes and edges in a graph, which considers both the weights of nodes and edges. The contextspecific external feature Fk is generated as a node in graph Gm. The entire construction step of the external feature Fk is done independent of the training and test sets given.
Table 5 shows the example case about external information usage. Without external information, the model extracts “system” as the subject. By utilizing external information, the model can know that “DDB” is short for “Distributed Database” and identifies “DDB system” as the subject.
A.4 Ralated Work
The main distinctive work in our paper is to concatenate the constraints into the traditional tuples to enhance the validity of the extracted knowledge tuples. Technically, as constraints are usually written into the sentences with various latent forms, extracting them correctly from free texts with no
semantic conflicts is a challenging task. To this end, we propose the constrained tuple extraction task.
As for the Open IE, it mainly concerns about how to extract the predicates and the corresponding arguments. By contrast, in our work, we focus on how to extract the constrained tuples, which are formatted as (subject, relation, object, constraint). In other words, we emphasize the validity of knowledge tuples by introducing constrains, which provides a standardized and unified representation for knowledge tuples.
We adopt the two-stage extraction strategy similar to the Multi2OIE model. Our IAN model facilitates the combinatorial interactions among the context-specific external features and the distinctgranularity internal features to effectively mine the potential constraints in knowledge.
The difference between our IAN model and the Multi2OIE model is that the Multi2OIE model simply exploits BERT representations and predicate features to extract tuples, while our IAN model mines the combinatorial interactions among the external and internal features.
More specifically, Multi2OIE model feeds predicate feature, positional embedding, and BERT representation into multi-head attention blocks for Open IE tuple extraction. In our proposed IAN model, we design distribution-sparse multihead attention to select the dominating attentions, and feed distinct-granularity internal features and context-specific external features to distributionsparse multi-head attention to mine the combinatorial interactions for constrained tuple extraction. Concretely, distinct-granularity internal features include word-level, sentence-level, phrase-level textual features, BERT hidden, heterogeneous fused feature, predicate feature and position embedding.
The constrained tuple extraction task is actually a fundamental task for constructing the constrained knowledge graph construction with good quality. Technically, the phrases in sentences that act as subjects and objects in constrained tuples are usually with the type of entities. Thereby, in this situation, the form of the constrained tuples is “ (head entity, relation, tail entity, constraint) ”. Accordingly, these constraint tuples (i.e., quadruples) can be transformed into triples and their constraints. Hence, on the one hand, triples are the basic components of knowledge graphs. On the other hand, constrained tuples with different kinds of constrains
constitute a specific kind of constrained knowledge graph (for example, temporal knowledge graph) (Chen et al., 2022; Gracious et al., 2021).
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section "Limitations"
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? abstract,Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 1,2,3,4,Appendix
3 B1. Did you cite the creators of artifacts you used? Section 1,2,3,4
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Section 1,3
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 3,4
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Section 4, Appendix
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 4
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4
C 3 Did you run computational experiments? Left blank.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 4, Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4, Appendix
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 4, Appendix
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Appendix
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: In the future, we will address
the issues about the vertical domain-oriented constrained tuple extraction. Limitations
The open information extraction methods may amplify the bias of the corpus by extracting any relation occurring in the data. The models with deep learning may learn the relation bias from the training corpus and extract those biased statements. To mitigate the effect of data bias, we try to balance the relations in constrained tuples and the ratio of constraints when constructing the CTEB dataset. In addition, the utilization of external auxiliary information increases additional computation time. Our IAN model has still achieved superior performance when the external auxiliary information is removed