Open Information Extraction via Chunks
Authors: Kuicai Dong
Aixin Sun
Jung-jae Kim
Xiaoli Li
Venue: EMNLP 2023 Main (2023)
Keywords: Information Extraction
sentence chunking

Abstract: Open Information Extraction (OIE) aims to extract relational tuples from open-domain sentences. Existing OIE systems split a sentence into tokens and recognize token spans as tuple relations and arguments. We instead propose Sentence as Chunk sequence (SaC) and recognize chunk spans as tuple relations and arguments. We argue that SaC has better properties for OIE than sentence as token sequence, and evaluate four choices of chunks (i.e., CoNLL chunks, OIA simple phrases, noun phrases, and spans from SpanOIE).  Also, we propose a simple end-to-end BERT-based model, Chunk-OIE, for sentence chunking and tuple extraction on top of SaC. Chunk-OIE achieves state-of-the-art results on multiple OIE datasets, showing that SaC benefits the OIE task.

Open Information Extraction via Chunks
Kuicai Dong"?, Aixin Jung-Jae Kim", Xiaoli Li
1 School of Computer Science and Engineering, Nanyang Technological University, Singapore
kuicai001@e ntu. www.nnu..uuuuuued e axsunêntu edu.sg
2 Institute for Infocomm Research, A A*STAR, Singapore
A *STAR Centre for Frontier AI Research, Singapore {jjkim, xlli)ei2r.a-ee a-star. edu.sg
Abstract
Noun
Noun
Verb
Noun
Noun
Verb
Prep Noun
the headmaster,tool
ulyand Jimmy responsible for this
Open Information Extraction (OIE) aims to tract relational tuples from open-domain sen- tences. Existing OIE systems split a sentence into tokens and recognize token spans as tuple relations and arguments. We instead propose Sentence as Chunk sequence (SaC) and recog- nize chunk spans as tuple relations and argu- ments. We argue that SaC has better properties for OIE than sentence as token sequence, and evaluate four choices of chunks (i.e., CoNLL chunks, OIA simple phrases, noun phrases, and spans from SpanOIE). Also, we propose simple and-to-end SERT-based model, Chunk- OIE, for sentence chunking and tuple extraction top of SaC. Chunk-Ol achieves state-of- the-art results on multiple OIE datasets, show- ing that SaC benefits the OIE task. Our model will be publicly available in Github upon paper acceptance.
(a) OIA simple phrases
NP
Ms. Lee, the headmaster, told Lily and Jimmy she is responsible for this
(b) Noun phrases (NPs)
NP
NP
VP
NP
NP
VP
ADJP
PP
NP
the headmaster, Lily Jimmy she responsible this
(c) CoNLL-chunked phrases
Figure I: A sentence in different chunk sequences.
Recently, Sun al. (2020) and Wang al. (2022) propose use Open Information Annota- tion (OIA) as an intermediate layer between the input sentence and OIE tuples. OIA represents a sentence as graph where nodes are simple phrases, and edges connect predicate nodes and their argu- ment nodes. By employing dataset-specific rules, these OLA graphs can be transformed into OIE tu- plcs. Nevertheless, accurately generating the com- plete OIA graph for given sentence poses chal- lenge.
1 Introduction
Open Information Extraction (OIE) is to extract structured tuples from unstructured open-domain text (Yates et al.. 2007). The extracted tuples are in the form of (Subject, Relation, Object) in the case of binary relations, and (ARGo. Relation, ARG1, ...., ARGn) for n-ary relations. The structured rela- tional tuples are beneficial to many downstream tasks, such as question answering (Khot et al., 2017) and knowledge base population (Martínez- Rodríguez et al., 2018; Gashtcovski et al., 2020). When observing benchmark OIE datasets, most relations and their arguments are token spans. As a domain-independen information extraction task, OIE does not specify any pre-defined extraction schema. As result, determining the granularity length of these text spans becomes challenging. Consequently, many existing OIE systems adopt tagging- based methods, such as BIO or similar tagging schemes, to extract tuples at the token level. Beeiin, Inside, and Outside of a subjecttrelationnojjcct
Inspired by OIA, we propose a novel notion of Sentence as Chunk sequence (SaC), as an alter- native intermediate layer representation. Chunk- ing, form of shallow parsing, divides a sentence into syntactically related non-overlapping phrases, known as chunks (Tjong Kim Sang and Buchholz, 2000). For instance, the simple phrases in OIA can be considered as chunks (Figure la). To jus- tify the adaptability of SaC for OIE, we also em- ploy other chunking options including Noun Phrase chunks (Figure lb) and CoNLL chunks (Figure Ic). Figure shows an example sentence with differ- ent chunking schemes. Subsequently, propose Chunk-OlE, an end-to-end tagging-basse neural OIE model. Chunk-OIE performs multi-task learn- ing among two subtasks: (i) to represent sentence in SaC, and (ii) to extract tuples based on SaC. Our findings reveal that SaC-based OIE outperforms the

---

traditional OIE approach representing sentences as token sequences, particularly when the OIE tuple relations and arguments align well with the chunks, as is often the case.
ral OIE as well. RnnOlE (Stanovsky et al., 2018) and SenscOIE (Roy et al., 2019) encode POS/ / de- pendency as additional embedding features. MGD- GNN (Lyu et al., 2021) connects words, if they are in dependency relations, in an undirected graph and applies GAT as its graph encoder. RobustOIE (Qi et al., 2022) uses paraphrases (with various con- stituency form) for more syntactically robust OIE training. SMiLe-OIE (Dong al., 2022) incor- porates heterogeneous syntactic information (con- stituency and dependeney graphs) through GCN en- coders and multi-view learning. Inspired by them, we design a simple strategy to model dependency relation at the chunk level. Note that chunks in SaC partially reflect constituency structure as words in chunk are syntactically related, by definition.
Our contributions are as follows. Firstly, we pro- pose novel notion of Sentence as Chunk sequence (SaC) for OIE. On top of SaC, we further propose to simplify token-level dependency structure of sen- tence into chunk-level dependency structure in or- der to also encode chunk-level syntactic informa- tion for OIE. Secondly, we propose Chunk-OLE, an end-to-end learning model that (i) represents a sentence SaC, and (ii) extracts tuples based on the SaC. Finally, experimental results show the ef- fectiveness of Chunk- against strong baselines. Through data analysis against gold tuples, we show that chunks provide a suitable granularity of token spans for OIE.
Sentence Chunking. Our proposed notion of SaC is based the concept of chunking. Chunk- ing is to group tokens in sentence into syntac- tically related non-overlapping groups of words, i.e. chunks. Sentence chunking is a well studied pre-processing step for sentence parsing. We can naturally use the off-the-shelf annotations as exter- nal knowledge to enhance OIE. The carliest task of chunking was to recognize non-overlapping noun phrases (Ramshaw and Marcus, 1995) exem- plified in Figure 1b. Then CoNLL-200 shared task (Tjong Kim Sang and Buchholz, 2000) pro- posed to identify other types of chunks such as verb and prepositional phrases, see Figure 1c.
2 Related Work
OIE Systems. OIE was first proposed by Yates et al. (2007), and TextRunner is the first system that generates relational tuples in open domain. Many statistical and rule-based systems have been proposed, including ReVerb (Fader et al., 2011) 1),
OLLIE (Mausam et al., 2012), ClausIE (Corro and Gemulla, 2013), Stanford OIE (Angeli et al., 2015), OpenlE4 (Mausam, 2016), and MINIE (Gash- teovski et al., 2017). These models extract rela- tional tuples based on syntactic structures such as part-of-speech (POS) tags and dependency trees. Recently, two kinds of neural systems have been explored, generative and tagging-basee sys- tems (Zhou et al., 2022). Generative OIE sys- tems (Cui et al., 2018; Kolluru et al., 2020a; Dong et al., 2021) model tuple extraction as a sequence- to-sequence generation task with copying mech- anism. Tagging-based OIE systems (Stanovsky et al., 2018; Kolluru et al., 2020b; Kotnis et al., 2022) tag each token a sequence labeling task. SpanOIE (Zhan and Zhao, 2020) uses different approach. enumerates all possible spans (up to a predefined length) from a sentence. After rule- based filtering. the remaining candidate spans are classified to relation, argument, or not part of a tuple. However, enumerating and filtering all possi- ble spans for scoring is computationally expensive.
OIX and OIA. Sun et al. (2020) propose Open Information eXpression (OIX) to build OIE sys- tems. OIX is to represent a sentence in an interme- diate layer, so that reusable OIE strategies can be developed on OIX. As an implementation, they pro- pose Open Information Annotation (OIA), which is single-rooted directed-acycli graph (DAG) of sentence. Its basic information unit, i.e., graph node, is simple phrase. A simple phrase either a fixed expression or a phrase. Sun et al. (2020) define simple phrases to be: constant (e.g., nominal phrase), predicate (e.g., verbal phrase), and func- tional (e.g., wh-phrase). Edges in an OIA graph connect the predicate/funntion nodes to their argu- ments. Wang et al. (2022) extend OLA by defining more simple phrase types and release an updated version of the OlA dataset. The authors also pro- pose OIA @ OIE, including OIA generator to pro- duce OLA graphs of sentences, and rule-based OIE adaptors to extract tuples from OIA graphs.
Early neural models typically seldom utilize syn- tactic structure of sentence, which was required by traditional models. Recently works show that en- coding explicit syntactic information benefits neu-

---

3 Methodology: Chunk-O1E
Then, we chunk the sentence according the boundary predictions, i.e., the sentence is chunked to m pieces there are m boundary tokens. The token is marked to be boundary if argmax(pf) = 1. The type of each chunk is determined by the type of boundary token, which is argmax(p)). In overall, we convert the token sequence [41 (4,,...,4 into chunk sequence (cI.....Cml by SaC.
3.1 Task Formulation
We formulate the OIE tuple extraction process as a two-level sequence tagging task. The first level sequence tagging is to perform sentence chunking by identifying boundary and type of each chunk, and representing Sentence as Chunks (SaC). The second level sequence tagging to extract OIE tuples on top of SaC.
3.3 SaC-based OIE Extractor
Formally, given a sentence with input tokens = (f1....,tt) tnj, we first obtain the chunk se- quence S. = [ei,...,Com Cml (m S n) (Section 3.2). This process can be formulated as two sequence tag- ging sub-tasks: (i) binary classification for chunk boundary, and (ii) multi-class classification for chunk type (See example chunk boundaries and types in the outputs of "Boundary & Type Tag- ging" module in Figure 2). Note that tokens at boundaries are tagged as 1 and non-boundaries as O. Subsequently, we perform the tagging on the chunk sequence [c1,...,,Cm to extract OIE tu- ples (Section 3.3). A variable number of tuples are extracted from sentence. Each tuple can be represented as [21,...,22,, #L]), where each 2; is a con- tiguous span of chunks, either an exact match or chunk concatenation. One 2; is a tuple relation (REL) and the others are tuple arguments (ARG)). For instance, the tuple in Figure 2 can be repre- sented as (argo='Ms. Lee', rel='told', argı='Lily and Jimmy"). We address the two-level sequence tagging via multi-task learning (Section 3.4).
We design SaC-based OIE extractor on top of SaC. Given the typed chunks inferred by SaC (Sec- tion 3.2), we convert the BERT token represen- tations into chunk representations, and encode the chunk types. Subsequently, we model the chunk sequence into chunk-level dependency graph. Fi- nally, we use Graph Convolution Network (GCN) to get the chunk-level dependency graph represen- tations. The last tagging layer performs tagging at the chunk-level to extract OIE tuples, based on the concatenation of BERT-based and GCN-based chunk representations.
BERT-based Chunk Encoder. The chunk rep- resentations are based on the token represen- tations h in Equation verb in a sentence is a potential relation in- dicator, verb embedding is useful to highlight this candidate relation indicator (Dong et al., 2022). We follow Dong et al. (2022) to en- code tokens with additional verb embeddings, i.e. htoken h,; + Wverb caree__cadddate((t (
Also, as each
where rel__cre_ candidate(t.)) returns if is the candidate relation indicator of the instance (otherwise, 0), and W verb is a trainable verb embedding matrix.
3.2 Representing Sentence as Chunks (SaC)
We first use BERT to get the contextual representa- tions of input tokens [4,, (.,,... (n)] and then concate- nate them with the POS representations to obtain the hidden representations of tokens as follows:
For a single-token chunk (c: = [t,), its chunk representation hy" is the same as the token rep- resentation M For a chunk with multiple tokens (c; [tj....,tt]] the chunk represen- tation hy is the averaged token representations (avg((h) ,...,,...,...................... h h.okee)) Moreover, we encode chunk types with a trainable chunk type embedding Wehunk for additional type information:
h4 WBEET (4+) + WPos(pos. (ype(4))((44)) (1)
where WBER is trainable and initialized by BERT word embeddings, and WPos is trainable embedding matrix for POS types. The function pos_type(-) returns the POS type of input token. h4 is then passed into tagging layers for chunk boundary and type classification concurrently.
hy = ne + (chnnkkcchuu_
(4)
where the function chunk chunk_typel type(-) returns the type (e.g., Noun Phrase, Verbal Phrase) of input chunk.
p softmax(h: W Wound + b1)
(2)
A44 softmax(h; W type b2)
(3)
Chunk-level Dependency Graph. Recent stud- ies show that syntactic structures benefit neural models for NLP tasks including OIE (Fei et al.,
where P and p are the softmax probabilities for chunk boundary and type of token f;, respectively.

---

POS tags
Noun Ms.
*ago
Lee
Boundary
Lee
the
headmaster
POS
headmaster
Noun the
Dep-GCN Encoder
+
told
BERT Encoder
Embedding
Verb
Chunk Encoder
Chunk-level Dependencies
Tagging Layer
rel
the headmaster
Lily
Lily
and
Type Tagging
Noun
Sppes
+arg1
Jimmy
Jimmy
Ms. Lee
much
told
dabj
Lily and
she
Noun she
Jimmy
0+
coomp
is
Verb is
responsible
responsible
responsible
this
for this
Prep for
nv.bj
prop
pobj
-+
she
for
Noun this
Representing Sentence Chunks (Sac)
SaC-based OIE
Figure 2: The overview of Chunk-OIE. Punctuation marks in the sentence are neglected for conciseness. Chunk-OIE is an end-to-end model with (i) representing Sentence as Chunks (SaC) and (ii) SaC-based OIE tuple extraction.
neubj
Dependency Graph Encoder. Given the chunk- level dependency graph G = (C, E), we use GCN to encode the chunk-level dependency struc- We compute the node type embedding = deep (dep_ (epppyypp(ee) with trainable matrix Wdep IRd where di is the embedding dimension and Ndep is the number unique de- pendency relations. The function "dep_type(-) returns input chunk type. Subsequently, we use GCN to encode G with representations as follows:
compound
dobj
con
nsub
acomp
prop
pab
pppos
Ms. Lee, the headmaster, told Lily Jimmy she responsible this
(a) Dependency tree (results from spaCy).
nsubj
ccomp
dobj
prop
pobl
appos
Ms. Lea,the headmaster, told Lily and Jimmy she is responsible for this
Noun
Noun
Verb
Noun
Noun
Verb
Prep
Noun
(b) Dependency
chunk level with OIA-SP.
Figure 3: Dependency trees token-level and chunk- level (in OIA simple phrases), respectively. Note that we use spaCy to extract the dependency relations for sentences.
A RcLU (ng Wi 17 ba) (5)
where m refers the total number of chunk nodes in G. Wi Rd, is a trainable weight matrix for dependency type embeddings, and E Rd is the bias vector. The neighbour connecting strength distribution Ouj is calculated below:
2021: Dong et al., 2022). Thus, given the sen- tence represented in SaC, we model the depen- dency structure of input sentence at chunk level. For this purpose, we convert token-level depen- deney structure to that of a chunk level by ignoring intra-chunk dependencies and retaining inter-chunk dependencies. Figure shows the chunk-level de- pendency tree of the example sentence Figure (with OIA simple phrases as chunks) and its depen- dency tree at word level.
Gú exp((m.)" my) aij E exp (mp( my)
(6)
where m, = hy ① 4, and ( is concatenation oper- ator. In this way, node type and edge information are modelled in a unified way.
For OIE extraction, we aggregate chunk repre- sentations from the BERT-based representations in Equation 4 and from the GCN-based representa- tions in Equation 5. We then pass them into tagging layers for OIE span classification.
The chunk-level dependency graph is formulated G - (G-(C, (C, E), where the nodes in C' correspond to chunks .ci,.... Gm] and Gij in E equals to if there is dependency relation between token in node o and a token in node cy: otherwise, Each node 6; C has a node type. We label a node with the type of the dependency from the node to its parent node. Notice that SaC greatly simplifies the modelling of sentence syntactic structure.
p softmax ((h
dep h$)) WT WOEE b4) (7)
3.4 Multi-task Learning Objective
As mentioned, we perform the two-level sequence tagging of sentence chunking and OIE extraction.

---

We combine losses from SaC and OIE tagging to jointly optimize the Chunk-OIE model.
LSOIE is a large-scale OIE dataset converted from QA-SRL 2.0 in two domains, i.e., Wikipedia and Science. It 20 times larger than the next largest human-annotated OIE data, and thus is reli- able for fair evaluation. LSQIE. provides n-ary OIE tuples in the (ARGo, Relation, ARG,, .... , format. We use both datasets, namely LSOIE-wiki and LSOIE-sci, for comprehensive evaluation.
For SaC, considering that boundary and type classification are complementary to each other, we combine the following cross-entropy losses:
Lhound
Σ
of log(())) + (1 (-,)))log vl))og(11 - pp))
(8) CaRB dataset the largest crowdsourced OIE dataset. CaRB provides 1, 1,282 sentences with bi- nary tuples. The gold tuples are in the (Subject, Relation, Object) format.
Ltype
Mi log(r(
(9)
Lehunk = Lhhonnad + (1 a.) Leype
(10)
BenchlE dataset supports a comprehensive eval- uation of OIE systems for English, Chinese, and German. BenchlE provides binary OIE annota- tions and gold tuples are grouped according to fact synsets. In our experiment, we use the English corpus with 300 sentences and 1,350 fact synsets. Note that the multi-task training requires both chunking and OIE labels of ground-truth. However, chunking labels are nol present in OIE datasets. We construct chunk labels for the OIE datasets used in our experiment (in Section 4.2).
where y6 and y" are gold labels for chunk boundary and type, respectively. pd and o' are the softmax probabilities for chunk boundary and type tagging obtained from Equatios 2 and 3, respectively. C1 refers to the number of unique chunk types. a is hyperparameter balancing the two losses.
For OIE, the gold labels are provided at token level, whereas our predicted labels are at chunk level. To enable evaluation of the generated chunk- level tuples against the token-level gold labels, we assign the predicted probability of a multi-token chunk to all its member tokens. The correspond- ing cross-entropy loss is computed between the predicted and the gold OIE tags:
Evaluation Metric. For LSOIE-wiki and LSOIE-
sci datasets, we follow Dong et al. (2022) to act tuple matching. A predicted tuple is counted correct if its relation and all its arguments are iden- tical to those of a gold tuple; otherwise, incorrect. For the CaRB dataset, we use the scoring function provided by authors (Bhardwaj et al., 2019), which evaluates binary tuples with token level matching, i.e., partial tuple matching. The score of a pre- dicted tuple ranges from 0 to 1. For the BenchlE dataset, we also adopt the scoring function pro- posed by authors (Gashteovski et al., 2022), which evaluates binary tuples with fact-based matching. predicted tuple counted as correct if it exactly matches to one fact tuple, and otherwise incorrect.
LOIE
Dig log(p?)))
(11)
where yoi is the gold label, and po is the softmax probability obtained from Equation 7. C2 is the number of unique OIE span classes.
Finally, we combine losses from Equations 11 and 12, and minimize the following multi-task learning loss:
Lmmatti = Lelunnk + LOLE
(12)
4.2 Chunk Choices and Labels Construction
where 3 is hyperparameter balancing the chunk- ing and OIE losses. More training details are in Appendix A.2.
SaC is to represent a sentence in syntactically re- lated and non-overlapping chunks. However, there is no standard definition on what word groups should be chunked, though SaC can be achieved by any chunking scheme. We use four types of chunks to realize SaC. Also, we construct the chunking la- bels among OIE datasets through (i) our pre-trained chunking model (ii) existing parser.
4 Experiments
4.1 Chunk-OIE Setups
OIE Dataset. We conduct experiments on four datasets: the two LSOIE datasets (Solawetz and Larson, 2021), CaRB (Bhardwaj et al., 2019), and BenchIE (Gashteovski et al., 2022). More details for the OIE train/test sets are in Appendix A.6.
CONLL chunks. The CoNLLL-2000 (Tjong Kim Sang and Buchholz, 2000) chunking task de- fines 11 chunk types based on Treebank (Bies et al.,

---

LSOIE-wild
LSOIE-sc sci
CaRB
BenchlE
Models
F1
AUC
AUC
AUC
Pr
Re
Token-level OIR Systems
CopyAtteniin (Cui al., 2018)
39.5
35.9
48.8
46.81
$1.67
32.83
21.5
26.4
IMoJIE (Kolluru et al., 2020a)
49.2
47.5'
58.7
55.81
53.51
33.31
18.4
38.3
12.1
CIGL-OIE (Kolluru al., 2020h)
44.7
41.9
56,67
52.31
54.0#
35,7
25.4
31.1
21.4
BERT (Solawetz and Larson, 2021)
47.57
44.77
$7.07
53.21
51.41
30.67
23.1
32.5
BERT+Dep-GCN (Dong et al., 2022)
48.7
47.9*
58.17
55.31
52.51
32.91
25.1
35.3
19.5
SMiLe-OIF (Dong al., 2022)
51.7
50.8
60.5
57.21
53.81
34,91
25.7
37.5
19.6
Chunk-level OIE Systems
SpanOIR (Zhan and 7hao, 2020)
47.5
57.5
49.4
23.4
38.1
16.9
O OLA (Wang ol., 2022)
52.3'
32,6*
No (2-stage)
50.7
48.9
60.3
58.4
53.0
33.8
25.3
40.2
18.5
Nathon (end-to-end)
51.0
49.1
60.1
58.8
52.0
33.5
25.1
40.5
18.2
NPtngg (2-stage)
48.5
46.4
57.2
56.7
50,9
31.7
23.4
35.1
17.6
Chunk OIE
NPung (end-to-end)
49.4
48.2
58.0
57.3
51.5
32.1
24.3
36.8
18.2
with various
OIA-SP (2-stage)
52.1
50.4
61.2
60.1
53.6
35.5
26.7
41.5
19.7
choice
OLA-SP (end-to-end)
52.1
51.0
61.0
60.4
54.2
35.2
27.2
42.4
20.1
CoNLL (2-stage)
52.6
50.2
60.8
60.2
53.2
34.7
26.9
42.0
19.8
CoNLL (end-to-end)
52.8
50.5
61.5
59.7
53.5
34.0
26.9
42.7
19.6
Table I: Results four OIE datasets (best scores in boldface and second best underlined). Scores with special mark are from (Kolluru et al., 2020b) (Gashteovski et al., 2022) (Wang et al., 2022)", (Dong et al., 2022)
1995). We train our own CoNLL-style chunking model, as deseribed Appendix A.3.
Spans/Chunks
Number of Spans
Average Length
Gold Spans
76,176
4.40
OIA simple phrases (OIA-SP). The OIA sim- ple phrases has 6 types defined by Wang et al. (2022), We also train our own OLA-style chunking model, as described in Appendix A.3.
CoNLL
339,099
1.62
OLA-SP
307,505
1.77
NPshoo
335,939
1.53
NP
225,796
2.28
SpanOIE
1,995,281
4.34
NP chunks. In this scheme, the tokens of sentence tagged with binary phrasal types: NP and O, where O refers the tokens that are not part of any noun phrases. We notice that there often exists nested NP. Accordingly, we create two types NP chunks, iee,, short the phrase "Texas music player" is nested NP. NPlong will treat it as single NP, whereas NP short will split it to "Texas" and "music player" as two NPs. We use Stanford constituency parser to get NP chunks.
Table 2: Number and average length of gold tuple spans, proposed phrases for SaC, and SpanOIE spans.
short and Nolgng For example,
tion and produces a variable number extractions per sentence. It iteratively generates the next tu- ple, conditioned on all previously generated tuples. CIGL-OIE (Kolluru et al., 2020b) models OIE as a 2-D grid sequence tagging task and iteratively tags the input sentence until the number of extractions reaches pre-defined maximum. Another base- line, BERT (Solawetz and Larson, 2021), utilizes BERT and linear projection layer to extract tuples. SMiLe-OIE (Dong et al., 2022) explicitly models dependency and constituency graphs using multi- view learning for tuple extractions. BERT+Dep- GCN is baseline used in Dong et al. (2022), which encodes semantic and syntactic information using BERT and Dependency GCN encoder. It is the closest baseline our Chunk-OIE. The difference that Chunk-OIF encodes dependency structure at chunk level and the chunks partially reflect the sentence syntactic information.
SpanOIE spans. SpanOIE (Zhan and Zhao, 2020) enumerates all possible spans of a sentence, up to 10 words. To reduce the number of candidate spans, it keeps only the spans with certain syntactic dependency patterns.
The total numbers, and average lengths of the chunks of the four types and of the gold spans of the four datasets are listed in Table 2.
4.3 OIE systems for Comparison
Token-level OIE systems. CopyAttention (Cui al., 2018) is the first neural OIE model which casts tuple generation as a sequence generation task. IMOJIE (Kolluru al., 2020a) extends Copy Atten-

---

Chunk-level OIE systems. SpanOlË (Zhan and Zhao, 2020) enumerates all possible spans from a given sentence and filters out invalid spans based on syntactic rules. Each span is subsequently scored to be relation, argument, or not part of a tuple. OIE@OL OIA (Wang et al., 2022) is rulc-based sys- tem that utilizes OIA graph. As the nodes of OIA graph are simple phrases (i.e., chunks), we consider OIE OIA chunk-level OIE system.
LSkIE-wiki
LSOIE-sci
Chunk-OIF
Fi
AUC
AUC
OIA-SP
52.1
50.4
61.2
60.1
w/o Dep-GCN
51.3
50.2
59.0
57.8
wwo Chunk type
50.7
49.8
59.7
58.1
CONLL
52.6
50.2
60.8
60.2
w/o Dep-GCN
52.0
49.6
58.4
58.7
w/o Chunk type
50.4
49.1
59.8
58.5
Table 3: Ablation study of Chunk-OIE.
Chunk-OIE is our proposed model that is based on SaC for tuple extraction. To explore the effect of different chunks in SaC, we implement four vari- ants: Chunk-OIE shon), Chunk-OfE (NP1ong): Chunk-OrE (OIA-SP), and Chunk-OIE (CoNLL). Besides the end-to-end Chunk- -IE proposed in Section 3, we also experiment on variants that con- duct two-stage training, i.e., the SaC part is pre- trained with chunking dataset and frozen during the training of OIE tuple extraction (more details about 2- stage Chunk-OIE are in Appendix A.5).
Scenario
Example
Exact
emissions trading system the editor of the journal
Match
Concat,
Overlap
observed increase in trade wwater, dissolves miineals''
Mismatch
NoOverlap
Table 4: Four scenarios for matching a gold tuple span (in blue) generated chunk (in green).
and BenchIE datasets. In Chunk-OIE OIE, SaC treats chunks with coordination (e.g,, "Lily and Jimmy") as a single unit, resulting in poor scores in such cases, Except on CaRB, CIGL-OIE cannot gener- alize well to other datasets.
4.4 Main Results
Experimental results in Table show that Chunk- OIE, in particular its Sac-OIA-SP and SaC- CoNLL variants, achieve ofate-of-the-art results on three OIE datasets: LSOIE-wiki, LSOIE-sci. and BenchIE. Mcanwhile, their results on CaRB are comparable with baselines. We evaluate the statistical significance of Chunk-OfF against its token-level baseline based on their Fi 's (each ex- periment is repeated three times with different ran- dom seeds). The p-values for Chunk-OP OIE (OIA-SP)
Comparing to Chunk-level system: Chunk- OIE with OLA-SP and CoNLL chunking schemes outperform the other two variants with NPshon and NP for all the four datasets. This indi- cates that multi-label chunking is more effective for the chunk-level OIE than simply recognizing noun phrases in a sentence. And, Chunk-OIE with NP NPshort short outperforms Chunk-OIE with NP for all the four datasets, which may reflect the fact that OIE tuple arguments are often simple noun phrases rather than cascaded ever, Chunk-OlE with OIA-SP and CoNLL chunk- ing schemes show comparable performance.
and Chunk-OIE (CoNLL) are 0.0021 and 0.0027, indicating both results significant at p 0.01.
Comparing to token-level system: Chunk-O1r surpasses its token-level counterpart BERT+Dep- GCN on all the four datasets. Note that both Chunk-OfF and BERT+Dep-GCN rely BERT and Dependency GCN encoder; the only differ- ence is the input unit, i.e., chunks for Chunk-On and tokens for BERT+DEp-GGN Consequently, we suggest using chunks is more suitable to OIE. We observe SMiLe-OIE is a strong baseline. It explicitly models additional constituency informa- tion and the multi- view learning is computational complex. Comparing to it, Chunk-OIE is simple yet effective. CIGL-OIE performs good on CaRB dataset. adopts coordination boundary analysis to split tuples with coordination structure, which well aligns with the annotation guidelines of CaRB dataset, but not with the guidelines of the LSOIE
phrases. How-
Chunk-OIE achieves better results than SpanOIE, indicating that SaC more reasonable than the spans enumerated by SpanOIE. Note that OIE@ OIA generates tuples with rules manually crafted for OIE2006 and CaRB datasets. Also, the authors have not released source code of their rules. Therefore, OIE@ OLA cannot be evaluated LSOIE-wiki, LSOIE-sci, and BenchIE.
Chunk-OIE: 2-stage versus end-to-end: We notice that Chunk-OlE trained end-to-end achieves slightly better performance than Chunk-OIE with 2-stage training. This indicates that learning of sentence chunking can benefit OIE learning.

---

Chunk
CoNLL
OIA-SP
NP
SpanOIE
Match Case
Percent
Lp
Percent
Lp
Percent
Ly
Percent
Lp
Percent
Lp
Precision
Match
51.0%
1.8
49.7%
2.0
49.0%
1.7
40.5%
2.3
3.3%
3.5
Exact
8.4%
2.3
10.2%
2.5
7.2%
2.1
11.0%
3.4
3.3%
3.5
Concatenation
42.6 %
1.7
39.5%
1.9
41.8%
1.6
29.5%
1.9
Mismatch- Misnatch--ooOverlap
49.0%
1.4
50.3%
1.6
51.0%
1.4
59.5%
2.3
96.7%
4.4
Matching case
Percent
La
Percent
L,
Percent
L,
Percent
L,
Percent
L,
Match
90.5% %
4.4
89.9%
4.5
79.7%
4.3
58.7%
4.7
86.0%
3.3
Recall
Exact
45.7%
2.3
48.9%
2.5
37.1%
2.1
36.7%
3.4
86.0%
3.3
Concatenation
44.8 %
6.4
41.0%
6.8
42.6%
6.2
22.0%
7.1
Mismatch- Mismatch-Overlay Overlap
9.5%
4.3
10.1%
3.6
20.3%
4.4
41.3%
4.1
14.0%
12.7
Table 5: Precision and Recall Analysis. L, and Lp are length of gold spans and generated chunks, respectively. For each type match/mismatch case, the highest score is in boldface second highest score underlined.
Candidate chunks
Precision
Recall
Ai
gold spans? (2) Recall: How often do the bound- aries of gold spans match those of SaC chunks? There four scenarios of boundary alignment, as exemplified in Table 4. Match-Exact: A gold span is exactly matched to a chunk span. Match- A gold span is mapped to multiple chunks in a consecutive sequence. Mismatch- Overlap: A chunk overlaps with a gold span, and at least one token of the chunk is not in the gold span. Mismatch-NoOverlle overlap with any gold span.
CoNLL
51.0
90.5
65.2
OIA-SP
49.7
89.9
64.0
NP sP
49.0
79.7
60.7
NP
40.5
58.7
47.9
SpanOIE
3.3
86.0
6.4
Concatenation:
Table 6: Precision, Recall, and Fi of generated chunks; best scores are in boldface, second best underlined.
A chunk does not
4.5 Ablation Study
We ablate each part of Chunk-OIE (OIA-SP, CoNLL), and evaluate the ablated models on LSKiE-wiki and LSOIE-sci. The results are re- ported in Table 3. We first remove the dependency graph encoder. In this setting, chunking representa- tion obtained in Equation 4 is directly used for tuple extraction. Results show that removing chunk level dependencies decreases the performance of Chunk- OIE, indicating the importance of chunk-level de- pendency relations. To explore the importance of chunk type, we ablate the chunk type embedding as described in Equation 4. Observe that this also leads to performance degradation.
We show the precision and recall analysis of four boundary alignment scenarios in Table and summarize the overall scores in Table 6. Observe that CoNLL chunks and OIA simple phrases show higher precision and recall of the Match bound- ary alignment than the other chunks. We note that the boundary alignment of CoNLL chunks LSOIE is better than that of OIA simple phrases to LSOIE, but the two Chunk Chunk-OIE variants with CoNLL chunks and with OLA simple phrases show comparable performance. This may indicate that the precision and recall analysis of boundary align- ment is 'generally good' but not 'precise' indicator for Chunk-OIE performance. We also note that SpanOIE has only 3.3% of precision, indicating that enumerating all possible spans should bear heavy burden detect correct spans.
4.6 Boundary Analysis on SaC
It critical understand the suitability of adopting chunks as the granularity for OIE. In this section, we perform boundary alignment analysis of SaC against gold spans a benchmark OIE dataset named LSOIE. Gold Spans are the token spans of tuple arguments relations in ground truth anno- tations. We analyze CoNLL chunks, OIA simple phrases, NP chunks, and SpanOIE spans as de- scribed in Section 4.2.
Conclusion
We propose Sentence as Chunk sequence (SaC) an intermediate layer for OIE tuple extraction. We then propose Chunk-OIE, by leveraging SaC and chunk-level dependencies, achieves state-of-the-art results on several OIE datasets. We experiment on
The boundary alignment analysis conducted from two perspectives. (1) Precision: How often do the boundaries of SaC chunks match those of
"This is not applicable to SpanOIF, since it emulates all possible spans; if there is match, it should be an exact match.

---

Chunk-OIE with various chunk choices as SaC, and perform detailed statistical study to understand to what extent these chunks align with OIE gold tuple spans, and how the boundary alignment impacts the overall OIE performance.. We indicate that CoNLL and OIA-SP chunks have better boundary alignment with OIE gold tuple spans than noun phrases, and Chunk-OIE adopting them as SaC achieves best results. In our future work, aim build upon SaC develop even more effective OIE models.
References
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence label- ing. In Proceedings of the 27th International Con- ference on Computational Linguisties, pages 1638- 1649, Santa Fe, New Mexico, USA. Association for Computational Linguisties.
Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D. Manning. 2015. Leveraging linguis- tic structure for open domain information extraction. In Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volame Long Papers), pages 344-354, Beijing, China. Association for Computational Lin- guistics.
Acknowledgments
This research is supported by the Ageney for Sci- ence, Technology and Research (A*STAR) under its AME Programmatic Funding Scheme (Project #A18A2b0046 and #A #A192200098).
Sangnie Bhardwaj, Samarth Aggarwal, and Mausam Mausam. 2019. CaRB: A crowdsourced benchmark for open IE. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP, pages 6262-6267, Hong Kong, China, Association for Computational Linguistics.
Limitations
The limitations of Chunk-OTE are analyzed from three perspectíves: SaC chunking errors, syntac- tie parsing errors, and multiple extractions issue. (1) Both CoNLL-chunked phrases and OIA simple phrases suffer around 10% boundary violations as shown in Table 5 (under Recall analysis). Since we use SaC as intermediate layer for OIE and per- form tagging at chunk level, the chunk boundaries become hard constraint of the extracted tuples. Among these violations, we examine 100 examples of OIA simple phrases and find that 55% of these vi- olations are caused by chunking errors due some complicated sentence structures. The rest is mainly caused by tuple annotation errors, meaning that all OIE systems will suffer from these annotation errors. (2) Chunk-OIE relies on the chunk-level dependency relations as additional syntactic knowl- edge. Therefore, Chunk-OTE will inevitably suffer from the noises introduced by the off-the-shelf de- pendency parsing tools. Also, we use POS tagger to extract all verbs in the sentence as tuple rela- tion indicators. It is reported that the POS tagger fails to extract 8% of verbs that are suppose to be relation indicators (Dong et al., 2022). Therefore, the discrepancy between POS verbs and tuple re- lations may affect the OIE quality. (3) Moreover, there are 6% of relation indicators corresponding multiple tulle extractions (one verb leading to more than one tuple), while our system extracts up to one tuple per relation indicator.
Ann Bies, Mark Ferguson, Karen Katz, Robert Mac- Intyre, Victoria Tredinnick, Grace Kim, Mary Ann Marcinkiewicz, and Britta Schasberger. 1995. Brack- eting guidelines for treebank ii style penn treebank project.
Luoxin Chen, Xinyue Liu, Weitong Ruan, and Jianhua Lu. 2020. Enhance robustness of sequence labelling with masked adversarial training. In Findings of the Association for Computationll Linguistics: EMNLP 2020, pages 297-302, Online. Association for Com- putational Linguistics.
Luciano Del Corro and Rainer Gemulla. 2013. Clausie: clause-based open information extraction. In 22nd International World Wide Web Conference, WWW '13, Rio de Janeiro, Brazil, May 13-17, 2013, pages 355-366 International World Wide Web Conferences Steering Committee/ / ACM.
Lei Cui, Furu Wei, and Ming Zhou, 2018. Neural open information extraction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), pages 407-413, Melbourne, Australia. Association for Computational Linguistics.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.

---

Kuicai Dong, Aixin Sun, Jung-Jae Kim, and Xiaoli Li. 2022. Syntactic multi-view learning for open infor- mation extraction. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, Online. Association for Computational Linguistics.
Iterative memory- based joint open information ex- traction. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 587 1-5886, Online, Association for Computa- tional Linguistics.
Keshav Kolluru, Adlakha Vaibhav, Samarth Aggarwal, Mausam, and Soumen Chakrabarti. 2020b. Open1E6: Iterative Grid Labeling and Coordination Analysis for Open Information Extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3748-3761, Online. Association for Computational Linguistics.
Kuicai Dong, Zhao Yilin, Aixin Sun, Jung-Jae Kim, and Xiaoli Li. 2021. DocOlE: A document-leve context-aware dataset for OpenlE. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 2377 2377-2389, Online. Association for Computational L Linguistics.
Bhushan Kotnis, Kiril Gashteovski, Daniel Rubio, Am- mar Shaker, Vanesa Rodriguez-Tembraa Makoto Takamoto, Mathias Niepert, and Carolin Lawrence. 2022. MILIE: Modular & iterative multilingual open information extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 6939- 6950, Dublin, Ireland. Association for Computational Linguistics.
Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011. Identifying relations for open information ex- traction. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1535-1545, Edinburgh, Scotland, UK. Associ- ation for Computational Linguisties.
Hao Fei, Shengqiong Wu, Yafeng Ren, Fei Li, and Donghong Ji. 2021. Better combine them together! integrating syntactic constituency and dependency representations for semantic role labeling. In Find- ings of the A Association for Computational Linguisties: ACI-IJCNLP 2021, pages 549-559, Online. Associa- tion for Computational Linguistics.
Zhiheng Lyu, Kaijie Shi. Xin Li, Lei Hou, Juanzi Li. and Binheng Song. 2021. Multi-grained dependency graph neural network for chinese open information extraction. In Advances in Knowledge Discovery and Data Mining, pages 155-167, Cham. Springer International Publishing.
Kiril Gashteovski, Rainer Gemulla, and Luciano del Corro. 2017. MinlE: Minimizing facts in open in- formation extraction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Lan- guage Processing, pages 2630-2640, Copenhagen, Denmark. Association for Computational Linguis-
José- Lázaro Martinez Rodriguez, Ivan López-Arévalo, and Ana B. Ríos-Alvarado. 2018. Openie-based ap- proach for knowledge graph construction from text. Expert Syst. Appl.. 113:339 -35.
tics.
Mausam. 2016. Open information extraction systems and downstream applications. In Proceedings of the Twenty- FFifihh International Joint Conference on Artifi- cial Intelligence, IJCAI 2016, New York, NY, USA, 9- 15 July 2016. pages 4074-4077 DCCA/AAAI Press.
Kiril Gashteovski, Rainer Gemulla, Bhushan Kotnis, Sven Hertling, and Christian Meilicke. 2020. On aligning OpenlE, extractions with knowledge bases: A case study. In Proceedings of the First Work- shop on Evaluation and Comparison of NLP Systems, pages 143-154, Online. Association for Computa- tional Linguistics.
Mausam, Michael Schmitz. Stephen Soderland, Robert Bart, and Oren Etzioni. 2012. Open language learn- ing for information extraction. In Proceedings of the 2012 Joint Conference on Empirical Methods in Nat- ural Language Processing and Computational Natu- ral Language Learning, pages 523-534, Jeju Island, Korea, Association for Computational Linguistics.
Kiril Gashteovski, Mingying Yu, Bhushan Kotnis, Car- alin Lawrence, Mathias Niepert, and Goran Glavaš. 2022. BenchlE: A framework for based open information extraction evaluation. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 1:
fact-
Qi. Yuxiang Chen, Lei Hou, Juanzi Li, and Bin Xu. 2022. Syntactically robust training on partially- observed data for open information extraction. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguisties.
Long Papers). pages 1472-4490 Dublin, Ireland. As- sociation for Computational Linguistics.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2017. Answering complex questions using open informa- tion extraction. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 311-316. Vancouver, Canada. Association for Computational Linguistics.
Lance Ramshaw and Mitch Mareus. 1995. Text chunk- ing using transformation-basse learning. In Third Workshop on Very Large Corpora.
Arpita Roy, Youngja Park, Taesung Lee, and Shimei Pan. 2019. Supervising unsupervised open information ex- traction models. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference
Keshav Kolluru, Samarth Aggarwal, Vipul Rathore, Mausam, and Soumen Chakrabarti, 2020a. IMoJIE:

---

on Natural Language Processing (EMNLP-IJCNLP) pages 728-737 Hong Kong, China. Association for Computational Linguistics.
tion on the web. In Proceedings of Human Lan- guage Technologies: The Annual Conference of the North American Chapter of the Association for Com- putational Linguistics (NAACL-HLT), pages 25-26, Rochester, New York, USA. Association for Compu- tational Linguistics.
Jacob Solawetz and Stefan Larson. 2021. LSOIE: A large-scale dataset for supervised open information extraction. In Proceedings of the 16th Conference of the European Chapter of the Association for Compu- tational Linguistics: Main Volume, pages 2595-2600, Online. Association for Computational Linguistics.
Junlang Zhan and Hai Zhao. 2020. Span model for open information extraction accurate corpus. In The Uhirty-Fourthy AAAI Conference on Artificial Intelli- gence, AAAI 2020, The Thirty-Second Innovative Ap- plications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 9523 9530. AAAI Press.
Gabriel Stanovsky, Julian Michael, Luke 7ettlemoyer, and Ido Dagan. 2018. Supervised open information extraction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long Papers), pages 185-895, New Orleans, Louisiana. Association for Computa- tional Linguistics.
Shaowen Zhou, Bowen Yu, Aixin Sun, Cheng Long, Jingyang Li, and Jian Sun. 2022. A survey on neural open information extraction: Current status and fu- ture directions. In Proceedings of the Thirty-Fits First International Joint Conference on Artificial Intel- ligence, IJCA1-22, pages 5694-5701. International Joint Conferences on Artificial Intelligence Organi- zation. Survey Track.
Mingming Sun, Wenyue Hua, Zoey Liu, Xin Wang, Kangjie Zheng, and Ping Li. 2020. A Predicate- Function-Argumment Annotation of Natural Language for Open-Domain Information eXpression. In Pro- ceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pages 2140-2150, Online. Association for Computa- tional Linguistics.
A Appendix
A.1 Implementation Details and Resources
Erik F. Tjong Kim Sang and Sabine Buchholz. 2000. In- troduction the CoNLL-2000 shared task chunking. In Fourth Conference on Computational Natural Lan- guage Learning and the Second Learning Language Logic Workshop.
We build and run our system with Pytorch 1.9.0 and AllenNLP 0.9.0 framework. The experiments are conducted with RTX 24GB 3090 GPU and Intel Xcon @ W-2245 3.90GHz CPU. Each epoch takes roughly 20 minutes for training on a single RTX 24GB 3090 GPU. We run each experiment with three random seeds and report the averaged results. We use NLP toolkit spaCy to extract the POS tags and dependency relations for sentences. In addi- tion, we obtain constituency parsing results through Stanford CoreNLP4 and use the noun phrases create NP-chunked phrases as part of our phrase selection exploration. The hidden dimension d, for BERT representation hebert chunked phrase repre- sentation h², and Dep-GCN graph representation dep are all set to 768. The hidden dimension di for Dep-Encoder type embedding ب! is 400.
Xin Wang. Minlong Peng, Mingming Sun, and Ping Li. 2022. OIE@OIA OIA: an adaptable and efficient open information extraction framework. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 6213-6226, Dublin, Ireland. Association for Computational Linguistics.
Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. 2021. Automated concatenation of embeddings for struc- tured prediction. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers). pages 2643-2660., Online. Association for Computational Linguistics.
The datasets, and their corresponding scoring scripts if applicable. used in this study are listed in Table 7. The table also list the source code URLs of the bascline models.
Michihiro Yasunaga, Jungo Kasai, and Dragomir Radev. 2018. Robust multilingual pfrt-of- pfrt-of-speech tagging via adversarial training. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Popers), pages 976-986, New Orleans, Louisiana. Association for Computational Linguistics.
A.2 Multi-Tass Mast Training
The multi-task training is elaborated in Algo- rithm 1. Inputs are the sentence with tokens s; [t...., tn), its chunk boundary labels i.!....,
Alexander Yates, Michele Banko, Matthew Broadhead, Michael Cafarella, Oren Etzioni, and Stephen Soder- land, 2007. TextRunner: Open information extrac-
"nttps://stanfordnl..gii ttttuub..o/CoreNL.)

---

Dataset
Resource URL
CoNLL-2000 (Tjong Kim Sang and Buchholz, 2000)
pttps://www.cc
www.werrpen. https//weerrrree.////////ooo//ee
OIA dataset v1.1 (Wang al., 2022)
LSOIE dataset (Solawetz and Larson, 2021)
nttps://gi httpb/..om//ccccsoooettt//////aa
scale-oie
CaRB dataset and scoring code (Bhardwaj al.,
BenchlE and scoring code (Gashteovski al., 2022)
https://gi thub. www.b.coom/kkirr//ennnhii
Model
Source code URL
(bssuuue (Devlin et al., 2019)
https://huggingfacee htt.r///w.....eeeeeebb-----aaaae
Copy (Cui et al,, 2018)
https://gi thu.. com//dirr-iitt//immjj
IMoJIE (Kolluru al., 2020a)
https://gt wttp..comdaarriit//mmmoiji.
SpanOIE (Zhan and Zhao, 2020)
https: //gj thub.htub... http://///haanjinloo////smm..o
CIGL-OIE (Kolluru et al., 2020b)
https://ga thub. htt..///aair-iitt///peeeeeee
SMiLe-OIE (Dong et al., 2022)
https://gi ttt.ggghhhhu..... http..ccc//avooooogg/c////taaa/
Table 7: Online resources for datasets and models.
chunk type labels .yf...., ..). and OIE span labels lyfie, ,..., (. We target to train two sub-modules Nsc (SaC described in Section 3.2) and NoTE (OIE extractor described in Section 3.3) via multi- task learning.
Algorithm Multi Masti-Task Training
Require:
Input sentence s, = [t,,,.., ......... tn].
Chunk boundary labels 3² = lv Mhll
Chunk type labels y' = Ivi Vnl
We first feed input tokens into NsoC and ob- tained token-level contextualized representations hg. and the softmax probabilities p², p² for chunk boundaries and chunk types, respectively. We then chunk the sentence to generate SaC as 8. = (c....., m. Subsequently, we pass the token-level representations hę and the SaC sequence &, into NOIE. We can obtain the softmax probabilities nd for OIE span predictions. For chunking, we compare the predicted chunk boundaries and types (obtained from Nsac) with the true labels. For OIE extraction, compare the predicted OIE spans (obtained from NOIE) with the true spans. The cross-entropy losses for chunking and OIE extrac- tion are combined. Finally, we calculate the gradi- ent of the combined loss with respect to the weights of both Nsac and NoIE. These gradients will be used for updating the model parameters during the training process.
OIE span labels you = [V 10io]
Ensure:
1: Model parameter Nac and NOIE; Initialize Nsacc and NOIE:
repeat
hū pb, Nsoc(s);
Se [cl,..., cml t chunking(p', ,p'
paic Nosn(hi 8c);
Lehank = CE(p', y²) CEE(', v²);
8:
LOSE = CE , ;00i));
9:
Lmultts = Lohunk + LOIE:
10: Calculate the gradient;
11: Update model parameter Nsoc and NOIE;
12: until Stop criterion reached;
13: return Nsac and NOIE
Ms.
Ms.
POS tags
Noun
Lee
Boundary & Type Tagging
Lee
A.3 Pre-trained Chunking Model
the
Noun the
headmaster
POS
headmaster
Model Details. The sentence chunking is dis- played in Figure 4. identical to the SaC part of Chunk-OIE, described in Section 4.6. However, the training and optimization of chunking model purely relies on chunking loss in Equation 12.
told
BERT Encoder
Embedding
Verb told
Lily
Lily
and
Noun and 0
Jimmy
Jimmy
she
Noun she
Verb is
Sentence Chunking Datasets. CoNLL-2000
responsible
responsible
for this
Prep for
Shared Task dataset by Tjong Kim Sang and Buch- holz (2000) annotates 8,936 / 2.012 sentences for Train/Test sets, respectively. Open Information An- notation (OIA) v1.1 dataset by Wang et al. (2022) contains 12,543 / 2,002 / 2,077 examples for Train
Noun this
Figure 4: Sentence Chunking model. It is used to pro- vide chunking labels for CoNLL and OIA-SP.

---

Chunking model on CoNLL2000
Chunk type Fi
A.4 Chunk Boundary and Type Analysis of Pre-trained Chunking Model
AT (Yasunaga et al., 2018)
95.3
Flair (Akbik et al., 2018)
96.7
Table 10 reports the chunk boundary accuracy of our SaC model (Section 3.2) by chunk length in number of tokens. Observe that the Fi of chunk boundary decreases when chunks become longer on both datasets. As expected, the longer the chunks, the harder the boundary detection becomes. Nevertheless, the Fi of long chunk (e.g., more than 5 tokens) is 94% and 80.44% CoNLL-2000 and OIA datasets, respectively. This shows that our chunking model performs reasonably well in detecting long chunks. On the other hand, short chunks (e.g,, with or 2 tokens) dominate the num- bers, leading to high overall accuracy. We observe that the annotated sentences in CoNLL-2000 are longer and more formally written than that in OIA dataset. This could be reason contributing to the higher Fi on the CoNLL-2000 dataset.
MAT (Chen et al., 2020)
97.0
ACE (Wang et al., 2021)
97.3
Ours (BERT-Muultitusk)
97.0
Table 8: Chunk type Fi on CONLL 2000 chunking dataset. Detailed results in Appendix A.4.
Chunking model on OIA dataset
Boundary Fi
Type Fi
Rule-hased (Sun et al., 2020) Neural model (Wang al,, 2022)
82.4
88.5
85.31
Ours (BERT-Multttask)
90.9
87.1
Table Performance of chunking on OIA dataset. Note that Wang et al. (2022) report chunk boundary result only and state that 96.4% of them are labelled with correct types. We hence estimate their chunk type Fi (marked with f) based on the given percentage.
Recall that chunk type classification condi- tioned on the boundary provided, i.e., type is mean- ingful only if boundary is correctly detected. If the ground truth chunk boundaries are known, the overall type classification F1 is 99.2% and 95.8% CoNLL-2000 and OIA datasets. However, in reality, the chunk boundaries have be detected as well.
Development / Test sets. Each OIA annotation is a eereecee-rrahh pair. We only utilize the graph nodes (i.e., simple phrases) for the chunking task.
SaC Evaluation Metric. We report Precision Recall Fi for both chunk boundary detection and chunk type classification. For chunk bound- ary detection, we consider exact boundary match between a predicted chunk and a gold chunk as correct. For chunk type classification, the chunk is counted correct if both the boundary and type are exactly matched. That is, chunk type is meaningful only if its boundary is detected correctly.
respectively,
Tables la and lb report the F1 of chunk type classification by the major chunk types in both datasets. In this set of experiments, the chunk boundaries are detected together type classifica- tion (i.e., the same setting as in Section 3.2). In both datasets, noun, verbal, and prepositional phrases dominate the chunks. The F1 scores are reasonably high on these major types. Again, as the sentences in CoNLL-2000 datasets are much longer, the num- ber of chunks in CoNLL-2000 is much larger than that in OIA dataset, although the two datasets have comparable number of test sentences.
SaC (Chunking) Results. Reported in Table 8, our SaC model is comparable to the statc-of-the the-art on the CoNLL-2000 dataset. Note that the best model ACE (Wang et al., 2021) leverages multiple PLMs including GloVe, fastText, ELMo, BERT, XLM-R, and XLNet, which requires significant computational resources and is slow in inference. By contrast, our model is much simpler and only utilizes BERT. On OIA dataset (see Table 9), our SaC model outperforms all the previous methods. The detailed results of chunk boundary detection and type classification, by chunk length and types, are summarized in Appendix A.4.
A.5 Two stages Chunk-OIE model
Instead of training an end-to-end Chunk-OIE. model, we also experiment on a pipeline method that consists of two-stage training, corresponding to two sub-models as shown in Figure 5. The first stage is to pre-train a SaC chunking model with chunking datasets as described in Appendix A.3. We then obtain the chunking labels for sentence in OIE datasets through the SaC sub-model. The second stage is to train the OIE extractor, during which the chunking labels are given as inputs to the
In overall, our proposed SaC (chunking) model achieves SOTA or comparable results in all datasets. We believe simple BERRT-based SaC sufficient to support our study on SaC- BaC-based OIE extraction, as chunking is not the key focus of our study.

---

Dataset
CoNLL-2000
OIA
Lebunk
#Chunk
Pr
Re
#Chunk
Pr
Fi
token
19,414
98.5
98.1
98.3
11,201
92.8
92.8
92.8
tokens
6,267
97.3
97.4
97.3
2,924
88.0
89.5
88.7
tokens
2,865
96.8
97.3
97.1
1,245
84.5
85.8
85.1
tokens
945
94.1
96.4
95.2
440
78.3
78.0
78.1
5+ tokens
541
98.7
90.0
94.1
421
95.7
69.4
80.4
Overall
30,032
97.8
97.7
97.8
16,231
90.4
91.4
90.9
Table 10: Chunk boundary extraction accuracy by chunk length.
Typechmk
#Chunk
Pr
Re
Fi
TypCormk
#Chunk
Pr
Re
Pi
NP
12,422
97.5
97.3
97
Noun
7.159
85.8
VP
4,658
96.7
96.8
96.7
Verbal
3,673
83.2
86.3
PP
4,811
98.4
98.9
98.7
Prepositional
91.7
92.5
92.1
ADVP
866
88.0
96.0
87.0
Logical
75.2
86.9
80.7
SBAR
535
94.1
95.9
95.0
Modifier
336
75.9
75.0
75.5
ADJP
438
84.8
93.1
94.0
Function
60
37.8
70.0
49.1
PRT
106
77.9
89.6
83.3
O
2,675
96.5
88.3
92.3
6,180
97.7
97.0
97.4
Total
16,231
86.6
87.5
87.1
Total
30,032
97.1
97.0
97.0
(b) OIA dataset
(a)
CONI
Table 11: Accuracy of chunk type classification by chunk type. Note that, for CoNLL-2000 datasets, CONJP, INTJ,
LST and UCP each has fewer than chunks, hence are excluded from the results.
OIE sub-model. The OIE sub-model is to train with the OIE datasets (in Section 4.1) and loss function (in Equation 11).
Dataset
Source
#Sent
#Tuple
LSOIE-wiki-train LSOIE-wikk-test
QA-SRL 2.0
19,591
45,890
QA-SRL 2.0
4,660
10,604
LSOIE-sci-train
QA-SRL 2.0
38,826
80.271
A.6 Details of OIE Datasets
LSOIE--ccittstt
QA-SRI. 2.0
9,093
17,031
In this section, elaborate more details about the train/test set of OIE datasets as mentioned in Section 4.1. For LSOIE, we follow Solawetz and Larson (2021) and Dong et (2022) to split the train/test set in LSOIE-wiki and LSOIE-sci domain, respectively. The statistics of LSOIE train/test sets are listed in Table 12.
CaRB-train CaRB-test
OpenIE Crowdsourcing
92,774
190,661
1,282
5,263
BenchIE-traii BenchIE-test
OpenIE 4 Expert
92,774
190,661
300
1.350
Table 12: Statistics of OIE datasets used in training and evaluating Chunk-OIE.
CaRB only provides 1,282 annotated sentences and BenchlE provides 300 sentences, which are insufficient training neural OpenlE models. As a result, we use the CaRB and BenchIE dataset purely for testing. We follow Kolluru et al. (2020b) to convert bootstrapped OpenlE4 tuples as labels for distant supervised model training. The statistics of CaRB and BenchlE train/test sets listed in Table 12.
the phrase "Ms. Lee" that is appositional to the phrase "the headmaster". If we want model the relation between "Ms." and "the" through token dependencies, we need pass through three hops ('compound" - 'appos" + 'det') in order link them up. In contrast, connecting "Ms." and "the" via chunk-level dependencies only requires a sin- gle hop ('appos'). In another case, "Lee" is the nominal subject ("nsubj') and "Lily" is the direct object ('dobj") of verb "told". Apparently, need additional dependency relations to locate the com- plete subject and object of "told". If model dependencies chunk level, the complete subject and object of "told" can be easily located to be "Ms. Lee" and "Lily and Jimmy"" respectively.
A.7 Chunk-level Dependency Modelling
We argue that SaC simplifies the modeling of sen- tence syntactical structure. We elaborate this point with the example sentence shown in Figure 3a. In this sentence, "Lee" is the appositional modifier ('appos') of "headmaster". However, it is actually

---

argo
Ms.
Noun Ms.
tags
Lee
Lee
the
Boundary & Type Tagging
Noun the
Dep- GCN Encoder
headmaster
POS
headmaster
Chunk Encoder
Chunk-level Dependencies
Tagging Layer
ree
told
BERT Encoder
Embedding
Verb told
the headmaster
Lily
Lily
Appas
>rgg
and
Noun and
nzub)
dobj
Jimmy
Jimmy
Lee
Lily and
Jimmy
she
Noun she
-0
0
Verb
responsible
this
responsible
responsible
prep
reubj
for this
Prep
for
for
Noun
this
(b) Stage Training OIE extractor extracting OIE tuples based chunks
(a) Stage I: Training SaC and extracting chunk lahels
Figure 5: The overview of Chunk-OIE with two sub-models. Punctuation marks in the sentence are neglected for conciseness. Note that stage (sentence chunking) and stage (OIE extraction) are trained separately.
The conversion to chunk-level dependency rela- tions from token-level is performed in two steps. (I) We remove a dependency relation between two tokens if both tokens are within the same chunk. The following dependency relations in Figure 3a are removed: "compound" relation between "Ms." and "Lee", "der" relation between "the" and "head- master", "cc" relation between "Lily" and "and", "conj" relation between "Lily" and "Jimmyy", and "acomp" relation between "is" and "responsible". (2) We map the remaining dependency relations. that between tokens from different chunks, to be the relations between chunks. For example, the "ap- pos" relation between "Lee" and "headmaster" is map to "Ms. Lee" and "the headmaster" as shown Figure 3b. Similarly, "Ms. Lee" turns into the nominal subject (nsubj) and "Lily and Jimmy" be- comes the direct object (dobj) of verb "told".


Reviews:
[{"review_id": "OoPqTe3KJf", "reviewer": "Reviewer_s459", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. The focus of our work is to introduce sentence chunking as additional knowledge to guide OIE. Like most of previous OIE works, we conduct our research using English datasets. We leave it for future work to explore SaC for OIE in multi-lingual setting.\n2. We assume that UIE can be adapted for OIE, by writing OIE-specific SSI. In the relation work section, we focused on previous works of OIE, not generic IE works like UIE. We will mention it as a future work to adapt generic/univeral IE works like UIE for OIE and compare them with OIE-specific works like Chunk-OIE.\n3. Indeed, SaC is task-independent and can be adapted for traditional IE tasks like NER and RE. We will include some of such adaptation experiments in the discussion section.\n4. In the related work section, we briefly discussed differences between generative OIE systems like GPT and tagging-based OIE systems like Chunk-OIE. In this paper, we do not argue which of the two approaches is better than the other. We simply compared Chunk-OIE with other tagging-based OIE systems because they show state-of-the-art performance at the moment and because SaC can be straightforwardly applied only for tagging-based OIE systems.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposes a new approach for Open Information Extraction (OIE) called Sentence as Chunk sequence (SaC), which is argued to have better properties for OIE than the traditional approach of using a sentence as a token sequence. The proposed approach achieves state-of-the-art performance on OIE datasets.", "reasons_to_reject": "1. Limited experimentation with non-English languages: The evaluation of the proposed approach is limited to English datasets, and it is unclear how well the approach would perform in non-English languages or low-resource settings.\n2. Limited discussion of related work: An important and similar related work UIE [1] is not discussed. I wonder about the differences between UIE and OIE\n3. The experiments are limited to open-domain IE tasks, I wonder whether the approach works on traditional NER and RE tasks (for example, CoNLL 03 NER).\n4. GPT-based models are proved to be strong on open-domain and 0-shot tasks, but this paper does not compare OIE with these models.\n\nReference:\n[1] Unified Structure Generation for Universal Information Extraction\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a new approach for Open Information Extraction (OIE) called Sentence as Chunk sequence (SaC) and an end-to-end BERT-based model called Chunk-OIE for sentence chunking and tuple extraction on top of SaC. The main contribution of this paper is to demonstrate that SaC as an intermediate layer for OIE tuple extraction has better properties than the traditional approach of using a sentence as a token sequence. Additionally, the paper evaluates various chunk choices as SaC and performs a statistical study to understand how these chunks align with OIE gold tuple spans and how boundary alignment impacts overall OIE performance. Finally, the Chunk-OIE model achieves state-of-the-art results on multiple OIE datasets, demonstrating the effectiveness of the SaC approach. The paper also provides publicly available software and pre-trained models for use by the community.\n\n", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jaVUklrno0", "reviewer": "Reviewer_zi3N", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. We describe two kinds of dependency trees: 1) token-level dependency trees and 2) chunk-level dependency trees. The token-level dependency trees are identical to the tree generated from dependency parsing, as specified in Appendix A.1 \"We use NLP toolkit spaCy to extract the POS tags and dependency relations for sentences.\" We will move this sentence from the appendix to the methods section to avoid the confusion.\n2. We do not agree with the reviewer on this point. [1] and [2] enumerate potential spans of predicate and entity mentions from sentences, form them into a graph and recognize relationship between predicate and entities using the graph structural information. In contrast, the chunks of Chunk-OIE are not used as potential spans of predicate and entity mentions, since they are often parts of mention spans. Moreover, the model architecture of Chunk-OIE simultanenously recognize mention spans and their relationship at the second part of Chunk-OIE (sequence tagging at the chunk level), not first identifying spans and then grouping them into relationship.\n3. As clarified above, the chunks of Chunk-OIE are not used as potential spans of predicate and entity mentions, but guide the selection of boundaries of OIE relations/arguments. Prior works like [3] utilize syntactic structures, but are essentially token-level sequence taggers. The main difference of Chunk-OIE from them is that Chunk-OIE works on top of SaC, thus chunk-level sequence tagger, and we showed that SaC can enhance OIE in Table 1.\n4. Chunking is application-independent, though chunk types/boundaries are arguable. We adapted existing/well-known chunking schemes like CoNLL and NP and also a new scheme from OIA due to its relevance to OIE. We leave it as a future work to learn span finding for OIE.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The presentation of the overall method is clear. The method is backed by a series of experiments and ablations. It might be beneficial for the community to see how to incorporate syntactic annotations from different datasets into an open IE model. It can also be seen as a practice of integrating [1] and [2] kind of model with such syntactic information.", "reasons_to_reject": "1. The dependency trees introduced in the paper are inherently similar to trees generated from dependency parsing while the authors did not discuss any relations between these two tasks/formulations. \n\n2. Despite the novel intermediate representation claimed by the authors, the model architecture is basically the same  as (or similar to) prior work [1] and [2]. From the modeling perspective, the contribution is limited.\n\n3. It is not (directly) clear from the paper what are the semantics behind chunks. To some extent, these spans seem to be predicates and entity mentions. While most choices of chunks mentioned are defined at syntax level, it is hard to distinguish the work from prior work using syntactic structures for IE, e.g. [3].\n\n4. The choice of span finding  method should be justified. Potential through references like  [4] or other ablations.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on Open Information Extraction (OIE), which involves extracting structured tuples from unstructured text without a particular ontology. The traditional token-level extraction approach faces challenges in determining the appropriate text spans. To address this, the paper introduces an alternative method called Sentence as Chunk sequence (SaC), where chunks are recognized as relations and arguments. The paper evaluates different choices of chunks for SaC and proposes Chunk-OIE, an end-to-end tagging-based model for sentence chunking and tuple extraction. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019. Entity, Relation, and Event Extraction with Contextualized Span Representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5784–5789, Hong Kong, China. Association for Computational Linguistics.\n\n[2] Yunmo Chen, Tongfei Chen, and Benjamin Van Durme. 2020. Joint Modeling of Arguments for Event Understanding. In Proceedings of the First Workshop on Computational Approaches to Discourse, pages 96–101, Online. Association for Computational Linguistics.\n\n[3] Kuicai Dong, Aixin Sun, Jung-Jae Kim, and Xiaoli Li. 2022. Syntactic Multi-view Learning for Open Information Extraction. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4072–4083, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.\n\n[4] Weiwei Gu, Boyuan Zheng, Yunmo Chen, Tongfei Chen, and Benjamin Van Durme. 2022. An Empirical Study on Finding Spans. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3976–3983, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.\n"}}, {"review_id": "hws9SGwZ1q", "reviewer": "Reviewer_Pzff", "summary": "", "questions": "Although OIE via chunks may be a novel idea, it is not reasonable that chunking may yield better performance than token-level. Chunks may semantically represent entities and relations better, but not necessarily, since a token may be considered as the head word of the entity or relation span.\n\nThe chunk boundary and type identification are explained in the Appendix, but it is still vague. Theoretically, how can chunks be defined and annotated? In this case, it is thanks to the available dataset. In other cases, it may not be so, especially in other domains/datasets and languages.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Our design of modelling OIE via chunks does not necssarily mean all chunks are compound words or phrases. The chunked piece can be a single word token as well. The main objective is to learn a better OIE, wheras the sentence chunking can be considered as the auxiliary objective. The chunking can be learnt as the supplementary information for OIE. Thus our experiment results in Table 1 show that using chunked sequence can yield better performance than token sequence. \n2. Sentence chunking is a well-defined research task as explained in Related Work (sentence chunking). Since there are many available chunking datasets for mainstream languages, we can naturally use the off-the-shelf annotations as external knowledge to enhance OIE. Similar to most of the existing works, we conduct our research in English dataset. We agree that there might not be chunking annotations in other minor languages. For OIE for other minor languages, transfer learning or data augmentation techniques can be applied.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Experimental results showed that the proposed model achieves state-of-art results.", "reasons_to_reject": "Although OIE via chunks may be a novel idea, it is not reasonable that chunking may yield better performance than token-level. Chunks may semantically represent entities and relations better, but not necessarily, since a token may be considered as the head word of the entity or relation span.\n\nThe chunk boundary and type identification are explained in the Appendix, but it is still vague. Theoretically, how can chunks be defined and annotated? In this case, it is thanks to the available dataset. In other cases, it may not be so, especially in other domains/datasets and languages.", "questions_to_authors": "Although OIE via chunks may be a novel idea, it is not reasonable that chunking may yield better performance than token-level. Chunks may semantically represent entities and relations better, but not necessarily, since a token may be considered as the head word of the entity or relation span.\n\nThe chunk boundary and type identification are explained in the Appendix, but it is still vague. Theoretically, how can chunks be defined and annotated? In this case, it is thanks to the available dataset. In other cases, it may not be so, especially in other domains/datasets and languages.", "paper_topic_and_main_contributions": "In this paper, the authors proposed a novel method for relation classification using chunks instead of tokens in previous works. BERT is used for contextual representations. Then, sentence chunking is performed by identifying boundary and type of each chunk. A chunk-level dependency graph is derived from a token-level dependency structure. The model makes use of Graph Convolution Network (GCN) to encode the chunk-level dependency structure above. Finally, BERT and GCN-based representations are combined together and put through a softmax layer to perform relation extraction. Experimental results showed that the proposed model achieves state-of-art results.\n\nAlthough OIE via chunks may be a novel idea, it is not reasonable that chunking may yield better performance than token-level. Chunks may semantically represent entities and relations better, but not necessarily, since a token may be considered as the head word of the entity or relation span.\n\nThe chunk boundary and type identification are explained in the Appendix, but it is still vague. Theoretically, how can chunks be defined and annotated? In this case, it is thanks to the available dataset. In other cases, it may not be so, especially in other domains/datasets and languages.\n\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviews have a consensus on slightly positive soundness and excitement.

With respect to excitement, the proposed idea of introducing chunks as an intermediate representation for Open Information Extraction (OIE) is novel.  The applicability of the proposed method is limited because chunk annotations may not be readily available in non-English languages or low-resource settings.

Regarding soundness, the experiments show that the proposed method achieves state-of-the-art performance in four OIE datasets.  The core hypothesis (why employing chunks as an intermediate representation is beneficial to OIE) is not well presented.
Decision: Accept-Main