From Speculation Detection to Trustworthy Relational Tuples in Information Extraction
Authors: Kuicai Dong
Aixin Sun
Jung-jae Kim
Xiaoli Li
Venue: EMNLP 2023 Findings (2023)
Keywords: speculation detection
text classification
information extraction

Abstract: Speculation detection is an important NLP task to identify text factuality. However, the extracted speculative information (e.g., speculative polarity, cue, and scope) lacks structure and poses challenges for direct utilization in downstream tasks. Open Information Extraction (OIE), on the other hand, extracts structured tuples as facts, without examining the certainty of these tuples. Bridging this gap between speculation detection and information extraction becomes imperative to generate structured speculative information and trustworthy relational tuples. Existing studies on speculation detection are defined at sentence level; but even if a sentence is determined to be speculative, not all factual tuples extracted from it are speculative. In this paper, we propose to study speculations in OIE tuples and determine whether a tuple is speculative. We formally define the research problem of tuple-level speculation detection. We then conduct detailed analysis on the LSOIE dataset which provides labels for speculative tuples. Lastly, we propose a baseline model SpecTup for this new research task.

From Speculation Detection to Trustworthy Relational Tuples in Information Extraction
Kuicai Dong 1.2, Aixin Sun', Jung-Jae Kim', Xiaoli Li
1 School of Computer Science and Engineering, Nanyang Technological University, Singapore kuicai@01@e enttteu.......e...e ntu. edu. sg, axsun@ntu. axsun@ntu.edu.ss
Institute for Infocomm Research, A A*STAR, Singapore **TARR Centre for Frontier Al Research, Singapore {jjkim, xllijei2r a-star edu.sg
Abstract
sentences documents. Consequently, the identi- fication of speculative information, as potentially opposed to facts, has garnered considerable inter- est from NLP community (Szarvas et al., 2008; Konstantinova et al., 2012; Ghosal et al., 2022).
Speculation detection is important NLP task to understand text factuality. However, the ex- tracted speculative information (e.g., specula- tive polarity, cue, and scope) lacks structure and poses challenges for direct utilization in downstream tasks. Open Information Extrac- tion (OIE), on the other hand, extracts struc- tured tuples as facts, without examining the certainty of these tuples. Bridging this gap be- tween speculation detection and information ex- traction becomes imperative to generate struc- tured speculative information and trustworthy relational tuples. Existing studies on specula- tion detection are defined sentence level; but even if sentence is determined to be specu- lative, not all factual tuples extracted from are speculative. In this paper, propose to study speculations in OIE tuples and determine whether a tuple is speculative. We formally define the research problem of tuple-level spec- ulation detection. We then conduct detailed analysis on the LSOIE dataset which provides labels for speculative tuples. Lastly, pro- pose a baseline model SpecTup for this new research task.'
The conventional task of speculation detection involves two subtasks: cue identification and scope detection, which are typically performed at the sen- tence level. Speculation cue refers to a minimal lin- guistic unit that indicates speculation (e.g., "might", "plans to", "subject to"). Speculation scope is the text fragment governed by the corresponding cue in sentence. Consider the following example sentence: "The UN plans to [release a report].", the speculative cue "plans to" affects the mean- ing of the scope "release a report". Note that the speculation, along with its cue and scope, is of- ten unstructured. Question I arises: Can unstruc- tured speculative information be directly applied to downstream tasks?
To extract structured information, we can lever- age on Open Information Extraction (OIE). OIE aims to generate relational factual tuples from un- structured open-domain text (Yates et al., 2007). The extracted tuples are in form of (ARGo, Relation, ARG ARG,), also called facts. By definition, OIE system domain-inddependet and highly scal- able, thus allowing users to obtain facts with low cost. The extracted factual tuples are beneficial to many downstream tasks such as question answer- ing (Khot et al., 2017), knowledge base popula- tion (Martinez-Rodrrguez et al.. 2018; Gashteovski et al., 2020), and text summarization (Fan et al., 2019). Here, Question II arises: Shall we trust all relational tuples extracted by OIE as facts? Appar- ently, the answer is no, as not all sentences state facts, partly due to speculation. Current studies on OIE do not consider speculation, which can lead to the use of unreliable tuples facts, affecting the accuracy of downstream applications.
1 Introduction
The concept of speculation is closely connected to modality, which has been extensively explored in both linguistics and philosophy (Saurí, 2008). Modality is defined as the expression of the speaker's level of commitment to the events men- tioned in text (Saurí al., 2006). Other related terms, such "hedging", "evidentiality", "uncer- tainty", and "factuality", are also used when dis- cussing different aspects of speculation. is argued that information falling under speculation cannot be presented as factual information (Morante and Daclemans, 2009). In NLP applications, factual information plays a critical role in comprehending
We will release the source code upon paper acceptance.
To mitigate issues related to Question and II,

---

we propose to serialize speculative information into relational tuples. In such a way, we can empower information extraction with speculation detection to produce trustworthy relational tuples by identi- fying structured speculative information. However, there are typically multiple relational tuples in one sentence. A sentence that contains speculative in- formation does not necessarily mean that all its tuples speculative. Also, the scopes defined in existing speculation datasets do not align well with the tuples extracted by OIE. Therefore, current sentence-level speculation detection cannot be ap- plied to OIE. To bridge this gap. we propose a new research task that focuses on tuple-level specula- tion detection. In simple words, we aim to indicate whether tuple is speculative. In other words, the speculation scope becomes tuple of OIE.
Sup adds additional relation embedding into BERT transformers, and uses BERT's hidden state of the tuple relation word as semantic representa- tion. For syntactic modeling, SpecTup explicitly models the sub-graph of dependency structure of in- put sentence, which includes immediate neighbours of the tuple relation word. It adaptively aggregates nodes in the sub-grage sub-graph using novel relation-bar based GCN, and uses the aggregated representation syntactic representation. The concatenated seman- tic and syntactic representations are then used for speculation detection.
Our contributions in this paper are threefold. First, we propose a new research task to detect tuple-level speculation. This task links specula- tion detection and information extraction. It ex- amines the reliability of relational tuples, which aligns well with the goal of OIE to extract only factual information. Second, we conduct a de- tailed analysis on the tuple-level speculation labels from two aspects: (i) their presence in language, and (ii) the level of detection difficulty. Third, we propose SpecTTup, a baseline model to detect tuple- level speculation. SpecTTup leverages both seman- tic (BERT) and syntactic (Sub- Sub--eepeddency- )raapp representations. We perform extensive experiments to analyze the research task of tuple-level specula- tion, and our results show that SpecTTup effective.
To the best of our knowledge, there is no dataset specifically designed for tuple-level speculation detection. Nevertheless, the recently released LSOIE (Solawetz and Larson, 2021) provides a timely preview of this interesting research task.
LSOIE is a large-scale OIE dataset converted from the QA-SRL 2.0 dataset (FitzGerald et al., 2018). We observe that LSOIE provides additional anno- tation to some OIE tuples with modal auxiliary verbs. Table 1 lists few example sentences and their annotated OIE tuples with speculation. The modal auxiliary verbs include 'might'. 'can', 'will', "would', 'should', and 'had"; these words express the meaning of possibility, ability, intention, past intention, suggestion, and past event, respectively. It is important to note that these modal auxiliary verbs annotated on OIE relations may (43.3%) may not (56.7% % appear in the original sen- tences of the LSOIE dataset. In this study, we these modal auxiliary verbs as speculation labels for relational tuples, as they express the degree of uncertainty at tuple level.
Tuple-level Speculation Analysis
We first review the annotation processes of the QA- SRL Bank 2.0 and LSOIE datasets, with a key focus tuple-level speculation labels. We then study the distribution of speculation labels. by the perceived level of difficulty in detection.
2.1 Annotation
QA-SRL Bank 2.0 dataset consists of question- answer pairs for modeling verbal predicate- argument structure in sentence (FitzGerald et al., 2018).² A number of questions are crowdsourced for each verbal predicate in sentence, and each answer corresponds to a contiguous token span in the sentence. Examples of QA-SRL annotations can be found in Appendix A.2. Crowdworkers are required define questions following 7-slot tem- plate, i.e., Wh, Aux. Subj. Verb, Obj. Prep, Misc. Among them, 'Aux" refers to auxiliary verbs, 3 and
Tuple-Ievel speculation detection is challenging, because is common for only a certain portion of a sentence to carry speculative semantics. Certain words (e.g., "may", "if", "plan to"), also known speculation cues, are responsible for semantic uncertainty, making part of a sentence (or the cor- responding extracted tuples) vague, ambiguous, misleading. In this paper, we develop simple yet effective baseline model detect Speculation at Tuple level (called SpecTTup). SpecTupp de- tects speculation from two perspectives: semantic and syntactic. To model relation- aware semantics,
"Question-Answer Driven Semantic Role Labeling. ht tps: /ttp///aaaaeccc cs, washington. woo..eddu//rrrrr
'There are three main auxiliary verbs: 'be", "do', "have". Besides them, there's a special type that affects grammatical

---

ID Example sentence
OIE Tuple with speculation
Mcaning
Adults were allowed opt out of using computers. 2 It is unclear if the suspects left with any property. The UN plans to release a final report in two weeks.
(adults, can opr, using computers) (suspects, might left, any property) (the UN, will release, a final report) (warm salt water, should gargling)
ability
possibility
intention
Gargling with warm salt water are reasonable.
suggestion
Table I: Examples of speculation annotation in LSOIE dataset. The speculation labels are in boldface as part of tuple relation. To facilitate understanding, we underline the speculation cues in the example sentence, and elaborate the meaning of speculation label under the 'Meaning' column. Note that sentence usually contains multiple tuples, we truncate the long sentence and demonstrate only the tuple with speculation for conciseness.
Subset
#Sent
#Tuple
#Spec. Tuple
%Specc
Example sentence
Category
wikiga
4,670
10,635
1,015
9.5%
The UN will release a report.
Easy
wikigain
19,630
45,931
4,110
8.9%
The UN will recently release report.
Med
sches
6,669
11,403
1,569
13.8%
The UN plans to release report.
Hard
sclpain
19.193
33,197
4.337
13.1%
Total
50,162
101,166
11,031
10.9%
Table 3: Examples of 3 cases of speculation according to detection difficulty. All three sentences convey the same fact: (the UN, will release, report).
Table 2: Number of sentences, tuples, tuples with spec- ulation, and the percent of tuples with speculation.
dataset. 4 Overall, 10.9% of the ground truth tu- ples contain speculation, indicating the substantial presence of speculative 'facts', some of which are non-factual. However, as no OIE system considers the speculation information, considerable num- ber of unreliable facts are being extracted, without any speculative tags on them. We thus propose to develop an model that tags speculation on the extracted tuples.
answers to questions with modal auxiliary verbs reflect speculative information of the fact as well. Note that these modal verbs may may not explic- itly appear in the original sentence. Hence, they provided based on the annotator's understanding of the sentences.
The QA-SRL Bank 2.0 dataset is then converted large-scale OIE dataset (LSOIE) by Solawetz and Larson (2021). LSOIE defines n-ary tuples, in the form of (ARGo, Relation, ARG,, ... ARG,) in two domains, i.e., Wikipedia and Science. During the conversion, the modal auxiliary verbs in the QA-SRL questions are retained in tuple relations, shown in Table 1. In this study, we consider these modal auxiliary verbs to reflect speculation. Consequently, in the LSOIE dataset, tuples with speculation are those whose relation contains any of the following six modal auxiliary verbs: 'might', 'can', 'will', 'would', 'should', and 'had'. In this work, we follow Lobeck and Denham (2013) to interpret types of speculation labels as follows: "can' shows or infers general ability; 'will' and 'would' are used to show intention or to indicate certainty; 'might" shows possibility; 'should' is used suggest or provide advice; 'had" refers to past actions or events.
To the best of our understanding, neither the crowdsoureing process of QA-SRL Bank 2.0 nor the conversion of LOSIE specifically focuses on speculation, as it is not the primary focus of these two datasets. Without explicit focus, existing an- notations truly reflect the crowdworker's natural understanding, and there are no bias towards any specific type of speculation. We manually exam- ine large number of the speculative labels and
observe that labels are indeed reliable enough to be suitable for evaluating tuple-level speculations.
2.2 Perceived Level of Detection Difficulty
By analyzing the speculative labels, we have grouped them into three categories, as listed Table 3, based on the perceived level of detection difficulty. (i) The easy category refers to cases where the speculation label (e.g.. "will") literally
Table 2 reports the statistics of sentences, tu- ples, and the tuples with speculation in the LSOIE
"We notice that some sentences in the Wiki subset appear again (or repeated) in the Sei subset. In this study, we remove the repeated sentences from the Sci subset. Therefore, our reported numbers differ from those in the original paper.
mood (e.g.. 'will", 'might"), called modal auxiliary verbs.

---

Subset
#Spec. Tuple
Easy
Med
Hard
Dependency Relations
Composition
Operation
wikinest
1,015
20.0%
23.6%
56.4%
Dependency
PLM
wikimain
4,110
20.0%
22.3%
57.6%
Representation
Relations
sches
1,569
22.2%
19.1%
58.8%
Representation
scigain
4,337
22.7%
22.4%
54.9%
plans
esubi
Total
11,031
21.3%
22.0%
56.7%
Table 4: Distribution of the tuples with speculation by difficulty level.
UN det
release
prop
in
The
appears in the sentence and is located immediately beside the tuple relation (e.g., "will release")). (ii) In medium category, the speculation label is present in the sentence, but it is not located directly beside the tuple relation (e.g.. "will recently release"). (iii) The hard category refers to the cases where the speculation label is not present in the sentence at
to
report
ks
del
smod
rummodd
final
two
The UN plans to release final report in two weeks
Figure 1: The relation-based aggregation of SpecTupp The tuple relation 'release" adaptively aggregates its neighbours based on different dependency relations.
all (e.g., "plans to" means "will"".
Table 4 illustrates the distribution of speculation labels across the three levels of difficulty. As seen, 56.7% of speculative tuples in the LSOIE dataset fall under the hard category. It evident that detect- ing these hard cases of speculation is challenging, requiring deep semantic understanding of both the input sentence and the corresponding fact. Addi- tionally, we present more fine-grained breakdown of the difficulty distribution of speculation labels for each label in Appendix A.3.
As the tuples set typically contains multiple tuples, sentence s will be analyzed multiple times with different 4, We assume that the relation (or relation word) of tuple is different from the rela- tions of all the other tuples from the same sentence and thus that tuple can be represented with its relation. Note that the task can be extended predict if tuple speculative and, if speculative. which of the six modal auxiliary verbs indicates the speculation type of the tuple.
3 Task Formulation
As discussed, there are no existing OIE systems consider speculation when extracting tuples. To fully leverage the capabilitics of existing OIE mod- els, it is more meaningful and practical to formu- late speculation detection as a postt-proessingg task, where the goal is to determine whether tuple ex- tracted from a sentence is speculative.
Method: SpecTTup
The proposed baseline consists of three modules: pre-trained BERT to encode semantic represen- tation of input sequence (in Section 4.1), GCN en- coder to capture the syntactic representation (in Section 4.2), and línear layer for binary/multi- class speculation detection (in Section 4.3).
Formally, the inputs are the source sentence that is split into words ui (or tokens) s = [w) , wn), and a set of relational tuples T { tm extracted from this sentence." 5 Sentence-lever spec- ulation detection makes one-pass prediction on sen- tence s. In comparison, the task of tuple-level spec- ulation detection is to predict whether a tuple I; is speculative (i.e., binary classification task), based on l; and its source sentence 8. We focus on tuple- level speculation detection in this paper.
Given an example tuple (the UN, release, a re- port) in Figure 1, the words "plans to" in its source sentence, which modifies the relation word "re- lease", indicate the speculation of the tuple. Note that such cue of the speculation of a tuple is typ- ically syntactically related to the relation word of the tuple in its source sentence. We thus encode the dependency structure of a source sentence, as ex- emplified in Figure 1, as well as the word sequence of the sentence for the speculation detection task.
'aah tuple I; is represented by its components 4 [ 22]] where one z is the relation and the rest x's are arguments. Fach x corresponds to a contiguous span of words [,.....,, 20)++]

---

4.1 Semantic (Word Sequence)
dependency graph of the input sentence as 'full- graph', and compare their impact on the specula- tion detection task in Section 5.3.
Representation
We leverage BERT (Devlin et al., 2019) as token en- coder to obtain token representations. Specifically, we project all words (wi,.... ven] into embedding space by summing their word embeddings and tuple relation embeddings:
Relation-aware GCN. Inspired by CompGCN (Vashishth et al., 2020), we devise strategy to embed each dependency relation as a vector and aggregate the corresponding neighbouring nodes together. The representation of word denoted by hv. is updated:
h4 = W Wword(Wss (w) + (rea(tuple_rel(uu (1)
where W Woord is trainable and initialized by BERT word embeddings. Wret is trainable tuple relation embedding matrix, and the function tuple_red (w;) returns 1 if the word to; is the tuple relation of the instance; otherwise, O. Specifically, Wret initializes all words into binary embeddings (positive embed- ding for relation word, and negative embedding to non-relation words). It is worth noting that, we only utilize the tuple's ' relation word for specula- tion detection and not its arguments. Therefore, the model explicitly emphasizes the difference be- tween relation tokens and non-relation tokens.
h.m = Σ
(o r) W, sem
(3)
(ar))N(())
where S(.) is the activation function, W, C
Rd xda is trainable transformation matrix. (u(,, r) is the neighbour connecting strength, com- puted based on the dependency type
(u, r) = hare (d)
(4)
where is the dot production operator. Wdep C d XN is a trainable matrix. Ndep is the number of unique dependency relations.
Then, we use h2 [hi...., hn] as the input BERT encoder and utilize BERT's last hidden states as semantic representations:
4.3 Speculation Detection
heem = BERT(h;) E dR
(2)
Finally, we concatenate the semantic representation in Equation (2) and the syntactic representation from GCN in Equation (3) as follows:
4.2 Syntactic (Dependency Graph) Representation
Relation Sub-graph Modelling. Speculation cue refers to a minimal unit that indicates specula- tion egg,, "might", "plans to", "subject to".. How- ever, such cue words are not explicitly labelled in the source sentence. We assume that speculative tuple is the outcome of a speculation cue in the sen- tence directly modifying the tuple's relation word. Therefore, we model the words that are syntacti- cally related to the tuple's 's relation word v, through the sentence's 'ss dependency structure.
h fefinel
heem haym
(5)
where he final is used by the classification layer perform speculation detection.
For the binary classification taskof identifying if a tuple is speculative in the source sentence or not, we use binary cross-entropy loss:
N
LCE N
Σ y:log(ps) + (1 v:)log(1 pt)
Specifically, we extract a sub-dependency--ree (w) from the sentence's dependency structure, which consists of the immediate (or one-hop) neigh- bours of the tuple relation word v. In (v)), each node directly links to with an associated de- pendency relation r, as highlighted by the clear (or non-blurry) lines in Figure 1, where (u, r) de- notes the link (or edge), omitting v since all links share the common word node v. We call this sub- dependency-grapph as sub--gaph' and the whole
(6)
where is the number of training instances. is the gold standard label, and is the Softmax probability for the 7th training instance.
For the multi-class classification task of classi- fying speculative tuples into the 7 classes (non- speculative and the six modal auxiliary verbs), we use the multi-class cross-entropy loss:
'If a word maps to multiple sub-words by BERT tokeniza- tion, we average the sub-word representations.
LCEnutti
yo lo.glo(pt.s) (7)
N
1=1
1=1

---

5 Experiments
evaluate the effectiveness of using weighted pool- ing of all tokens in sentence (SEMaentence), in tuple (SEMmpl)) and in tuple relation (SeMrelation) for speculation detection.
5.1 Experiment Setup
LSOIE Dataset. To avoid potential bias intro- duced by any specific OIE model, in our experi- ments, we only use tuples from the LSOIE dataset as inputs to the proposed tuple-level speculation detection model. For the example sentence "The UN plans to release report", the tuple would be (the UN, release, report). Our task is de- termine whether this tuple is speculative and, if speculative, which the six modal auxiliary verbs indicates the speculation type. In our experiments, we combine two subsets (Wikipedia and Science) into one big dataset. As the result, we have 38,823 sentences and 79,128 tuples for training, originally from wikinain and schmain sets. We have 1 11,339 sentences and 22,038 tuples for testing, originally from the wiki and sciest sets.
Syntactic-Only. Based on the sentence depen- deney parsing, we evaluate Sulllllg and SY sub-gragahh The former uses GCN to aggre- gate node information from the entire dependency graph, while the latter only aggregates the sub- graph with one-hop neighbors of the relation word, as described in Section 4.2.
Semantic and Syntactic. We combine both se- mantic and syntactic representations and imple- ment two methods. The first one is SEMgemennc + S gullhh that uses semantic embedding and dependencies of all tokens in the sentence. In com- parison, Spec leverages only the embeddings and sub-graph dependencies of tuple relation token, which equivalent to SEMrelation SYN sub- graph.
Evaluation Metries. We use Precision, Recall, and Fi measures for evaluation. We report the results of the binary classification task from the following three perspectives:
Keywords Matching in Dependency sub-graph. Besides the neural baselines mentioned above, we also experiment with simple keywords matching methods. For these methods, we use a pre-defined speculative keywords dictionary. A tuple is classi- fied as speculative if any of its immediate neigh- bours in dependency parse tree is one of the words in the dictionary. We first use the modal verbs as the dictionary. Then we additionally include the most frequent speculative keywords (top 10, 20, and 30). Speculative keywords selection is described in details in Appendix A.4.
(1) Macro-averaged scores. This is the un- weighted average over the two classes, i.e,, specu- lative and non-speculative tuples. Micro-average is not used because nearly 89.1% of tuples are non- speculative, which would dominate the measures. (2) Positive scores. This is the set of measures computed for the positive (i.e,, speculative) class. (3) Recall by difficulty. This is recall of the speculative tuples by the perceived difficulty level (Section 2.2): the percentage of easy, medium, and hard cases that are correctly detected."
5.3 Main Results
5.2 Baselines
Table 5 reports the experimental results of the pro- posed baseline methods.
As this is new research task, there are no exist- ing baselines for comparison. We have designed four sets of methods to compare: (i) only semantic representations, (ii) only syntactic representations, (iii) both semantic and syntactic representations, and (iv) keywords matching.
Semantic Syntactic. The three baselines using semantic representations significantly outperform the two baselines using syntactic representations. This highlights that the semantic information is more crucial for the model to understand specula- tion. We also observe that the recall scores of syn- tactic models are comparable to those of semantic models. By combining both semantic and syntac- tic information, SpecTup outperforms all baselines, particularly in recall and F1 scores.
Semantic-Only. We leverage on BERT to get the semantic representations of the source sentence s. Note that all tokens in the sentence are encoded with tuple relation embedding (see Equations I and 2). As result, the tuple relation information is explicitly fused to all source tokens. We then
Semantic: Tuple Relation vs Full Sentence Tu- plc. For the baselines with semantic representations only, the baseline using tuple relation performs not
'Precision is not applicable here, as the task is not to pre- dict the difficulty level of each tuple.

---

Macro-averaged Re
Positive Pr
Recall by difficulty Medium
Models
Easy
Hard
Keyword Matching (Dep sub-graph)
Modal Verbs
79.6
66.6
70.6
67.4
35.7
46.6
99.8
59.9
3.6
Modal Verbs Top 10 speculative words
79.3
72.3
75.1
65.4
48.0
55.4
99.8
60.8
24.5
Modal Verbs Top 20 speculative words
77.8
72.2
74.6
62.4
48.4
54.5
99.8
60.8
25.3
Modal Verbs Top 30 speculative words
77.4
72.2
74.4
61.6
48.5
54.3
99.8
60.8
25.5
Semantic-only (BERT)
SEM.eanece
86.6
72.0
76.9
80.0
45.6
58.
99.8
46.4
25.6
SEMMupic
84.0
73.2
77.1
76.0
47.9
58.8
99.8
57,6
23.2
SEMaciaion
84.5
73.1
77.3
75.7
48.3
59.0
99.8
64.4
23.4
Syntactic-only (Dependency)
saNghh-gnahh
72.3
70.2
71.0
53.2
46.6
49.7
95.4
41.0
35.5
72.8
70.0
71.2
54.3
46.2
49.9
95.6
42.5
34.7
Semantic and Syntactic
SEMseattnce SYNna-gaph
82.4
74.0
70.5
52.3
60.1
32.8
SEMectaion guggh
75.4
78.3
67.2
57.0
61.7
62.3
37.5
SEMrclsion S Natggophh (SpecTuup)
80.7
77.5
79.0
66.9
58.9
62.6
40.9
Table 5: Main results for binary tuple speculation detection. The best results are in boldface.
worse than the other two baselines using more in- formation (tuple and full sentence), suggesting that
Spec Type
#N
Pr
Re
Fi
% Hard
Non-Spec
19,456
93.5
97.7
95.6
some relations are more likely be speculative than others, and the BERT encoding has implicitly considered the contextual information.
Spec
2,616
66.7
46.6
54.9
57%
1,003
87.6
47.9
61.9
40%
might
720
53.9
39.0
45.3
94%
will
339
82.4
59.3
38%
should
283
39.4
56.2
46.3
82%
Syntactic: Sub-graph vs Full-graph. In terms of syntactic modeling, the baselines using sub- graphs are slightly better than those using full- graphs, indicating that is valuable to encode spec- ulation cues with the other immediate neighbours the tuple relation.
would
170
85.3
37.6
28%
had
94.4
33.7
Table 6: Breakdown results multi-class speculation classification by speculation class.
Neural Methods Keywords Matching Ob- serve that SpecTup outperforms all keywords matching methods by a large margin. In particular, the recall of hard case speculations in SpecTTup nearly double that of keyword matching, indicating the advantages of semantic and syntactic modelling through neural networks.
6.1 Results Overview
As reported in Table 6, the multi-class Fi score of speculative classes decreases 54.9%, compared to the positive Fi score of 62.6% in Table As expected, detecting speculative type is much more challenging than the existence of speculation.
Overall, SpecTTup achieves the best results. How- ever, the Fi of speculative class is only 62.6, leav- ing a big room for future investigation.
Table also reports the break-down precision, re- call, scores the 6 different speculative classes. The performance of these classes are determined mainly by two factors: (i) The hard cases are nat- urally more challenging, because the speculation labels do not appear in source sentences. They can only be inferred based on the semantics. We ob- serve that 'might' class consists of 94% of hard cases, leading the lowest F1 score 45.3% among all speculative classes. (ii) The performance is af- fected by the number of instances. The 'had' class minor class, with a very low F1 score 49.6%, even when only 36% of its labels are hard cases.
6 Multi-class Speculation Classification
As mentioned in Section 3, we extend the task to predict both the existence of speculation and its specific type, as defined by the auxiliary modal verbs: 'might", 'can', 'will', 'would', 'should', and 'had'. this way, we perform multi-class classi- fication among classes: non non-speculative and types of speculative classes.

---

ID Example Sentence
Spec Cue
OIE Tuple with Speculation
The UN plans to release a final report.
plans to
(the UN, will release, a final report)
The UN plans reduce troops.
plans
(the UN, might reduce, troops)
3
Charlemagne planned to continue the tradition.
planned to
(Charlemagne, would continue, the tradition)
Table Three examples of speculative tuples in LSOIE with similar speculation cue ("plan to"). but with different speculation labels. We truncate the long sentence and demonstrate only the tuple with speculation for conciseness.
20200.
Related Work
17300
Frequency of Speculative Labels
19000-
Speculation Detection. Speculation detection from text essential in many applications in in- formation retrieval (IR) and information extrac- tion (IE). Research in computational linguistics has made significant advances in detecting spec- ulations. Remarkably influential datasets include the BioScope Corpus for uncertainty and negation in biomedical publications (Szarvas al., 2008), the CoNLL 2010 Shared Task (Farkas al., 2010) for detecting hedges and their scope in natural lan- guage texts, unifying categorizations of semantic uncertainty for cross-genre and cross domain un- certainty detection (Szarvas et al., 2012), and the SFU Review Corpus (Konstantinova et al., 2012) for negation, speculation and their scope in movie, book, and consumer product reviews.
800-
7500
5050
2500
200
son-spe
Spec
might
whould
would
had
Figure Distribution Non-spec and spec labels, and the break-down distribution of different speculative types.
6.2 Long-tail Distribution
The labels exhibit a long-tail class distribution. As shown in Figure 2, 10.9% tuples are speculative. Among them, 'might' and 'can' contribute 65.9% of the speculative labels. Other speculative types are associated with relatively small number sam- ples. Such class imbalance is a key source of per- formance degradation in classification. For com- parison, the F1 scores of head class 'Non-spec' and tail class 'Spec' are 95.6% and 54.9%, respectively (with big difference of 40.7%). Among specula- tive classes, the Fi scores of head class 'can' and tail class 'had" are 61.9% and 49.6%, respectively (with big difference of 12.3%), although the per- cent of hard cases of 'can' is even higher than that of haad (40% > 36%).
Current work on speculation is to detect the existence of speculation and/or the speculation cue/scope in given sentence. The speculation scope is the maximum number of words affected by the phenomenon. The speculation cue refers to the minimal unit which expresses speculation. For instance, the most frequent 4 keywords ("if", "or", "can", and "would") contribute 57% of the total speculation cues in SFU Review Corpus. We argue that the detection of such speculation cue at sentence-level is relatively casier. In comparison, tuple-level speculation cue detection requires not only locating the correct speculative keywords, but also assigning them to the correct tuples. It is crit- ical for many real-world applications relying on updated and accurate tuples in knowledge base.
6.3 Ambiguous Speculative Types
The annotation of speculative types can be ambigu- ous, due to subtle difference between speculation labels. Table 7 lists three example sentences, all containing "plan to"", which can express both like- lihood and intention. Different annotators might interpret the sentence in different ways, thus an- notating the same speculation cue with different labels. Manual checking of annotations in LSOIE shows that such ambiguous labels are rather com- mon, particularly among hard cases. Such ambigu- ous hard cases are more challenging to distinguish.
Traditional and Neural OIE systems. Open Information Extraction (OIE) was first proposed by Yates et al. (2007). Before deep learning era, many statistical and rulc-based systems have been proposed, including Reverb (Fader et al., 2011), Clausie (Corro and Gemulla, 2013), and Stanford OpenTE (Angeli al., 2015), to name a few. These models extract relational tuples based on hand- crafted rules or statistical methods. The extraction

---

mainly relies on syntactic analysis. Recently, neural OIE systems have been devel- oped and showed promising results. Neural OIE systems can be roughly divided into two types: generative and tagging-based (Zhou et al., 2022). Generative OIE systems (Cui et al., 2018; Kolluru et al., 2020a; Dong et al., 2021) model tuple ex- traction sequence to-sequence generation task sequencce-o-ssequence with copying mechanism. Tagging-based OIE sys- tems (Stanovsky al., 2018; Kolluru et al., 2020b; Dong et al., 2022) tag each token as a sequence tagging task. Most neural OIE models are designed for end-to-end training and the training data are mostly silver data generated by traditional systems.
worker's natural understanding without any bias towards specific focusing on speculation. There- fore, we argue that the annotation is suitable for evaluating tuple-level speculations.
Quality of OIE Tuples. We use the ground truth tuples of LSOIE as inputs, to avoid potential bias introduced by any specific OIE model (see Sec- tion 5.1). However, existing OIE systems are not so perfect, as we note that the state-of-the-art Fi score in LSOIE dataset 0.71 (Vasilkovsky et al., 2022). The tuple-Ievel speculation detection, as a post-possee post-possessing task, will inevitably suffer from the imperfectly extracted tuples. However, SpecTup can largely mitigate such issue. SpecTuup only relies on the tuple relation for the speculation detection, rather than taking the full tuple with all arguments. Tuple relation is usually verb or verbal phrase that are straightforward to obtain. Therefore, so long as an OIE system can extract correct tuple re- lations, SpecTup can make predictions accordingly.
8 Conclusion
Speculation detection, which is essential in many applications in IR and IE, has not been explored in OIE. So formally define a research task to perform luple-level speculation detection on OIE extracted tuples. We notice that the LSOIE dataset, although not dedicated for speculation detection, provides us a timely preview of this interesting research task. We conduct a detailed analysis on the speculative tuples of the LSOIE dataset. To provide glimpse of this research task, we develop simple but effective model named Suec to detect tuple- level speculation. As emerging research area, believe there is a big room for further exploration, from dataset construction to more effective models.
Modelling Speculation Cues. We model the im- mediate neighbours of the tuple relation in depen- deney parsing speculation cues (sce Section 4.2). However, some speculation cues are not the im- mediate neighbours of the tuple relation. However, considering the full dependency tree leads to poorer results. We leave for future work to explore a better way to effectively model speculation cues.
A Acknowledgments
References
This research is supported by the Agency for Sci- ence, Technology and Research (A (A*STAR) under its AME Programmatic Funding Scheme (Project #A #A18A2b0046 and #A19E2b0098).
Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D. Manning. 2015. Leveraging linguis- tic structure for open domain information extraction. In Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing ( (Volmme I: Long Papers), pages 344-354, Beijing, China. Association for Computational Lin- guistics.
Limitations
We analyze the limitations of our work from three aspects as follows.
Luciano Del Corro and Rainer Gemulla. 2013. Clausie: clause-based open information extraction. In 22nd International World Wide Web Conference, WWW '13 Rio de Janeiro, Brazil, May 13-17, 2013, pages 355-366. International World Wide Web Conferences Steering Committee/ / ACM.
Annotation Quality of Speculation. Annotating speculation is challenging due to the subtle dif- ference between different speculative types. As discussed in Section 2.1, neither the crowdsoure- ing of QA-SRL Bank 2.0 nor the conversion of LOSIE pays specific attention to speculation. The annotation of auxiliary modal verbs in the question is based on crowd workers' natural understanding of the sentences. Without explicit focus on spec- ulation, the existing annotations reflect the crowd
Lei Cui, Furu Wei, and Ming Zhou. 2018. Neural open information extraction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), pages 407-413, Melbourne, Australia, Association for Computational Linguistics.

---

Jacob Devlin, Ming-Wor Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume 1 (Log and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Tirthankar Ghosal, Kamal Kaushik Varanasi, and Valia Kordoni. 2022. Hedgepeer: A dataset for uncertainty detection in peer reviews. In Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries, JCDL '22, New York, NY, USA. Association for Computing Machinery.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2017. Answering complex questions using open informa- tion extraction. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 31 311-316, Vancouver, Canada. Association for Computational Linguistics.
Kuicai Dong, Aixin Sun, Jung-Jae Kim, and Xiaoli Li. 2022. Syntactic multi-view learning for open infor- mation extraction. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, Online. Association for Computational Linguistics.
Keshav Kolluru, Samarth Aggarwal, Vipul Rathore, Mausam, and Soumen Chakrabarti. 2020a. IMoJIE: Iterative seeoory-bssed joint open information ex- traction. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 5871 Online. Association for Computa- tional Linguisties.
Kuicai Dong, Zhao Yilin, Aixin Sun, Jung-Jae Kim, and Xiaoli Li. 2021. DocOIE: A document-level context-aware dataset for OpenIE. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 2377-23888 Online. Association for Computational L Linguistics.
Keshav Kolluru, Adlakha Vaibhav, Samarth Aggarwal, Mausam, and Soumen Chakrabarti. 2020b. Open1E6: Iterative Grid Labeling and Coordination Analysis for Open Information Extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3748-3761, Online. Association for Computational Linguistics.
Anthony Fader. Stephen Soderland, and Oren Etzioni. 2011. Identifying relations for open information ex- traction. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1535-1545, Edinburgh, Scotland, UK. Associ- ation for Computational Linguistics.
Natalia Konstantinova, Sheila C.M. de Sousa, Noa P. Cruz, Manuel Maña, Maite Taboada, and Rus- lan Mitkov. 2012. A review corpus annotated for negation, speculation and their scope. In Proceed- ings of the Eighth International Conference on Lan- guage Resources and Evaluation (LREC'12), 12), pages 3190-3195, Istanbul, Turkey. European Language Resources Association (ELRA).
Angela Fan. Claire Gardent, Chloé Braud, and Antoine Bordes. 2019. Using local knowledge graph con- struction to scale Seq2Seq models to multi-document imputs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natu- ral Language Processing (EMNLP-IJCNL pages 4186 166--115 4196, Hong Kong, China. Association for Com- putational Linguistics.
Anne Lobeck and Kristin Denham. 2013. Navigating English grammar: A guide to analyzing real lan- guage. John Wiley & Sons.
Richard Farkas, Veronika Vineze, György Móra, János Csirik, and György Szarvas. 2010. The CoNLL- 2010 shared task: Learning to detect hedges and their scope in natural language text. In Proceed- ings of the Fourteenth Conference on Computational Natural Language Learning- Shared Task, pages 1- 12, Uppsala. Sweden, Association for Computational Linguistics.
José-Lázar Martinez Rodriggzzz Ivan López-Arévalo, and Ana B. Ríos-Alvarado. 2018. Openie- based ap- proach for knowledge graph construction from text. Expert Syst. Appl., 113:339-355.
Roser Morante and Walter Daelemans. 2009. Learning the scope of hedge cues in biomedical texts. In Pro- ceedings the BioNLP 2009 Workshop, pages 28-36, Boulder, Colorado. Association for Computational Linguistics.
Nicholas FitzGerald, Julian Michael, Luheng He, and Luke Zettlemoyer. 2018. Large-scale QA-SRL pars- ing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- ume I: Long Papers), pages 2051-2060. Melbourne, Australia. Association for Computational Linguistics.
Roser Saurí. 2008 A factuality profiler for eventualities in text. Brandeis University.
Kiril Gashteovski, Rainer Gemulla. Bhushan Kotnis, Sven Hertling, and Christian Meilicke. 2020. On aligning OpenIE extractions with knowledge bases: case study. In Proceedings of the First Work- shop on Evaluation and Comparison of NLP Systems, pages 143-154, Online. Association for Computa- tional Linguistics.
Roser Saurí and James Pustejovsky. 2009. Factbank: corpus annotated with event factuality. Lang. Resour. Evaluation, 43(3):227-268.
Roser Saurí, Mare Verhagen, and James Pustejovsky. 2006. Annotating and recognizing event modality in text. In Proceedings of the Nimeteenth International

---

Florida Artificial Intelligence Research Society Con- ference, Melbourne Beach, Florida, USA, May 11-13, 2006, pages 333-339. AAAT Press.
A
Appendix
A.1 Implementation Detail
Jacob Solawetz and Stefan Larson. 2021. LSOIE: A large-scale dataset for supervised open information extraction. In Proceedings of the 16th Conference of the European Chapter of the Association for Compu- lational Linguistics: Main Volume, pages 2595- 2600, Online. Association for Computational Linguisties.
We build and run our system with Pytorch 1.9.0 and AllenNLP 0.9.0 framework. The dependency annotations are by spacy-transformers. 'Theee are in total 45 types of dependency labels. We tokenize and encode input sentence tokens using bert-base- uncased. 9 The experiments are conducted with Tesla V100 32GB GPU and Intel Xcon @ Gold 6148 2.40 GHz CPU. The experimental results are averaged over 3 runs with different random seeds. Each epoch is around 15 minutes V100 32GB GPU. The hidden dimension d, for semantic representation h and that of syntactic representation h, are both 768.
Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer. and Ido Dagan. 2018. Supervised open information extraction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long Papers), pages 885-895, New Orleans, Louisiana. Association for Computa- tional Linguistics.
single Tesla
György Szarvas, Veronika Vineze, Richárd Farkas, and János Csirik. 2008. The BioScope corpus: anno- tation for negation, uncertainty and their scope in biomedical texts. In Proceedings of the Workshop on Current Trends in Biomedical Natural Language Pro- cexsing, pages 38-45, Columbus, Ohio. Association for Computational Linguistics.
A.2 QA-SRL Annotation
QA-SRL stands for Question-Answer (QA) driven Semantic Role Labeling (SRL). It is task for- mulation which uses question-answer pairs la- bel verbal predicate-argumeet structure (FitzGerald et al., 2018). Given sentence s and a verbal pred- icate u from the sentence, annotators are asked to produce a set of wh-questions that contain v and whose answers are phrases in s. The questions are constrained in a template with seven fields, E Whx Aubjxx Verb Objx Prepx each associated with list of possible options. Answers are constrained to be a subset of words in the sen- tence but not necessarily have to be contiguous spans. Table 8 shows example annotations of an input sentence "The UN plans to release a final re- port in two weeks.". In its first annotated question: "When will someone release something" contains a verb and its answer is phrase "in two weeks" in the sentence. The answer tells us that "in two weeks" is argument of "release". Enumerating all such pairs provides a relatively complete representation of the verb's arguments and modifiers.
György Szarvas, Veronika Vineze, Richárd Farkas, György Móra, and Iryna Gurevych. 2012. Cross- genre and cross-domain detection of semantic uncer- tainty. Computational Linguistics, 38(2)3335-36
Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, and Partha Talukdar. 2020. Composition- based multi- relational graph convolutional networks. In Interna- rional Conference on Learning Representations.
Michael Vasilkovsky, Anton Alekseev, Valentin Ma- lykh, Ilya Shenbin, Elena Tutubalina, Dmitriy Sa- likhov, Mikhail Stepnov, Andrey Chertok, and Sergey Nikolenko. 2022. Detie: Multilingual open informa- tion extraction inspired by object detection. In Pro- ceedings of the 36th AAAI Conference on Artificial Intelligence.
Alexander Yates, Michele Banko, Matthew Broadhead, Michael Cafarella, Oren Etzioni, and Stephen Soder- land. 2007. TextRunner: Open information extrac- tion on the web. In Proceedings
Human Lan- guage Technologies: The Annual Conference of the North American Chapter of the Association for Com- putational Linguisties (NAACL-HLT), pages 25-26, Rochester, New York, USA. Association for Compu- tational Linguistics.
A.3 Fine-grained Breakdown of Difficulty Distribution
Table 9 presents more fine-grained breakdown of the difficulty distribution of speculation labels for each label. It shows that 92.5% of 'might' la- bels and 81.5% of 'should' labels belong to hard category, while the distribution of other labels is
Shaowen Zhou, Bowen Yu, Aixin Sun, Cheng Long. Jingyang Li, and Jian Sun. 2022. A survey on neural open information extraction: Current status and fu- ture directions. In Proceedings of the Thirty-First International Joint Conference on Artificial Intel- ligence, IJCAI-22, pages 5694-5701. International Joint Conferences on Artificial Intelligence Organi- zation. Survey Track.
http://ppoy..oonnieeererrrrreeeo spacy- trassfrranssrr "https://huggingfee http://hwggrnnaaaacceeoeeeeeeeee

---

7-stot Question
Verbal predicate
Answer
Wh
Aux
Subj
Verb
Obj
Prep
Misc
When
will
someone
release
something
in two weeks a final report The UN
release
What
will
someone
release
Who
will
release
something
Who
plans
do
something
The release a report
plans
What
does
someone
plan
to do
Table 8: The QA-SRI. annotations for a newswire sentence: "The UN plans to release a final report in two weeks."
even. This can help use understand better the challenges of speculation detection task.
expected. We then include more speculative key- words in our keywords dictionary.
Specifically, we sum the frequency of all fre- quent cues across four datasets Table 10, and keep the 30 most frequent cues as exemplified Table 11. We then build three variants on top of the basic dictionary, by adding the top 10, 20, and 30 speculative cues to the dictionary. The enriched dictionary largely increases the recall scores (see Table 5). We notice that including top 10 key- words significantly inereases the recall of spec- ulative class by 12.3%, and recall of hard case speculations by 20.9%. comparison, including additional top 1 11-0 keywords only marginally in- creases the recall of speculative class by 0.4%, and recall of hard case speculations by 0.8%. Further- more, the increase of including additional top 21-30 keywords is negligible.
A.4 Speculative Keywords
Speculation keyword or cue is the minimal unit which expresses speculation (Szarvas et al., 2008). Konstantinova et al. (2012) annotate the Simon Fraser University Review corpus 10 with negation, speculation, and their cues. This corpus consists of 400 user reviews from Epinions. com. The 10 most frequent speculative keywords the SFU Review Corpus are listed in Table 10.
Szarvas et al. (2012) select three corpora (i.e., BioScope, WikiWeasel, and FactBank) from differ- ent domains, e-g., biomedical, encyclopedia, and newswire. The BioScope corpus (Szarvas al., 2008) contains clinical texts as well as biological texts from full papers and scientific abstracts; the texts are manually annotated for hedge cues and their scopes. The WikiWeasel corpus (Farkas et al., 2010) is annotated for weasel cues and semantic uncertainty, from randomly selected paragraphs taken from Wikipedia pages. The FactBank is newswire dataset (Saurí and Pustejovsky, 2009). Events annotated in the dataset and they are evaluated on the basis of their factuality from the viewpoint of their sources. The 20 most frequent speculative keywords the BioScope, WikiWeasel, and FactBank are also shown in Table 10.
Speculative Keywords Matching. As discussed Section 5.2, we consider keyword matching as simple baselines. The matching relies on a pre- defined speculative keywords dictionary. A tuple is classified as speculative if its immediate neigh- bours in dependency parsing tree contain any word in the dictionary. We first use the 6 auxiliary modal verbs ('mighi", 'can', 'will', 'would', 'should'. and "had') as the dictionary. Shown in Table 5, modal verbs based matching leads to very low recall, as
10 #http:// http://www.stte www. Sttaabbbdd//FURRevvi/ia Corpus. html

---

Type
can
might
should
would
DifT.
Num
(%)
Num
(%)
Num
(%)
Num
(%)
Num
(%)
Num
Easy
1,111
36.1
114
3.9
512
36.3
84
8.1
349
39.7
189
34.4
Medium
1,404
33.0
104
3.6
384
27.2
108
10.4
267
30.3
162
29.5
Hard
1,743
40.9
2,676
92.5
516
36.5
845
81.5
264
30.0
198
36.1
Total
4,258
100
2,894
100
1,412
100
1,037
100
880
100
549
100
Table 9: Number and percentage (%) of speculation labels by class and by difficulty level.
SFU
BioScope
FactBank
WikiWeasel
Cue
Frequency
Cue
Frequency
Cue
Frequency
Cue
Frequency
if
876
suggest
810
expect
75
may
721
or
820
may
744
if
65
if
254
can
765
indicate
404
would
50
consider
250
would
594
investigate
221
may
43
believe
173
could
299
uppear
213
could
29
would
136
should
213
197
possible
26
probable
112
think
211
possible
185
whether
26
suggest
108
may
157
examine
183
believe
25
possible
93
seem
150
whether
169
likely
24
allege
81
probably
121
might
155
think
24
likely
80
117
might
23
might
78
likely
117
will
21
seem
67
could
112
until
16
think
study
101
appear
regard
11
could
determine
87
potential
whether
putative
80
probable
perhaps
hypothesis
77
suggest
will
think
66
allege
8 appear
would
52
accuse
until
Table 10: Most frequent speculation cues according to different domains/datasets. The statistics of SFU are from paper (Konstantinova et al., 2012) and other statisties are from paper (Szarvas et al., 2012).
Top 1-10
Top 11-20
Top 21-30
Cue
Freq.
Cue
Freq.
Cue
Freq.
may
1665
appear
260
probahle
122
1294
might
256
probably
121
1017
consider
250
study
101
suggest
whether
247
allege
can
882
seem
228
determine
would
832
investigate
221
putative
80
could
495
likely
221
hypothesis
indicate
404
should
213
expect
75
think
362
believe
198
will
60
possible
304
examine
183
regard
Table 11: The 30 most frequent speculation cues by summing the frequency in Table 10.


Reviews:
[{"review_id": "1Or0pADS7Q", "reviewer": "Reviewer_g5sr", "summary": "", "questions": "1. Why not use the superior RoBERTa model, which has a parameter count similar to BERT?\n\n2. Is BERT fully fine-tuned in your approach, or is it just used as a feature extractor? How does the performance of directly fine-tuning BERT compare?\n\n3. The authors mentioned they manually checked a large number of labels (lines 221-224). Are there more specific details available regarding this aspect?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. The proposed baseline consists of three layers: (1) BERT, (2) GCN encoder, and (3) linear layer for binary/multi-class classification. Section 4 may be complicated without any illustration. We will include a diagram of the overall model architecture with the three layers in the paper.\n2. The BERT module used in our baseline is fine-tuned during the training process. Rather explore many different PLMs such as RoBERTa, ELECTRA, T5, etc, we believe using BERT is sufficient for the exploration of the new tuple-level speculation. We did experiments on RoBERTa for the task, and found the experimental results are similar to BERT. So we only reported the results of using BERT, as tuple-level speculation is our key focus. We will include the results of RoBERTa in the appendix.\n3. Regarding the manual checking of labels described in lines 221-224, our examination focuses on difficult labels that are not present in the sentence. The checking is to ensure the speculation labels in the dataset do carry speculative meaning. Thanks for pointing it out, we will report more details in the appendix in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposes an interesting research problem.\n\n2. The authors design a strong baseline model and demonstrat that there is still a huge room for improvement in this task.", "reasons_to_reject": "1. The proposed baseline model is complex and not easy to follow.\n\n2. Limited dataset.\n\n3. Limited comparison models, especially lacking comparison with the latest in-context learning methods.", "questions_to_authors": "1. Why not use the superior RoBERTa model, which has a parameter count similar to BERT?\n\n2. Is BERT fully fine-tuned in your approach, or is it just used as a feature extractor? How does the performance of directly fine-tuning BERT compare?\n\n3. The authors mentioned they manually checked a large number of labels (lines 221-224). Are there more specific details available regarding this aspect?", "paper_topic_and_main_contributions": "This paper presents an interesting research task: detecting the trustworthiness of tuples obtained from information extraction. The authors analyse the LSOIE dataset and find that about 10.9% of the labeled tuples contain speculative information. They model tuple-level speculation detection as a classification task and propose a detection model, SpecTup, that utilizes contextual semantic information and dependency structure information to classify speculative tuples. Comparison with keyword matching methods and ablation experiments demonstrate that the proposed method has better performance.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SpzuJPtgnU", "reviewer": "Reviewer_hXdC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. We agree that Large Language Model has achieved promising results on NLP tasks. The primary focus of our work is to propose a new research topic, rather than proposing a strong and sophiscated baseline to solve it. We will include some results of using LLM (e.g. Llama 2) in the appendix.\n2. The fusion of syntactic information (dependency parsing tree) and semantic information (PLM: BERT) has been explored in some NLP models. The novelty of our fusion methodoloy is to design a novel relation-aware GCN that can adaptively aggregate neighbours based on different dependency relations. Moreover, existing methods focus on the syntactic structure of entire sentences, whereas our method focuses on the local syntactic information, i.e., speculation cue words.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ Novelty: This paper presents a new research task that bridges the gap between speculative detection and information extraction, emphasizing the impact of speculative information on the trustworthiness of relational tuples.\n\n+ Effect: A comprehensive experimental analysis demonstrates the effectiveness of the proposed method.", "reasons_to_reject": "- The large language model has achieved amazing capabilities in the NLP community. Whether it can also be solved well in this task, the author did not mention the application and effect of LLM. It is recommended to take this factor into consideration.\n\n- This method combines semantic and grammatical information through different models, which is not uncommon, and the innovation of this method seems to be relatively small.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a new research task of detecting speculative information in Open Information Extraction (OIE) tuples. It highlights the impact of speculative information uncertainty on relational tuples, introducing a baseline model (SpecTup) based on semantic and syntactic modeling for tuple-level speculative detection. We analyze the existence and difficulty of tuple-level speculative labels, and verify the effectiveness of the proposed model on this task through experiments.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LGsnqwSzpI", "reviewer": "Reviewer_Qo8K", "summary": "", "questions": "In lines 221-224, the author mentioned that they manually examined speculation labels.  What are the results of these examinations?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Regarding the potential bias of annotation of speculation labels [214-224; 614-627], we agree that validating and aligning two methodologies are important. However, it is costly to hire annotators to complete it. So we manually examnined a large number of labels (about 1,000 cases). Our examination focuses on difficult labels that are not present in the sentence. The checking is to ensure the speculation labels in the dataset do carry speculative meaning. Thanks for pointing it out, we will report more details in appendix in the revised version to report the relaiblility of speculative labels.\n2. Thanks for the pointing out the citation of definition of model words, we will cite peer-reviewed sources instead of the blog post in our revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper puts forth an interesting OIE subtask that attempts to classify whether an extracted tuple is speculative. This often overlooked aspect of IE carries significant real-world implications. The method is quite simple, and it appears that the classification technique can be readily applied to other IE tasks, such as temporal IE. The proposed GNN baseline also offers a solid starting point for future research.", "reasons_to_reject": "The main issue with this work is that the authors directly derived speculation labels from modality labels. While modality and speculation are closely related, there are several distinctions. For instance, the authors consider the modal verb \"would\" speculative; however, it also indicates a habitual event. For example, in the sentence \"when I was an undergrad, I would fall asleep with music,\" the tuple _(I, would fall, asleep, with music)_ cannot be seen as speculative. Although the authors' direct conversion of modal verbs is an advantageous first step towards tuple-level speculation detection, it is necessary to label speculation information explicitly.\n\nThe authors acknowledged this issue [214-224; 614-627]. They argue since LOSIE does not primarily focus on explicitly labeling speculation information, it does not introduce bias towards any specific type of speculation [218-221]. However, I disagree with this particular claim. When the target of annotation differs, the first step should involve validating and aligning the two methodologies before discussing bias. The authors also mentioned they manually examined a large number of speculative labels [221-224], yet they failed to report any results from these examinations. Providing statistical evidence to support the claim that \"the labels are reliable enough\" would indeed be beneficial.\n\nMinor points:\nThe citation Ellis (2022) is merely a blog post: https://www.grammarly.com/blog/auxiliary-verbs/.  It would be better to obtain definition of modal from peer-reviewed sources.\n", "questions_to_authors": "In lines 221-224, the author mentioned that they manually examined speculation labels.  What are the results of these examinations?", "paper_topic_and_main_contributions": "This paper introduces a novel task of tuple-level speculation detection. The objective is to determine whether a tuple extracted by an OIE system is speculative. This task offers a more precise refinement compared to previous sentence-level speculation detection tasks. \n\nTo obtain the speculation label of tuples, the authors relied on a naive conversion of the modal verb annotation from the LSOIE dataset. Specifically, if the predicate label in LSOIE includes any of the six modal verbs (might, can, will, would, should and had), the tuple is considered speculative.\n\nFurthermore, the authors present a simple yet effective GNN baseline for tuple-level speculation detection. This baseline surpasses other naive baselines by a significant margin, providing a promising starting point for future research endeavours.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AZZyyI9TAR", "reviewer": "Reviewer_6jWm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper focuses on the tuple granularity speculation detection task from a new perspective, which can be better applied to downstream tasks.\n2. This paper organizes the corresponding dataset for the new task which can help further research in related fields.\n3. Different dimensions deeply analyze the value of the newly proposed task and the effectiveness of the newly proposed method of the newly constructed dataset.\n", "reasons_to_reject": "1. The experiment is not sufficient. Although it is a new task and there is no SOTA model, it should also be compared with the commonly used SOTA model for similar tasks.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on speculation detection tasks but at a tuple granularity that has not been addressed before, rather than the common sentence granularity. This paper organizes the corresponding dataset for the new task and verifies the effect of the proposed model to a certain extent through experiments on it.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. There are some formatting issues and typos in the article, such as missing content in the first sentence of the first paragraph."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers have agreed that this paper has proposed a very interesting new task to detect the trustworthiness of extracted tuples.

However, many concerns have also been raised regarding to the experiments. And it seems that through in-depth rebuttal, these concerns remain to a large degree. 

Generally, this paper anchors a new direction that worths further research.
Decision: Accept-Findings