DEIM Forum 2019 C6-1

Answering Single-Event Questions on Long-term News Article Archives
Wang Jiexin†, Adam JATOWT†, Michael Färber†, Masatoshi YOSHIKAWA††Graduate School of Informatics, Kyoto University
Yoshida-honmachi, Sakyo-ku, Kyoto, 606–8501, Japan
E-mail: †wangjiexin.scut@yahoo.com, ††adam@dl.kuis.kyoto-u.ac.jp, †††michael.faerber@cs.uni-freiburg.de,††††yoshikawa@i.kyoto-u.ac.jp
Abstract With the development of digital preservation techniques, more and more news articles have been digitized. Newsarchives contain immense value, helping people to know the detailed information of events that occurred at specific time points.
Question Answering systems can help to utilize these documents effectively and satisfy users’ information needs, which usuallyare represented as natural language questions that require returning of precise answers. This article presents a QAA (Question
Answering in Archives) system, a question-answering system designed specifically for answering news events based questions
like answering the number of participating nations in Olympic Games at a specific year and so on. In particular, the system uses
New York Times news corpus composed of articles published between 1987 and 2007 as an underlying dataset and can bothgenerate an answer sentence and provide its supporting sentences.
Keyword Document Archive, Question Answering System, Natural Language Processing, Deep Learning
documents, so that the users don’t need to locate the answer

1. Introduction
Nowadays, the amount and aggregated size of available
digital news archives is increasingly growing as numerous
news articles from newspapers or other media continue tobe added. News articles belong to documents of strongly
temporal character as they usually have publication date
information which determines their timeliness, and on
average constitute fairly reliable, accurate and timealigned information sources. News digital archives offer
immense value to our society, helping users to learn
detailed information of an event that occurred at a specifictime point in the past. Some professions, like historians,sociologists, or journalists need to deal with these temporal
documents a lot.
To satisfy the information need of a user querying
document archive such as news article archive, which is
often

representedasanaturallanguageby themselves. For example, for a question: "How many
countries attend the Summer Olympics in 1996?", the QA
system is expected to return a number like “197”.
There are usually three main modules in traditional QA
systems, which are Query Analysis Module, Document
Retrieval Module, and Answer Extraction Module. Firstly,
the user’s question will be analyzed to determine keywords,
question type, syntactic information, etc. Secondly, thesystem will use keywords to search candidate documents.
Finally, it will use the information from the query analysis
part, and will process the documents to extract ranked
candidate answers. The basic architecture of such a system
is depicted in Figure 1. We also utilize these components
in our proposal.

question,

researchers typically use document retrieval systems likesearch engines that return top-n relevant documents, whichmight contain the answer of user's question. Yet, this styleof output puts unnecessary burden on users who need toread or scan each retrieved documents to obtain the answer,
which is extremely inconvenient.
Question-answering systems can help to solve thisproblem, as the objective of a QA system is to retrieve themost correct answer for the user question. The answer
needs to be precise and concise, rather than a list of

Figure 1. The architecture of traditional question
answeringThis paper presents QAA (Question Answering in

discourse relationships and textual [11]. In recent years,

Archives) system for news article archives. In the

because the generation of large dataset, such as SQuAD

experiments we use New York Times Annotated News[12], MS-MARCO [13], that provide the possibilities tocorpus which contains articles published between 1987 andbuild neural network models. Seo et al. [14] proposed2007 as underlying dataset. Apache Solr engine is

BiDAF model (Bi-Directional Attention Flow) model,

implemented in order to index documents. Our model also

which represents the context at different levels of

has the above three main modules but is implemented using

granularity

diverse techniques. We improve some necessary functions

mechanism to obtain a query-aware context representation,in each module like question classification in Queryand finally get the answer by matching with the query.

Analysis Module by using Neural Networks on a large

Microsoft Asia Natural Language Computing Group [15]dataset that was generated by us, and add some functions

proposed R-Net model, that get the representation by not

like answer sentence generation function that can generate

only question passage matching but also self-passage

a natural language answer. We also add some functions

matching to refine passage representation with information

based on the characteristics of news archives like time-from the whole passage. Recently because of the strong

scoping procedure that can analyze the time of news events

power of Bert model, which is new language representation(if any are mentioned) in user question, so the retrieved

model that built by Google AI Language [16], most models

documents can be more accurate.

proposed combine with it. Chen et al. [17] built a DrQA

and

uses

bidirectional

attentionflow

system, which adds a document retrieval module, and the

2. Related Workperformance drops dramatically when given Wikipedia to

Question answering has been studied for quite a long

read, not a single paragraph. Therefore, the problem oftime, and many question-answering evaluation campaigns
have been held up to date, such as TREC [1], CLEF [2],NTCIR [3] etc. Naturally, multiple question answeringsystems have been designed so far. One of the most famousquestion answering systems was called BASEBALL that
can answer questions about baseball games played over aneural network models is that the dataset they use provides
both the answer and corresponding passage, rather than thewhole corpus, so when given the whole corpus, the modelcannot handle multiple documents effectively and theperformance will drop dramatically. In contrast to theabove-mentioned works, our QAA system is based on aperiod of one year [4]. More specifically, Moriceau et al.

traditional architecture of QA systems, and the answer[5] presented FIDJI, a question-answering (QA) system

extraction module of our system mainly uses traditional

whose main goal is to validate answers by checking that all
the information given in the question is retrieved in the
supporting texts. FIDJI combines syntactic information
with traditional QA techniques and does not require any
pre-processing other than classical search engine indexing.
Nanda et al. [6] built a WebBasedQA by parsing input,
tokenizes and extracts features using previously calculated
results and also using Naive Bayes as a classifier combinedwith known data store. Sun et al. [7] use syntactic
dependency analysis for query expansion, and then use

methods that can extract the evidence from each single
document and rank the candidate answers in statistics.

3. QAA System
QAA system that we propose is a traditional question
answering system though it operates on news article
archive. It also has three main modules as the othertraditional question answering systems, however we extend
it with more functions and also we split the Answer
Extraction Module into three sub-modules: Document
Processing, Sentence Selection and Answer Extraction. The

semantic relation analysis, based on the frame-basedarchitecture of QAA system is shown in Figure 2. In thesemantic representation generated by using a shallow

next parts of the paper, we will use a running example

semantic parser, for semantic answer extraction. Therefore,
there are many different techniques of building traditionalquestion answering systems, such as analyzing syntactic
information like named entity recognition and semantic
information [8,9], translating the natural language question
to a structured query language like SQL, and alsotranslating the of textual sentences into database that canbe retrieved by the structured query [10], analyzingquestion "How many people were killed in Oklahoma City
bombing in 1995?" to better demonstrate our QAA system.

3.1 Question Analysis
Question Analysis Module is the first module of QAA
system, which is the basis of the latter modules. In QAA
system, the Question Analysis Module helps to determine
the expected answer type, generate declarative answer
sentence with assumed answer content, identify named

entity information, extract predicate-argument structure ofdemonstrates the information that can be obtained by QAA

the declarative answer sentence, and detect event timesystem after question analysis step of the question "Howscope in the question if mentioned in the question, andmany people were killed in Oklahoma City bombing in

finally get keywords and their synonyms. Table 1

1995?".

Figure 2. The architecture of QAA System
by Xin et al. [18] that contains nearly 6,000 labelledquestions. This number however is still insufficient to train
a good classifier. Therefore, we constructed our own
dataset with common expected answer types by using thequestion-answer pairs from SQuAD [12] and MS-MARCO[13] datasets by applying the following steps: (1) transformthequestion

intodeclarative

answersentence(thetransformation method will be described later), (2) replace
the interrogative part with the corresponding correct
answer span, (3) use the named entity tool from spaCy
library to detect the named entity type of the answer spanin the answer sentence. The next step is to (4) correct somemisclassified

Table 1 Question Analysis of the question "How many
people were killed in Oklahoma City bombing in
1995?"

questionsandremoverecords

withuncommon or empty named entity types. Finally, (5) we
can obtain 91,145 (question, expected answer type) pairs,that contain 10 common answer types (DATE, PERSON,CARDINAL, GPE, ORG, QUANTITY, TIME, MONEY,

3.1.1 Expected answer type identification

PERCENT, NORP, LOC). An example record is ("WhichFirstly, the Question Analysis Module will determine

theater company was founded by several Northwestern

expected answer type of the question. Expected answer

alumni in 1988? ","GPE"). Then, we use this dataset totype helps to determine the proper type of assumed answer

train a BiLSTM model, that can reach about 82% accuracy.

content of the generated declarative answer sentence andvalidate the extracted answer in the Answer Extraction
Module. However, current available expected answer type

3.1.2 Declarative answer sentence with assumed
answer content generation

datasets are all too small or they do not contain enough

Secondly, Question Analysis Module will generate theexpected answer types. The most popular one was released

declarative answer sentence with assumed answer content,which not only helps to output a clear result, but also

and rank the candidate sentences in Answer Extraction

increases the accuracy of extracting predicate-argument

Module.

structure, since semantic parsers are mainly trained on

3.1.4 Time scope detection

declarative sentences. The main idea of transforming a

Next, the Question Analysis Module will detect event

question to a declarative sentence is to analyze the Penn

time scope in the question to identify the possible temporal

Treebank structure of the question, which can be obtained

information, whose result contains a begin date and an end

by using Stanford Parser. Firstly, the whole question will

date, and we only use the news articles that were published

be labeled as SBARQ in Penn Treebank structure and its

within this period, since news generally contains the mosttwo components: wh-phrase will be labeled as WHADJP,

detailed information about events happened near its

WHAVP, WHNP or WHPP, and main clause will be labeledpublication date. Therefore, it can help to narrow andas SQ. Secondly, the SQ clause will be readjusted using

locate the relevant detailed news documents of a specificdifferent rules that are based on internal Penn Treebank

event. Moreover, the publication date of New York Times

tags of the SQ clause to make each word locate in a correct

corpus that we used as knowledge source is between 1987

order in the declarative sentence. For example, for the SQ

and 2007, so the easiest 'begin' is '1987-01-01' and latest

clause "were killed in Oklahoma City bombing in 1995" in'end' is '2007-12-31'. For example, for a question likeour example question, the SQ clause can be directly"What was the number of assassinations and attempts toreturned because the first part of SQ is a verb and theassassinate in the U.S. since 1865? ", the result of event

second part is a verb phrase. In addition, for the SQ clausetime scope detection is {'begin': '1865-01-01', 'end': '2007-"was Christopher Columbus born" in the question "Where

12-31'}, and only the documents published within thiswas Christopher Columbus born ?", whose internal

particular period will be used. Specifically, the temporal

structure is comprised of verb, noun phrase and verb phrase,

expression phrase that is labeled as "DATE" or "TIME" init needs to be readjusted to "Christopher Columbus wasnamed entity recognition will be extracted firstly, thenborn" by changing the position of verb and noun phrase.

Stanford Temporal Tagger will be used to parse the phrase,

Then the non-interrogative words of wh-phrase will be

and the rules based on preposition information of the

combined with a default text with the same expectedphraseanswer type that detected previously to form a proper

information: '1987-01-01') will also be combined to get theanswer span. For example, "50" is the default text of

result. If the question has multiple temporal expression

CARDINAL answer type, and "New York" is the default

phrases, we will use the date that is the earliest as the

text of GPE answer type, etc. Finally, the proper answer'begin' value, and the date that is the latest as the 'end'

span will be inserted to the modified clause and form thevalue. Moreover, for some expressions that cannot be

declarative answer sentence with assumed answer content.

parsed well by Stanford Temporal Tagger, like the 20th-

3.1.3

Named

entityinformation

andpredicate-argument structure extraction
Thirdly, Question Analysis Module will identify the(e.g.,"before

1999"

has

the

latent'begin'century, between 1999 and 2000, etc., we obtain the event
time scope by applying different processing ways fordifferent expressions.

named entities of the question by named entity tool from

3.1.5 Keywords extraction and synonyms selection

spaCy library. Next, the predicate-argument structure of

Finally,

Question

Analysis

Modulewill

extract

the declarative answer sentence with assumed answer

keywords, that are single-token nouns, compound nouns,

content will be extracted, that is the task of semantic roleverbs and adjectives describing the nouns in the question,labeling. We use a deep BiLSTM model built by He et al.

which can be realized by analyzing the POS tags and[19] to do semantic role labeling, whose performance isdependency

close to the state of the art. As mentioned previously, thesynonyms that have most frequent senses will be selectedsemantic parsing of declarative sentences can get betterby using WordNet, and by further filtering by choosing theresult than the semantic parsing of questions because thesynonym whose POS type match the original word in thesemantic parsers, like semantic role labeling models aresentence and the Glove word embedding vector similarity

mainly trained on declarative sentences rather thanis over 0.5.

information

of

the

words.

Then,

their

questions. The named entity information as well as

3.2 Document Retrieval

predicate-argument information will be used to identify

A s mentioned before, the New York Times news corpus
has been indexed by a search engine Solr, which can easily

index underlying archival documents. We use the keywords

Secondly, Answer Extraction Module will calculate theand their synonyms to generate the query and also use thesimilarity of the semantic information and named entity

time scope information of the event if mentioned in theinformation. We use a_s to denote the question answer

question to return relevant document whose publication

sentence with assumed answer, and use c_s to denote thedate is within this period, otherwise the period isunbounded. Moreover, we only use the top 100 relevant
news documents for a question. Note that time scope
information will not be used in the Answer Extraction

candidate sentence, and we use Pre to denote predicate, Arg

Module to match the named entity information orsimilarity is calculated by the vector similarity:

to denote argument, and W to denote the word list of asentence, ENT to denote the named entity information. To
calculate the semantic similarity, firstly the predicate

predicate-argument structure because it has already been

𝑆𝑖𝑚$%& '𝑃𝑟𝑒+_- , 𝑃𝑟𝑒/_- 0 = 𝑉𝑒𝑐_𝑆𝑖𝑚(𝑃𝑟𝑒+_- , 𝑃𝑟𝑒/_- )used here, corresponding to the publication time of event

Secondly, we use jaccard similarity coefficient to calculate

news.

the similarities between two arguments:

3.3 Answer Extraction

𝑆𝑖𝑚6%7 (𝐴𝑅𝐺𝑋, 𝐴𝑅𝐺𝑌) =Answer Extraction Module uses the information from the|𝑊6?@A ∩ 𝑊6?@C ||𝑊6?@A ∪ 𝑊6?@C |question analysis, processes the candidate documents andBecause there are two pairs of ARG0, ARG1 in twothen matches the information between question andsentences. So we define the argument similarities of a_s,

document sentences to extract candidate answers.

c_s as following by using the above function:

Firstly, the Answer Extraction Module will process the

𝑆𝑖𝑚6%7 '𝐴𝑟𝑔+F , 𝐴𝑟𝑔/F 0 = 𝑆𝑖𝑚6%7 '𝐴𝑅𝐺0+_- , 𝐴𝑅𝐺0/_- 0 ∗ 𝑆𝑖𝑚6%7 '𝐴𝑅𝐺1+_- , 𝐴𝑅𝐺1/_- 0

candidate documents. It firstly concatenates all theThen, when calculating the similarity of the named entity

retrieved relevant documents to a single large document

information, we also use Jaccard similarity coefficient toand then separates them into a list of sentences. Then it

calculate

will extract predicate-argument structure of each sentence

𝑆𝑖𝑚JKL (𝐸𝑁𝑇𝑎_𝑠 , 𝐸𝑁𝑇𝑐_𝑠 ). Next, we calculate the similarity ofby performing semantic role labelling and remove thea_s, c_s as following:

structures whose predicates are not contained in thesynonym of the predicate of the declarative answer

thesimilarity,

that

we

denoteas

𝑆𝑖𝑚(𝑎- , 𝑐- ) = 𝛼 S𝑆𝑖𝑚$%& '𝑃𝑟𝑒+F , 𝑃𝑟𝑒/F 0 ∗ 𝑆𝑖𝑚6%7 '𝐴𝑟𝑔+F , 𝐴𝑟𝑔/F 0T

sentence with assumed answer content. For example, one+(1 − 𝛼)(𝑆𝑖𝑚JKL '𝐸𝑁𝑇+_- , 𝐸𝑁𝑇/_- 0)

sentence of a candidate sentence list is: "Mr. Nichols, 41,

We set 𝛼 = 0.7, and we only extract the top 20 sentences.

has not been implicated in the April 19 bombing that killed

Finally, we extract candidate answers by matching the

167 people and destroyed the Federal Building inargument that contains the assumed answer with the

Oklahoma City. ". Although, this sentence has 5 predicates,

corresponding argument from these sentences. Then we

we only extract the predicate "killed" and its ARG0 andvalidate the answer by expected answer type that is

ARG1, which are "the April 19 bombing" and "167 people".

obtained in the first module, and rank the answers by their

If there would not be any sentence that contain thefrequency and the similarity scores of their sentences.

predicate, we would extract predicate-argument structure

Finally, answer sentence can be generated by replacing theof top 100 sentences that has most common words with theassumed answer with correct answer.

generated answer sentence. After obtaining the predicateargument structure by using semantic role labelling, wethen extract the named entity information of the sentence,

4. Experiments
This section first talks about the generation of the test
set, and then presents evaluations of our QAA system.

and only store the named entity information whose types

To test the performance of our system, we built a test set

are also in the named entity information that we extract inwith question-answer pairs that are selected from SQuAD

Question Analysis Module. If the temporal information of[12]. We manually check the question-answer pairs andthe question, that is labeled as DATE or TIME in named

select the pairs whose questions can be answered by theentity recognition and also used as time scope information

New York Times. In addition, we divide the question-

to locate relevant documents, this type of named entity

answer pairs into two categories: temporal question-answer

information of the sentence will not be extracted. For

pairs are the pairs whose questions do not contain temporal

example, in the above sentence, the extracted named entity

expressions, temporal question-answer pairs are the pairs

information is "Oklahoma City", and "April 19" is not

whose questions contain temporal expressions.

extracted.

Table 2 presents the evaluation results of our QAA[ 1] E.M. Vorhees, The TREC question answering track, Nat.

system. Our system achieves 16.8% exact match and 19.1%

Lang. Eng. 7 (2001) 361–378.

F1 score on the non-temporal questions, and achieves[2] A. Vallin, B. Magnini, D. Giampiccolo, L. Aunimo, C.

20.0% exact match and 22.3% F1 score on the temporal

Ayache, P. Osenova, A. Penas, M. de Rijke, B. Sacaleanu,

questions.

D. Santos, R. Sutcliffe, Overview of the CLEF 2005

With the help of detecting time scope information ofmultilingual question answering track, in: Proceedings oftemporal questions, the document retrieval module can

the Third Cross Language Evaluation Forum (CLEF 2005),

locate more relevant documents of the corresponding

2005.

question. Therefore, the system works better on temporal[3] N. Kando, Overview of the fifth NTCIR workshop, in:questions. After analyzing the questions that cannot be

Proceedings of the Fifth NTCIR Workshop Meetinganswered correctly by our system, we found that most of

Evaluation

them is due to the non-existence of the matching predicate-

Information Retrieval, Question Answering and Cross-argument in the retrieved documents, which makes the

Lingual Information Access, 2005.

system hard to find a good answer candidate. So it is[4] Green BF, Wolf AK, Chomsky C, and Laughery K.

necessary to use other patterns or other syntactic logic

Baseball: An automatic question answerer. In Proceedings

forms to compute the matching score when the systemof Western Computing Conference, Vol. 19, 1961, pp. 219–

cannot determine good predicate-argument structures of224.

the documents sentences. But unlike other end-to-end[5] Veronique Moriceau and Xavier Tannier. 2010. FIDJI:

neural network models that do not have the ability to

Using syntax for validating answers in multiple ´aggregate the answer from relevant documents, our system

documents. Inf. Retr. 13, 5 (October 2010), 507–533.

can compare the evidence from each relevant document and

DOI:http://dx.doi.org/10.1007/s10791-010-9131-ythen return a final answer.

[6] Garima Nanda, Mohit Dua, Krishma Singla, "A Hindi

of

Information

AccessTechnologies:

Question Answering System using Machine Learning
Approach", ICCTICT, 2016.
[7] R. X. Sun, J. J. Jiang, Y. F. Tan, H. Cui, T. S. Chua, M.
Y. Kan. 2005. Using syntactic and semantic relation
analysis in question answering. In Proceedings of the
TREC.

Table 2. Evaluation Results of QAA system[8] Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Deepak
Ravichandran, Chin-Yew Lin, Deepak Ravich, Toward

5. Conclusions & Future Worksemantics-based answer pinpointing, in:Proceedings of the

In this paper, we present QAA system that is based on aFirst

International

Conference

on

Human

Language

traditional architecture of QA systems yet it focuses on

Technology Research, ACL, Stroudsburg, PA, USA, 2001,

long-term document archives. Our system combines manypp. 1–7.9.

techniques, like semantic role labelling, deep learning[9] Enrique Alfonseca, Marco De Boni, Jos-Luis Jara-

method and other NLP techniques. It can help certain users

Valencia,of professions, like historians, sociologists, and journalists

answering system using syntactic and semantic information

to better analyze the news events recorded in news archives

for answer retrieval, in: Proceedings of the 10th Text

to to extract useful desired information. Still the proposed

Retrieval Conference (TREC-10), 2001, pp. 680–686.

system does not work well in some cases, for example, the[10] Anette Frank, Hans-Ulrich Krieger, Feiyu Xu, Hans

performance of semantic role labelling is not high whenUszkoreit, Berthold Crysmann, Brigitte Jörg, Ulrichdealing with long or complex sentences.

Schäfer, Question answering from structured knowledge

In future work, we will build a larger evaluation dataset

SureshManandhar,A

prototypequestionsources, Journal of Applied Logic 5 (1) (2007) 20–48.

for more different kinds of events and also try to combine[11] Sanda Harabagiu, Andrew Hickl, Methods for using

some advanced techniques like Bert model to do thetextual entailment in open-domain question answering, in:comprehension on multiple documents.

Proceedings of the 21st International Conference on
Computational Linguistics and the 44th Annual Meeting ofReferences

the Association for Computational Linguistics, ACL,
Stroudsburg, PA, USA,2006, pp. 905–912.
[12] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang.
Squad: 100,000+ questions for machine comprehension of
text. In Empirical Methods in Natural Language Processing(EMNLP), 2016.
[13] Nguyen, Tri, Rosenberg, Mir, Song, Xia, Gao,
Jianfeng, Tiwary, Saurabh, Majumder, Rangan, and Deng,
Li. Ms marco: A human generated machine reading
comprehension dataset. arXiv preprint arXiv:1611.09268,
2016.
[14] Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, andHannaneh Hajishirzi. Bidirectional attention flow for
machine comprehension. arXiv preprint arXiv:1611.01603 ,
2016.
[15] Microsoft Asia Natural Language Computing Group.
R-net: Machine reading comprehension with self-matching
networks. 2017.
[16] Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K.
2018. Bert: Pre-training of deep bidirectional transformers
for language understanding.
[17] Danqi Chen, Adam Fisch, Jason Weston, and Antoine
Bordes.

Reading

wikipedia

toansweropen-domainquestions. arXiv preprint arXiv:1704.00051, 2017.
[18] Xin Li, Dan Roth,Learning Question Classifiers.
COLING'02, Aug., 2002.
[19] He, L., Lee, K., Lewis, M., and Zettlemoyer, L. (2017).
Deep semantic role labeling: What works and what’s next.
In the Association for Computational Linguistics annual
meeting (ACL).
[20] Pavlos Fafalios, Vaibhav Kasturia, Wolfgang Nejdl,"Towards a Ranking Model for Semantic Layers over
DigitalArchives", Digital

Libraries(JCDL)

2017

ACM/IEEE Joint Conference on, pp. 1-2, 2017.
[21] Helge Holzmann, Wolfgang Nejdl, Avishek Anand,
Exploring Web Archives Through Temporal Anchor Texts,
Proceedings of the 2017 ACM on Web Science Conference,
June 25-28, 2017, Troy, New York, USA