IJCNLP 2008 10th Jan.

Corpus-based Question
Answering for why-Questions
Ryuichiro Higashinaka
andHideki Isozaki
NTT Communication Science Labs.
1

Overview
 Corpus-based approach for whyQA
 Use relation-annotated corpora toautomatically acquire causal expression
patterns
 Use a corpus of why-questions and answers
to train an answer-candidate ranker
 Implemented system: NAZEQA Experimental results

2

WhyQA
 Answering generic “Why X?” questions byextracting causes from a text archive E.g., Why do ostriches run fast?

 Few implemented systems
 Previous approach (Fukumoto, 2007)
 Uses hand-crafted patterns to extractcause-bearing passages
 Patterns rely on explicit causal cue words
e.g., tame, node, de, kara (because/by/from)3

Problem Hard to cover causal expressions by hand
 Causes are expressed by a wide variety ofexpressions
 Half of causes are not marked by cue words(Inui and Okumura, 2005)
 Cue words are not always reliable
 Only 6-7 % of“de” (by) trigger causes(Abekawa and Okumura, 2004)

 Hand-crafted patterns are costly to make4

Approach Automatically acquire causal expression

patterns from relation-annotated corpora
 FrameNet, PropBank, EDR Corpus
 Typically annotated with a causal relation

 Use the acquired patterns to create

features to represent answer candidates
 Train an answer-candidate ranker thatranks answer candidates on the basis ofthe features5

Approach (cont’d)
Question
RelationAnnotatedCorporaAutomaticAcquisition
Causal Expression
Patterns

Text
archive

IR Engine

AnswerCandidates

Feature Extraction
FeaturesAnswer candidate
rankerRanked Answer
Candidatesg
TraininWhyQAcorpus

A set of why-questions
and answers
6Implemented System: NAZEQA A Japanese whyQA System Uses the EDR corpus as a relation

annotated corpus to acquire causal
expression patterns
 Derives three types of features for answer
candidates
 Uses Ranking SVM for ranker training Uses a manually constructed WhyQA
corpus (WHYQA Collection)7

Acquiring Causal Expression
Patterns from the EDR Corpus The EDR Corpus
 A collection of Japanese sentences fromvarious sources Provides a semantic representation foreach sentence Specifies relations of content words
 Relations include a causal relation

 8,474 text spans annotated with acausal relation in 207,802 sentences8

Acquiring Causal Expression
Patterns from the EDR Corpus
 Extract text spans annotated with a causal

relation as causal expressions
 Convert the expressions into patterns by Leaving only functional words Auxiliary verbs, case, aspect, tense markers

 Replacing others with wildcards “*”Kare ha sagi de taiho sareta (He was arrested for fraud)
Causal Expression

sagi de (for fraud)

Causal Expression Pattern* de (for *)9Acquired Causal Expression
Patterns from the EDR Corpus
 Obtained 394 distinct causal expression

patterns
 Examples
 de (by/for) tame (because) niyoru, niyotte (because of) no (-GEN) * wa (topic marker)
 de (by/for) * wo (-ACC) * teshimai (-PERF)
 koto niyotte (by the fact that)
10Feature Extraction
 For an answer candidate to be the correct

answer, it should have a causal expression
 be similar to the question in content
 show some causal relation to the question

 Three types of features to represent howeach candidate satisfies each condition
 Causal Expression Features
 Content Similarity Features
 Causal Relation Features

Based on theacquiredpatterns
11Causal Expression Features
 AUTO-Causal Expression Features
 394 binary features representing if eachacquired pattern matches the answercandidate
 MAN-Causal Expression Feature A binary feature representing if the answer

candidate is matched by existing hand-crafted
patterns (Fukumoto, 2007)

12

Content Similarity Features
 Question-Cand Cosine Similarity feature Cosine similarity of word frequency vectors Document-Question Relevance feature Similarity between the question and thedocument where the candidate is found Inverse rank of the retrieved document
 Synonym Pair feature A binary feature indicating if synonyms arefound in the question and answer candidate
13

Causal Relation Features
 Cause-Effect Pair feature A binary feature representing if a cause-effectword pair is found in the answer candidate
and the question Cause-effect word pairs
 Explode Î Die

Q: Why was John arrested?

 Murder ÎArrest Fraud Î Arrest

Cand: John was arrested for fraud.

 Generated from theEDR concept dictionary

Cause-Effect pair feature = 1
14

Q: Why was John arrested?

Cand: John was arrested for suspicion of fraud.

CausalExpression
Patterns

Pattern1 for * ofPattern2 by * ofPattern3 due toPattern394 because

Matches pattern1 1
Matches pattern2 0
Matches pattern3 0
Matches pattern394 0

Causal Expression Features

Hand-crafted patterns(Fukumoto, 2007)

Matches hand-crafted
patterns 0EDR
CorpusEDR
Concept
Dictionary

EDR
WordDictionary

Question-Candidate Cosine Similarity

0.3

Document-Question Relevance

0.1Thesaurus

Cause-Effectword pairs

Examplesarrest Ù apprehendarrest Ù capturearrest Ù nip
Examples
explode Î diemurder Î arrest
fraud Î arrest

Q and Cand havesynonyms 1
Content Similarity Features
Causal Relation Feature
Q and Cand have
A cause-effect wordpair 115

WHYQA Collection
 1,000 sets of why-questions and answers
 Procedure Create a why-question
 Retrieve top-20 documents for the question Extract all sentences in the top-20 documents Create sentence-level answers by selecting

sentences that contain answers Create paragraph-level answers by selecting
paragraphs that contain answer sentences16

Training an
Answer-candidate Ranker
 Ranking SVM (Joachims, 2002)
 Learns ranking by minimizing pairwise ranking

error Training data
 WHYQA Collection
 Train a ranker so that answers (e.g.,sentences selected as answers) get ranked
higher than non-answers
 Trained answer-candidate rankers forsentence and paragraph-levels
17Experiment Compare NAZEQA with two baselines
 Baseline-1 (COS) Uses cosine similarity to rank answer candidates

 Baseline-2 (FK) (Fukumoto, 2007)
 Selects answer candidates by hand-crafted patterns
 Ranks the candidates by cosine similarity

 NAZEQA and baselines process the sameanswer candidates All sentences/paragraphs in top-20 documents

retrieved by an IR engine
18

Experiment (cont’d) Question set
 All questions in the WHYQA Collection
 10-fold cross validation to evaluate NAZEQA

 Evaluation criteria
 Mean Reciprocal Rank (MRR) Coverage(Rate of questions correctly answered by
top-N answers)19Results
MRR
Top-N

COS

FK

Coverage
NAZEQACOSFK

NAZEQA

Sentences as answer candidates
Top-1

0.036

0.091++

0.113

3.6%

9.1%

11.3%

Top-5

0.086

0.139++

0.196**

19.1%

23.1%

35.4%

Top-100.102

0.149++

0.216**

31.3%

30.7%

50.4%

Top-20

0.115

0.152

0.227**

51.4%

35.5%

66.6%

Paragraphs as answer candidates
Top-1

0.065

0.152++

0.186*

6.5%

15.2%

18.6%

Top-5

0.140

0.245++

0.305**

29.2%

41.6%

53.1%

Top-100.166

0.257++

0.328**

48.8%

50.5%

70.3%Top-20

0.181

0.262++

0.339**

70.7%

56.4%

85.6%

** (p<0.01) * (p<0.05) NAZEQA’s statistical significance over FK++ (p<0.01) FK’s statistical significance over COS
20

Results
MRR
Top-N

COS

FK

Coverage
NAZEQACOSFK

NAZEQA

Sentences as answer candidates
Top-1

0.036

0.091++

0.113

3.6%

9.1%

11.3%

Top-5

0.086

0.139++

0.196**

19.1%

23.1%

35.4%

Top-100.102

0.149++

0.216**

31.3%

30.7%

50.4%

Top-20

0.115

0.152

0.227**

51.4%

35.5%

66.6%

Paragraphs as answer candidates
Top-1

0.065

0.152++

0.186*

6.5%

15.2%

18.6%

Top-5

0.140

0.245++

0.305**

29.2%

41.6%

53.1%

Top-100.166

0.257++

0.328**

48.8%

50.5%

70.3%

Top-200.181

0.262++

0.339**

70.7%

56.4%

85.6%

NAZEQA significantly outperforms FK21

Results
MRR
Top-N

COS

FK

Coverage
NAZEQACOSFK

NAZEQA

Sentences as answer candidates
Top-1

0.036

0.091++

0.113

3.6%

9.1%

11.3%

Top-5

0.086

0.139++

0.196**

19.1%

23.1%

35.4%

Top-100.102

0.149++

0.216**

31.3%30.7%

50.4%

Top-20

0.115

0.152

0.227**

51.4%

35.5%

66.6%

Paragraphs as answer candidates
Top-1

0.065

0.152++

0.186*

6.5%

15.2%

18.6%

Top-5

0.140

0.245++

0.305**

29.2%

41.6%

53.1%

Top-100.166

0.257++

0.328**

48.8%

50.5%

70.3%

Top-200.181

0.262++

0.339**

70.7%

56.4%

85.6%

High coverage achieved by NAZEQA

22

Results
MRR
Top-N

COS

FK

Coverage
NAZEQACOSFKNAZEQA

Sentences as answer candidates
Top-1

0.036

0.091++

0.113

3.6%

9.1%

11.3%

Top-50.086

0.139++

0.196**

19.1%

23.1%

35.4%

0.102 coverage
0.149++
0.216**•LowofFK
Top-200.115
0.152
0.227**•Possible
limitationof
Paragraphs as answer candidates
hand-crafted patterns

31.3%

30.7%

50.4%

51.4%

35.5%

66.6%

Top-1

0.065

0.152++

0.186*

6.5%

15.2%

18.6%

Top-5

0.140

0.245++

0.305**

29.2%

41.6%

53.1%

Top-100.166

0.257++

0.328**

48.8%

50.5%

70.3%

Top-200.181

0.262++

0.339**

70.7%

56.4%

85.6%

Top-10High coverage achieved by NAZEQA

23Impact of features
Feature Set

Sentence(Top-5 MRR)

Paragraph(Top-5 MRR)All features

0.181
0.138**

0.287
0.217**

0.179
0.131**

0.286
0.188**

w/o Synonym Pair

0.161
0.180

0.275
0.282

w/o Cause-Effect Pair

0.184

0.287

w/o AUTO-Causal Expression
w/o MAN-Causal Expression
w/o Question-Candidate Cosine Similarity
w/o Document-Question Relevance

MRR drops significantly without AUTO-Causal expression
features Î Verifies effectiveness of automatically acquiredcausal expression patterns
24

Weights of features assigned byRanking SVM

25

Weights of features assigned byRanking SVM•Many complex patterns were found to be important•Difficult to cover such patterns by handÎ shows effectiveness of an automatic approach
26

Conclusion
 Corpus-based approach for whyQA
 Use relation-annotated corpora to acquire causal

expression patterns
 Train an answer-candidate ranker using a corpus of
why-questions and answers
 Experimental results show validity of ourapproach Future work
 Use other relations (e.g., purpose) to increase

coverage of causal expressions
 Incorporate syntactic and semantic features
27