NTCIR-6 2007/05/16 (0) NT NTT’s Question Answering System for NTCIR-6 QAC-4 Ryuichiro Higashinaka Hideki Isozaki NTT Communication Science Labs. NTT Corporation
•  Our two systems: – NCQAW-1 and NCQAW-2 (NTT CS Labs’ QA System for ‘Why’ Questions)
• Features
– Focus on ‘why’ questions – Machine learning approach for ‘why’ and ‘how’ questions – Pattern-based approach for ‘definition’ questions •  Achieve good performance on ‘why’ and ‘definition’ questions
2
• NCQAW-1
• NCQAW-2
3
– ML-based approach for ‘why’ and ‘how’ – Pattern-based approach for ‘definition’ – Uses SAIQA-QAC2 (our factoid QA system) for other question types – Question type analysis is based on rules – Same as NCQAW-1 except that ‘why’ and ‘how’ questions are handled by rules
Improvement by ML-based approach
•NCQAW-1

•NCQAW-2 Could not answer many ‘how’ questions because of question analysis failures
• There are few systems for answering open-domain ‘why’ questions • Previous approach      (sentences having causal expressions) – Extract causal senténces by hand-crafted rules as answer candidates (e.g., using cue words such as ‘tame’, ‘node’ etc.) – Rank the candidates by their similarity to the question •  Systems based on the approach – System by Morooka and Fukumoto (2006) – NCQAW-2
5

•  Hand-crafted rules are costly to make – Cue words are not always reliable •  Only 6-7 % of words before ‘de (by)’ are causes (Abekawa and Okumura, 2004) – Difficult to cover all causal expressions by hand •  Difficult to express degree of causality – Some expressions are more strongly expressing causality than others – ‘no riyuude (by reason of) ’ vs. ‘kara (from)’, ‘tame (for)’, ‘de (by)’, etc.
6

•  Adopt a machine learning approach to learn a causal sentence classifier Document Retrieval Top-N documents All sentences as answer candidates Similarity score between the question and the candidate (NB. This is a sentence extraction task.)
Causality score output by the causal sentence classifier
Similarity score within the candidate) sim'why(C) = 1/(1+ exp(−simwhy(C)) N.B., Similarity score is normalized by the sigmoid function Final score of an answer candidate: candscorewhy(C) = causalwhy(C) + sim'why(C) (0) NTT
•  Use EDR Japanese corpus for training – Has annotation of ‘cause’ relation – 8,064 sentences with ‘cause’ out of 0.2M total sents. – Sentences with ‘cause’ → positive examples without ‘cause’ → negative examples
• Train a classifier by BACT
– In sentence classification tasks, lexical, syntactic, and semantic features are useful → Adopt tree feature representation of a sentence → BACT: a boosting algorithm for classifying trees (uses existence of sub-trees as weak learners)
9

(Rule-based: our implementation of Morooka and Fukumoto's causal sentence extraction rules) •Higher precision achieved by BACT •Precision may be important from NCQAW-1’results •F can be raised to ca. 25% by feature engineering
α : Weights given to each pattern
•  NCQAW-2 extracts sentences with a strong cue – QAC4-00030: What is the purpose of the green power marketing? – A:さらに、原子力や石油への依存度を少しでも下げるため、電源の多様化を図るのは時代の流れだろう。 •  NCQAW-1 can find answers without such cues – QAC4-00026: Why is the movable weir needed in the Yoshino river? – A:可動堰化計画は、第十堰の老朽化が進んで洪水時の障害になり、壊れる危険もある、などとして持ち上がった。
13

• Same process as ‘why’ questions • ‘condition’ relation is used instead of ‘cause’ to train a sentence classifier – sentences having ‘condition’ may have answers within the sentence • Q:聖火が消えたらどうしますか？ (How do we cope when the Olympic flame burns out?) • A:聖火が消えたら再点火します. (If the Olympic flame burns out, it is reignited.) Due to question type analysis failures, most of ‘how’ questions were not answered at all
14

•  There are many systems for answering ‘definition’ questions (e.g., in TREC) •  Common approach – Extract descriptive sentences/nuggets as answer candidates using patterns or ML – Rank the candidates using importance of keywords within the candidate (i.e., the more keywords, the better) •  We adopt a similar approach
15

•   Use patterns to find descriptive phrases ● Perform dependency tree matching to obtain phrases with all their modifiers Matched phrases •投松坂大輔・[躍る春]第７０回記念センバツ選手編松坂大輔投手・エースの松坂大輔投手・ＰＬ学園(大阪)・稲田学投手、横浜(神奈川)・松坂大輔投手・最速１５０キロの速球にプロも注目する横浜（神奈川）のエース・松坂大輔君Dependency tree matching Tgrep2 Daisuke Matsuzaka e.g.,松坂大輔Generated patterns松坂大輔「 X」（松坂大輔）松坂大輔（X）松坂大輔というX松坂大輔といったX松坂大輔以外のX松坂大輔などのX松坂大輔等のX松坂大輔のようなX松坂大輔の様なX松坂大輔というとX松坂大輔といえばX松坂大輔と言えばX Convert patterns into dependency trees Pattern Matcher All matched phrases as answer candidates
•  Answer candidates are ranked based on importance of words within the candidates • Importance of words: wordscoredef(w) = log(tf(w; {Ci})) (Term frequency within all answer candidates) •  Score of an answer candidate: candscoredef (C) = wordscoredef ( W ) wECW(C) (Sum of wordscores within the answer candidate)
•   QAC4-00018:スケルトンとはどのような競技ですか。 (What is Skelton, the competitive sport?) –  A: 54年ぶりに冬季五輪の正式種目として復活する、そり競技のスケルトン。 –  A:うつぶせでソリに乗ってボブスレーやリュージュと同じコースを滑走する氷上競技「スケルトン」 • QAC4-00034:「スポット傍受」とはどういったものですか。 (What is the spot wiretapping?) –  A:通話内容が犯罪と関係あるか試し聴きする「スポット傍受」 – A:会話の冒頭(ぼうとう)を試験的に傍受する「スポット傍受」 •Adnominal/adverbial phrases by including modifiers •Achieving good performance (9 A-rated answers/24)
Improvement by ML-based approach
by finding adnominal/ adverbial phrases by dep. tree matching Could not answer many ‘how’ questions because of question analysis failures

•NCQAW-1
20
•  Our two systems: – NCQAW-1 and NCQAW-2 • Features – Focus on ‘why’ questions – Machine learning approach for ‘why’ and ‘how’ questions – Pattern-based approach for ‘definition’ questions •  Achieved good performance on ‘why’ and ‘definition’ questions