Extracting Causal Knowledge Using
Clue Phrases and Syntactic Patterns
Hiroki Sakaji1 , Satoshi Sekine2 , and Shigeru Masuyama1
1

Toyohashi University of Technology, 1-1 Hibarigaoka, Tempaku-cho, Toyohashi-shi,
Aichi 441-8580, Japansakaji@smlab.tutkie.tut.ac.jp, masuyama@tutkie.tut.ac.jp
2
NewYork University, 715 Broadway, 7th ﬂoor, New York, NY 10003 USA
sekine@cs.nyu.edu

Abstract. This paper proposes a method to extract causal knowledge(cause and eﬀect relations) using clue phrases and syntactic patterns
from Japanese newspaper articles concerning economic trends. For example, a sentence fragment “World economy recession due to the subprime loan crisis ...” contains causal knowledge in which “World economy
recession” is an eﬀect phrase and “the subprime loan crisis” is its cause
phrase. These relations are found by clue phrases, such as “ため (tame:because)” and “により (niyori: due to)”. We, ﬁrst, investigated newspaper corpus by annotating causal knowledge and clue phrases. We foundthat some speciﬁc syntactic patterns are useful to improve accuracy to
extract causal knowledge. Finally, we developed our system using the cluephrases and the syntactic patterns and showed the evaluation results ona large corpus.

1

Introduction

A large amount of machine-readable textual documents including Web pages
and newspaper articles are now available. We can ﬁnd a lot of valuable information for many real applications in the documents by text mining. One of suchinformation is “causal knowledge”. We expect causal knowledge in the economic
domain would be useful to forecast economic trends and prevent loss of business
opportunities. For example, if we get causal knowledge where cause is “the Year
2000 problem” and eﬀect is “the decline of the sales of hotels”, we infer thatproblems like “the Year 2000 problem” may cause the decline of the sales ofhotels. (Note: Hotels heavily rely on IT technologies. However, hotel managers
are very busy handling daily business, and the hotel industry was not ready forsuch a problem until the last minutes.) Nevertheless, extracting such knowledge
costs prohibitively high and time-consuming. Thus, there are some attempts to
extract causal knowledge from textual documents automatically [1–4].
We propose a method to extract causal knowledge from Japanese newspaper articles concerning economic trends using syntactic patterns. For example,“World economy recession due to the subprime loan crisis ...”, here, “World economy recession” is an eﬀect phrase, “the subprime loan crisis” is its cause phraseand “due to” is a clue phrase, respectively. These cause and eﬀect relations are
explicitly expressed by the clue phrase “により (niyori: due to)”. However, if wejust use the clue phrase to extract causal knowledge, it extracts a lot of noises.
By investigating the corpus, we found that there are some syntactic patterns
in cause-eﬀect phrases and these patterns are useful to improve the accuracyof the results. Furthermore, causal knowledge is sometimes expressed in morethan one sentence. Our method can also handle causal knowledge stridden overtwo consecutive sentences. Finally, we developed our system to extract causal
knowledge using the clue phrases and the syntactic patterns and evaluated on alarge corpus.

2

Related WorkA lot of work has been done on causal information extraction from a large
corpus. Inui et al. proposed a method for acquiring causal relations (cause, eﬀect,
precond and means) from a complex sentence containing a Japanese resultative
connective “ため (tame: because)” [1]. The Japanese resultative connective “ため (tame: because)” is a strong clue for causal information. In their research, cluephrases other than “ため (tame: because)” are not used, therefore, their method
can not extract causal relations expressed by other clue phrases. In contrast, ourmethod can extract many causal relations expressed by 36 clue phrases.
Khoo et al. proposed a method for extracting cause-eﬀect information fromnewspaper articles by applying patterns made manually[2]. Furthermore, they
proposed a method for extracting causal knowledge from a medical database by
applying graphical patterns[3]. However, in their research, both cause and eﬀectneed to be contained together in the same sentence. Thus, these methods are notable to extract causal knowledge stridden over two sentences, while, our method
can extract causal knowledge in such a case.
Chang et al. proposed a method for extracting causal relations that exist
between noun phrases using clue phrases and word pair probabilities[5]. This
probability was deﬁned as the probability of a causal noun phrase pair. They
used a bootstrapping method for learning Naive Bayes causality classiﬁer. Girju
proposed a method for automatic detection and extraction of causal relations
based on clue phrases[4]. In their paper, the causal relation is expressed by apair of noun phrases. Girju used WordNet as semantic constraints for selecting
candidate pairs. Hence, her method can not extract unknown phrases that are
not in WordNet. In contrast, our method deals with causal knowledge expressedby not only noun phrases but also verb phrases and sentences. In addition, ourmethod can extract unknown phrases.
Sakai et al. proposed a method for extracting cause information from Japaneseﬁnancial articles concerning business performance[6]. Their method only extracts
cause phrases. On the other hand, our method can also extract eﬀect phrases.

Table 1. List of tags

Tag
CLUE

Description
Examplesclue phrase ため (tame: because), から (kara: from)
株式市場が下落した (kabushiki sijyou ga
CAUSE VP
verb cause phrase
gerakushita: stock market declined)
景気の回復 (keiki no kaihuku: recovery
CAUSE NPnoun cause phraseof economy)
世界不況 (sekai hukyou: world economy
EFFECTeﬀect phrase
recession)
農産物価格は、(nousanbutsu kakaku ha:
EFFECT SBJ
subject of eﬀect phrase
agricultural price)
EFFECT PRED
predicate of eﬀect phrase
下落した (geraku shita: fell oﬀ)
word which changes
INVの (no: of), こと (koto: thing)verb/noun of cause phrase

3

Investigation of Clue Phrases

We investigate 300 newspaper articles concerning economic trends using clues
for extracting appropriate causal knowledge. We expect that newspaper articles
concerning economic trends contain a large amount of causal knowledge. Newspaper articles concerning economic trends are acquired by Sakaji’s[7] methodfrom the Nikkei newspaper published from 1990 to 2005.
3.1

Tagging rules

Newspaper articles concerning economic trends are annotated with the following
tags (in Table 1) to investigate how causal knowledge and the clue phrases
are expressed. A human annotates tags to causal knowledge that is expressed
explicitly by a fact and its account in a sentence or two adjacent sentences. We
have two notes about the deﬁnition. In Japanese, the subject and the predicate
can be expressed separately in a sentence. In such a case, the subject of an eﬀectphrase is annotated with “EFFECT SBJ” and the predicate of an eﬀect phrase
is annotated with “EFFECT PRED”. Otherwise, the eﬀect phrase is annotatedwith “EFFECT”.
We found that speciﬁc clue phrases are used depending on the type of thecause phrase. For example, clue phrase “ため (tame: because)” is used only when
the cause phrase is a verb phrase. However, if the cause is a predicate, a special
word, such as “の (no: of)” or “こと (koto: thing)” is used to convert from a verb
to a noun. We annotated “INV” tag to such words.
3.2

Results of Tagging

Inui et al. also reported an investigation of clue phrases[8] on the social domainof a general newspaper, the Mainichi newspaper. Table 2 shows the number ofTable 2. The number of investigated articles, the total number of clue phrases, theset of clue phrases

inui’s investigation
our investigation

Num. of articles Total num. of clue phrases Set of clue phrases
750
219
34
300
695
154

Table 3. The number of phrases used as clue phrases(Frequency), the total number ofphrases(Sum), the ratio of phrases used as clue phrases(Rate)
Clue phrase
Frequency Sum Rate
で (de: by)
155 2385 0.065
による (ni yoru: by)
48 244 0.197
で、(de: by)
48 340 0.141
から (kara: from)46 646 0.071
ため、(tame: because)
30 55 0.545を背景に (wo haikei ni: behind)
24 26 0.923から、(kara: from)17 46 0.367

investigated articles, the total number of clue phrases and the set of clue phrasesof the investigation by Inui’s and ours. It shows that our investigation foundmore clue phrases per article than Inui’s. We believe that the articles concerning
economic trends have more causal knowledge than that of the social domain,
though minor diﬀerences of deﬁnitions may cause some diﬀerences. Clue phrases
with the seven highest frequencies found by our investigation are shown in Table
3.
In Table 3, clue phrases such as “で (de: by)“, “による (ni yoru: by)” and“から (kara: from)” frequently appear in articles. However, these clue phrases’
ratio between the frequency that the phrase is used as a clue phrase comparedwith the total frequency of the phrase in the corpus is very low. Therefore, it
is not appropriate to use them in our automatic causal knowledge extraction
system. We deﬁne the following score using the frequency and the ratio, in orderto accurately extract causal knowledge.
Score(ti ) = log(T F (ti )) × R(ti )(1)Here, T F (ti ) is frequency that phrase ti is used as a clue phrase. R(ti ) is a ratiothat phrase ti is used as a clue phrase. Clue phrases with the ten highest scoreare shown in Table 4.

4

Extraction of Causal Knowledge

From our investigation, we found that most of the cause phrases appear before
clue phrases in a sentence. On the other hand, eﬀect phrases appear in various

Table 4. Score of clue phrases

Clue phrase
Frequency Sum Rate Score
を背景に (wo haikei ni: behind)
24 26 0.923 1.274を背景に、(wo haikei ni: behind)
10 10 1.000 1.000を受け、(wo uke: under)
12 14 0.857 0.925を挙げる (wo ageru: quote)10 12 0.833 0.833
ため、(tame: because)
30 55 0.545 0.806に伴う (ni tomonau: with)
14 23 0.609 0.698に伴い、(ni tomonau: with)
6
7 0.857 0.667を反映して (wo haneishite: reﬂect)6
7 0.857 0.667に加え、(ni kuwae: besides)
9 13 0.692 0.661

Fig. 2. An example of Pattern B

locations in a sentence. Furthermore, we found that clue phrases have four typesof syntactic roles in indicating causal knowledge. Considering these observations
we propose a method based on syntactic information. That is, we categorize
patterns of cause-eﬀect phrases by the location of the phrases and syntacticroles of clue phrases into four patterns as follows. These four patterns cover 86%
of the cause-eﬀect phrases in the corpus that we examined.
Pattern A: both a predicate and a subject are eﬀect phrases in a sentence (seeFig. 1). The clue phrase has a role of connecting a predicate of an eﬀectphrase and a subject of an eﬀect phrase with a cause phrase.
Pattern B: an eﬀect phrase appears after a cause phrase in a sentence (see Fig.
2). The clue phrase has a role of connecting an eﬀect phrase with a cause
phrase.
Pattern C: an eﬀect phrase is the sentence just before a sentence including aclue phrase (see Fig. 3). The clue phrase has a role of indicating a previous
sentence that is a cause phrase.
Pattern D: an eﬀect phrase appears before a cause phrase in a sentence (seeFig. 4). The clue phrase has a role that is modiﬁed by an eﬀect phrase anda cause phrase.
Patterns A, B and D extract causal knowledge from one sentence. On the other
hand, Pattern D extracts causal knowledge from two adjacent sentences.
In the following three subsections, we will explain how we identify the patterns and how to extract cause-eﬀect phrases using syntactic information.

Fig. 3. An example of Pattern C

Fig. 4. An example of Pattern D

4.1

Parser

The sentences that include clue phrases are parsed by a Japanese dependency
analyzer, Cabocha3 . For example,
サブプライムローンの危機により、世界不況が起こった。
(World economy recession was caused by the subprime loan crisis.)

The parsed sentence is shown in Fig. 5. Here, a bunsetu is a basic block inJapanese composed of several words. The sequence of bunsetus in a sentence areexpressed as (b1 , b2 , ..., bn ), a subscript of b is a bunsetsu’s number. Bunsetus
are assigned consecutive numbers, beginning from one, in ascending order fromthe beginning of a sentence. For example, in Fig. 5, “サブプライムローンの(sabupuraimuro-n no: the subprime loan)” has a number 1.
4.2

Pattern Identiﬁcation

Here, we deﬁne the core bunsetu as the bunsetu that is rearmost bunsetus composing a clue phrase. In addition, we deﬁne the base point bunsetu as the bunsetu
modiﬁed by the core bunsetu (see Fig. 5). Our method searches causal knowledge
and identiﬁes an appropriate syntactic pattern.
Step 1: Search a sentence including clue phrases.
Step 2: If the clue phrase includes a punctuation “。” or a punctuation “。”
appears after the clue phrase, go to Step 4. Otherwise, go to Step 3.
3

http://chasen.org/˜taku/software/cabocha/

Fig. 5. An example of a parsed sentence

Step 3: If the base point bunsetu is a verb phrase and a bunsetu that modiﬁes
the base point bunsetu includes a particle, Pattern A is chosen. Otherwise,
Pattern B is chosen. Go to step 5.
Step 4: If a bunsetu that modiﬁes a core bunsetu includes a particle, Pattern D
is chosen. Otherwise, Pattern C is chosen.
Step 5: Stop.
4.3

Cause Eﬀect Phrase Extraction

Each pattern consists of a “cause phrase extraction process” and an “eﬀectphrase extraction process”. Each of these processes has a string variable CAN(an abbreviation of a candidate) and a numeric variable M . CAN is assigned“null” and M is assigned 0 as an initial value, respectively. First, we will describe
the cause extraction process for all patterns, then we will describe the eﬀect
extraction process.
Cause Extraction
Patterns A and B
Step 1: Search a core bunsetu in the parsed sentence. The core bunsetu’s number
is assigned to M .
Step 2: If CAN is “null”, CAN is assigned bM −1 except for a particle, else
connect bM −1 with CAN, and assign it to CAN. M is assigned M − 1.
Step 3: Repeat Step 2 until M − 1 becomes 0 or bM −1 modiﬁes a phrase thathas a number larger than the core bunsetu’s number.
Step 4: Acquire CAN as a cause phrase.
□
Patterns C and D
Step 1: Search a core bunsetu in the parsed sentence. The core bunsetu’s number
is assigned to M .
Step 2: If CAN is “null”, CAN is assigned bM −1 except for a particle, else
connect bM −1 with CAN, and assign it to CAN. M is assigned M − 1.
Step 3: Repeat Step 2 until M − 1 becomes 0 or bM −1 modiﬁes the core bunsetu and bM −1 includes a syndetic particle(kakari joshi) or a conjunctive
particle(setsuzoku joshi).
Step 4: Acquire CAN as a cause phrase.
□

Eﬀect Extraction
Pattern A
The procedure to extract the predicate of an eﬀect phrase:
Step 1: Search a base point bunsetu from the parsed sentence and assign it to
CAN. The base point bunsetu’s number is assigned to M .
Step 2: Connect bM −1 with CAN, and assign it to CAN. M is assigned M − 1.
Step 3: Repeat Step 2 until M − 1 becomes 0 or bM −1 becomes a core bunsetu.
Step 4: Acquire CAN as a predicate of an eﬀect phrase.
□
The procedure to extract the subject of an eﬀect phrase:
Step 1: Search a bunsetu that modiﬁes a base point bunsetu and includes asyndetic particle(kakari joshi) or a case particle(kaku joshi) from the parsed
sentence. Assign this bunsetu to CAN. The bunsetu’s number is assigned toM.
Step 2: If bM −1 modiﬁes bM , connect bM −1 with CAN, and assign it to CAN,
else go to Step 4. M is assigned M − 1.
Step 3: Repeat Step 2 until M − 1 becomes 0.
Step 4: Acquire CAN as a subject of an eﬀect phrase.
□
Fig. 1 illustrates an example of extracting causal knowledge by Pattern A. When
extracting a predicate of an eﬀect phrase, ﬁrst, CAN is assigned a base pointbunsetu “ある (aru: is)”. M is assigned a base point bunsetu’s number 9. Next,
connect b8 with CAN, and assign “状況にある (jyoukyou ni aru: is in a situation)” to CAN. M is assigned 8. Repeat this process, until bM −1 becomes a corebunsetu “反映し、(han ei shi: reﬂecting)”. Then, CAN “依然厳しい状況にある(izen kibishii jyoukyou ni aru: is still in a diﬃcult situation)” is extracted as apredicate of an eﬀect phrase.
When extracting a subject of an eﬀect phrase, ﬁrst, CAN is assigned a bunsetu “収益は、(syuueiki ha: beneﬁts)” that modiﬁes a base point bunsetu “ある (aru: is)” and includes a syndetic particle(kakari joshi) “は (ha)”. M is assigned this bunsetu’s number 2. Next, connect b1 with CAN “収益は、(syuueiki
ha: beneﬁts)”, and assign “売り上げと収益は、(uriage to syuueki ha: sales andbeneﬁts)” to CAN. Because b1 modiﬁes b2 . M is assigned 1. Then, M − 1 becomes 0. Therefore, CAN “売り上げと収益は、(uriage to syuueki ha: sales andbeneﬁts)” is extracted as a subject of an eﬀect phrase.
When extracting a cause phrase, ﬁrst, M is assigned a core bunsetu’s number
5. Assign b4 except for particles “停滞 (teitai: stagnation)” to CAN because
CAN is “null”. M is assigned 3. Next, connect b3 “需要の (jyuyou no: demand)”with CAN “停滞 (teitai: stagnation)”, and assign “需要の停滞 (jyuyou no teitai:
stagnation of demand)” to CAN. M is assigned 2. CAN “需要の停滞 (jyuyou noteitai: stagnation of demand)” is acquired as a cause phrase, because b2 modiﬁes“ある (aru: is)” that has a number larger than the core bunsetu’s number.

Fig. 1. An example of Pattern A

Pattern B
Pattern B extracts eﬀect phrases in a manner similar to Pattern A’s extraction of predicates of eﬀect phrases, except the base point bunsetu is a nounphrase. If a base point bunsetu is a noun phrase, eﬀect phrases are extracted ina manner described below:
Step 1: Search a base point bunsetu from the parsed sentence and assign it to
CAN. The base point bunsetu’s number is assigned to M .
Step 2: If bM +1 is modiﬁed by a bunsetu that has a number smaller than a corebunsetu’s number, connect CAN with bM +1 , and assign it to CAN, else go
to Step 4. M is assigned M + 1.
Step 3: Repeat Step 2 until bM becomes the rearmost bunsetu in the sentence.
Step 4: Acquire CAN except for a terminal particle as an eﬀect phrase.
□
Pattern C
The sentence just before the sentence including a clue phrase is extracted as
an eﬀect phrase.
Pattern D
The eﬀect phrases are extracted in a manner described below:

Table 5. Extraction results
Recall Precision F-Measure
Cause phrase 0.917
0.757
0.829
Eﬀect phrase 0.809
0.526
0.638
Both
0.770
0.449
0.567

Step 1: Search a bunsetu that modiﬁes a core bunsetu and includes a syndetic
particle(kakari joshi). Assign this bunsetu except for a particle and punctuation to CAN. The bunsetu’s number is assigned to M .
Step 2: If bM −1 modiﬁes bM , connect bM −1 with CAN, and assign it to CAN,
else go to Step 4. M is assigned M − 1.
Step 3: Repeat Step 2 until M − 1 becomes 0.
Step 4: Acquire CAN as an eﬀect phrase.
□5

Evaluation

In this section, we evaluate our method. A set of 200 new articles concerning
economic trends are used for the evaluation. The clue phrases we used are thehighest scored 40 clue phrases in Section 3.2 except for “で、(de: by)”, “ほか、
(hoka: except)”, “ため (tame: because)” and “による (ni yoru: by)”. Then werun the system.
First, the system output is evaluated automatically by matching the output with the human made answers. However, minor diﬀerences are sometimes
acceptable as correct, a human evaluates again when there is a diﬀerence. The
total evaluation results for cause and eﬀect are presented in Table 5. Table 6
shows the accuracy for each clue phrases with more than 5 frequency. Recall is
the percentage of correctly extracted phrases out of the phrases found by the annotator. Precision is the percentage of the correctly extracted phrases out of allphrases found by the system. F-Measure is a combination of recall and precisionequally weighted, and is calculated by the following formula 2.
F -M easure = 2 ∗ precision ∗ recall/(precision + recall)(2)

F-Measure attains a high value only when both recall and precision are high.
As shown in Table 6, for clue phrases “に加え、(ni kuwae: besides)”, “の効果が (no kouka ga: eﬀect)” and “の影響が (no eikyou ga: inﬂuenced by)” no eﬀectphrases are extracted. The F-measure for the cause and eﬀect phrases are 0.829
and 0.639, respectively. We believe that the result of cause phrase extraction isgood for a causal knowledge extraction task. Therefore, we consider that cause
phrase extraction is suﬃcient for a real-life application. However, the result ofeﬀect phrase extraction is not satisfactory.

Table 6. Precision for each cause and eﬀect phrases

Clue phrase

num. of
Precisionphrases Cause phrase Eﬀect phraseを背景に (wo haikei ni: behind)(B)
14
0.923
0.923
ため、(tame: because)(A)
6
0.500
0.200
ため、(tame: because)(B)
24
0.708
0.664
に伴う (ni tomonau: with)(B)
20
0.750
0.500
に加え、(ni kuwae: besides)(B)
10
0.400
0
ためだ。(tameda: because)(C)
9
0.889
0.889
ためだ。(tameda: because)(D)
6
1.000
0.833
により (ni yori: by)(A)
9
0.778
0.667
により (ni yori: by)(B)
11
0.455
0.273
の効果が (no kouka ga: eﬀect)(B)
8
0.750
0
の影響が (no eikyou ga: inﬂuenced by)(B)
15
0.733
0
により、(ni yori: by)(B)
14
0.929
0.786
によって (ni yotte: by)(B)
10
0.500
0.500
から、(kara: from)(A)
12
0.833
0.667
から、(kara: from)(B)
30
0.700
0.567

Table 7. The number of false positives and false negatives

Cause phrase
Eﬀect phrase

6num. of errors num. of false positives num. of false negatives
71
51
20
139
103
36

Error Analysis

We found two kinds of errors, “false positive” and “false negative”. False positiveis the one where the extracted phrase is not a causal phrase. False negative isthe one where the causal phrase is not extracted. We investigated the number offalse positives and false negatives, shown in Table 7. We can see that the numberof errors of cause phrases are twice the number of errors of eﬀect phrases. Also,
the number of false positives are more than the number of false negatives inboth cause and eﬀect phrase extraction. Therefore, we consider that a methodfor determining the presence or absence of causal knowledge is necessary.
Next, we investigated precision of each pattern. In the investigation, we exclude false positive examples in order to examine performance of syntactic patterns in detail. The results are shown in Table 8. A number of Pattern B’s faults
was the most highest in the four patterns. Most of Pattern B’s faults were theparser errors. The rest of them were algorithmic design errors. We expect thaterrors are decreased by improving an algorithm’s conditional expression.

Table 8. Each precision of pattern

Precision
num. of phrases Cause phrase Eﬀect phrase
Pattern A
38
0.978
0.816
Pattern B
113
0.901
0.761
Pattern C
13
0.867
0.917
Pattern D
6
1.000
1.000

7

Conclusion

We proposed a method that extracts causal knowledge using clue phrases andsyntactic patterns from newspaper articles concerning economic trends. We, ﬁrst,
investigated our newspaper corpus by annotating causal knowledge and cluephrases. Then, we found that some speciﬁc syntactic patterns are useful to improve accuracy to extract causal knowledge. Finally, we developed our system toextract causal knowledge using the clue phrases and the syntactic patterns andshow the evaluation results on a large corpus.

References
1. Inui, T., Inui, K., Matsumoto, Y.: Acquiring causal knowledge from text using theconnective marker tame. Journal of Information Processing Society of Japan 45(3)(2004) 919–933
2. Khoo, C.S., Kornﬁlt, J., Oddy, R.N., Myaeng, S.H.: Automatic extraction of causeeﬀect information from newspaper text without knowledge-based inferencing. Literary and Linguistic Computing 13(4) (1998) 177–186
3. Khoo, C.S., Chan, S., Niu, Y.: Extracting causal knowledge from a medical databaseusing graphical patterns. In: Proceedings of the 38th ACL. (2000) 336–343
4. Girju, R.: Automatic detection of causal relations for question answering. In: In
ACL Workshop on Multilingual Summarization and Question Answering. (2003)
76–83
5. Chang, D.S., Choi, K.S.: Incremental cue phrase learning and bootstrapping method
for causality extraction using cue phrase and word pair probabilities. Information
Processing and Management 42(3) (2006) 662–678
6. Sakai, H., Masuyama, S.: Cause information extraction from ﬁnancial articles concerning business performance, ieice trans. IEICE Trans. Information and Systems
E91-D(4) (2008) 959–968
7. Sakaji, H., Sakai, H., Masuyama, S.: Automatic extraction of basis expressions thatindicate economic trends. In: Paciﬁc-Asia Conference on Knowledge Discovery andData Mining (PAKDD). (2008) 977–984
8. Inui, T., Okumura, M.: Investigating the characteristics of causal relations injapanese text. In: The 43rd Annual Meeting of the Association for Computational
Linguistics, Workshop on Frontiers in Corpus Annotation II: Pie in the Sky. (2005)