NTCIR-7
Almost-Unsupervised Cross-Language Opinion Analysis
NLCL groupTaras Zagibalov*John Carroll

T.Zagibalov@sussex.ac.uk
J.A.Carroll@sussex.ac.uk
Department of Informatics
University of Sussex* supported by the Ford Foundation International Fellowships Program.

Overview●Introduction●Tasks●Our Approach●

Lexical Item Extraction●

Relevance Classification●

Subjectivity Classification●Results●

Error Analysis and Conclusion

19/12/2008

NLCL2

Introduction●●

Our main focus is portability of natural
language processing systems acrosslanguages
Our basic approach is an almostunsupervised approach

19/12/2008

NLCL

3

Tasks●Japanese●English●

Simplified Chinese●

Traditional Chinese

19/12/2008

NLCL

4

Tasks●Relevance Classification●

Subjectivity Classification●

Opinion Classification●

Target Detection●

Opinion Holder Detection

19/12/2008

NLCL

5

Our Approach●

Lexical Item Extraction●

Relevance Classification●

Subjectivity Classification

19/12/2008

NLCL

6

Lexical Item Extraction
Lexical Item (LI) extraction problems:●●

A problem of the word boundary detection inChinese and Japanese.
A problem of idioms / collocations

19/12/2008

NLCL7

Lexical Item Extraction
LI extraction technique used:●●

Any sequence of characters that occurs at least
three times is a candidate to be a LI
If the frequency of a LI is the same as that of ashorter sub-unit then the latter is deleted.

19/12/2008

NLCL

8

Lexical Item Extraction
LI extraction technique used:●●

Any sequence of characters that occurs at least
three times is a candidate to be a LI
If the frequency of a LI is the same as that of ashorter sub-unit then the latter is deleted.

19/12/2008

LI candidate

Frequency

Length美国司法314√美国司

313X司

519

1

√

NLCL

9

Relevance Classification●●●

All LI are ranked according to their frequency ineach document
LI frequency ranks are compared across all thedocuments
LI with the biggest rank differences are selected
as relevance indicators

19/12/2008

NLCL

10

Relevance Classification●●●

All LI are ranked according to their frequency ineach document
LI frequency ranks are compared across all thedocuments
LI with the biggest rank differences are selected
as relevance indicators

19/12/2008

LI

Topic 1 rank

Topic 2 rank

Difference

the2

31

X

netscape

010

10

law

24

618

√
√

NLCL

11

Relevance Classification
Example:

19/12/2008

Topic:'What is the relationship between AOL andNetscape?' (N11)
Relevance indicators:
america online, appliances, designed, dominant,link, maker, netscape, online, services, start-ups,
sun, technological change, they have, windows
operating

NLCL

12

Subjectivity Classification●For each LI we found immediate neighbours:
第五次缔约方大会的中国代表团

19/12/2008

NLCL

13

Subjectivity Classification●For each LI we found immediate neighbours:第五次缔约方大会的中国代表团

中国 : 的 _0, 大会的 _0, 代表团 _1

19/12/2008

NLCL

14

Subjectivity Classification
●

●

●

●

For each neighbour word we calculated chisquare (χ2) score2

LI with χ > 3.84 were included into the list
All such words were ranked according to theirscore
Lists of every two headwords were compared tofind how much of context words they shared

19/12/2008

NLCL

15

Subjectivity Classification●

Syntactic and Semantic relations separated:跟 中国 经济 的 快速对 美国 经济 的 信心

19/12/2008

Syntactic relations

Semantic relations

跟+ 中国
中国 + 经济美国 + 经济经济 + 的中国 + 美国

NLCL

16

Subjectivity Classification
Headwords

中国美国

经济

的

Contextwords经济经济中国经济Contextwords

跟对

的快速

●Good pairs:

中国 + 美国●

Bad pairs:

中国 + 经济 ; 美国 + 经济 ;经济 + 的

19/12/2008

NLCL

17

Subjectivity Classification●

Syntactic and Semantic relations separated:
there are good years and bad years
stable and good conditions

19/12/2008

Syntactic relations

Semantic relations

are + good
good + years
and + badand + good

good + bad

NLCL

18

Subjectivity Classification
Headwords

goodbadandyears

Context
words

andandbadbad

Contextwords

yearsyears

goodand●Good pairs:

good + bad●Bad pairs:

and + bad; and + good;and + years; years + bad;good + years

19/12/2008

NLCL

19Subjectivity ClassificationFiltering the paired headwords:

19/12/2008

Filter 1:
Excluded all pairs with a too small association
score (the score value less than x -1.96σ)
Filter 2:
Deleted all words that occurred in too manyx +1.96σ
pairs ( LI that occurred in more than pairs);

NLCL

20

Subjectivity Classification
RunID1:
Use manually filtered words:
important, difficult, effective, popular,successful, easily, troubled, striking,best, bad, painful, strong, good
Result: low recall

19/12/2008

NLCL

21

Subjectivity Classification


RunID1:
RunID2:


Use manually filtered words
RunID1 + (χ2 >average)

RunID3:

19/12/2008

RunID1 + (χ2 >3.84)

NLCL

22

Subjectivity Classification
Classification algorithm:
1. If a sentence contains a relevance marker >RELEVANT
2. If a sentence is RELEVANT and contains a
subjectivity marker >OPINIONATED
3. Otherwise >NA

19/12/2008

NLCL23

Results: Trad. Chinese (lenient)
100908070

P -relR -relF -relP -opinR -opinF -opin60

50403020

100119/12/2008

23NLCL24

Results: Simp. Chinese (lenient)
1009080

70

P -relR -relF -relP -opinR -opinF -opin

60

50

4030

20

100
119/12/2008

23NLCL25

Results: Japanese (lenient)
1009080

70

P -relR -relF -relP -opinR -opinF -opin

60

50

4030

20

100
119/12/2008

23NLCL26

Results: English (lenient)
100908070

P -relR -relF -rel
P -opinR -opinF -opin

60

50

40

30

20

1001

19/12/2008

23

NLCL

27

Results
Best results (lenient)
Language Sub-task (RunID) Precision
T. Chinese Relevance (3)48.2
Opinion (3)27.7
S. Chinese Relevance (3)97.1
Opinion (3)43.2Japanese Relevance (3)*47.7
Opinion (3)*30.2
EnglishRelevance (3)87.5
Opinion (3)47.6

Recall
68.9
84.6
58.5
69.9
63.8
91.0
41.1
74.2

F-value
56.7
41.7
73.0
53.4
54.6
45.3
55.6
58.0

*Note that the RunID3 results were obtained after the official submission.

19/12/2008

NLCL

28

Error Analysis●

Small amount of data●

More noise with higher recall●

Word segmentation for the Asian languages
●发展中国家 : 发展中 + 国家 / 发展 + 中国 + 家

POS tagging

19/12/2008

NLCL

29

Conclusion●●●

Simple almost unsupervised cross-lingualsystem
Satisfactory results for the Japanese andEnglish tasks
Rather poor performance for the Chinese (both)

19/12/2008

NLCL

30

Future Work●

Reduce noise●

Automate subjectivity marker selection●

Develop unsupervised language independent(quasi-)POS tagging technique

19/12/2008

NLCL31

ありがとうございます謝謝谢谢
Thank you

19/12/2008

NLCL

32

Results
Traditional Chinese (lenient)
Sub-task (RunID)
Relevance (1)Opinion (1)Relevance (2)Opinion (2)Relevance (3)Opinion (3)19/12/2008

Precision
84.9
53.6
86.4
49.4
85.7
47.6

Recall
14.5
26.8
28.6
50.6
41.1
74.2

NLCL

F-value
24.8
35.7
43.0
50.0
55.6
58.0

33

Results
Simplified Chinese (lenient)
Sub-task (RunID)
Relevance (1)Opinion (1)Relevance (2)Opinion (2)Relevance (3)Opinion (3)19/12/2008

Precision
96.3
44.3
97.5
48.297.1
43.2

Recall
32.6
39.9
28.0
36.9
58.5
69.9

NLCL

F-value
48.7
42.0
43.5
41.8
73.0
53.4

34

Results
Japanese (lenient)
Sub-task (RunID)
Relevance (1)Opinion (1)Relevance (2)Opinion (2)Relevance (3)*Opinion (3)*Precision
53.7
42.6
47.7
30.2

Recall
18.9
22.3
63.8
91.0

F-value
28.0
29.3
54.6
45.3

*Note that the RunID3 results were obtained after the official submission.

19/12/2008

NLCL

35

Results
English (lenient)
Sub-task (RunID)
Relevance (1)Opinion (1)Relevance (2)Opinion (2)Relevance (3)Opinion (3)19/12/2008

Precision
13.0
37.8
17.5
33.8
48.2
27.7

Recall
6.8
10.1
14.4
18.6
68.9
84.6

NLCL

F-value
9.0
16.0
15.8
24.0
56.7
41.7

36