Chinese Textual Sentiment
Analysis: Datasets, Resources
and Tools
Natural Language and Knowledge Processing Lab
Wei-Fan Chen and Lun-Wei Ku
December 11 @ Coling 2016, Osaka, Japan

Program and Speaker
Lecturer: Lun-Wei Ku• 1. Overall Introduction (40 min)• 2. Introduction to CSentiPackage (40 min)-------------Coffee Break: 20 min -----------------Lecturer: Wei-Fan Chen
• 3. Introduction to CSentiPackage:UTCNN(20 min)• 4. Hands on Real data (40 min)
11 December 20162

Overall Introduction
Sentiment Analysis

11 December 2016

3

4

Sentiment Analysis Is…
• Studying opinions, sentiments, subjectivities,
affects, emotions, views, etc. in text such as
news, blogs, reviews, comments, dialogs, orother kind of documents.
• An important research question:– Sentiment information is global and powerful.
– Sentiment information is valuable for companies,
customers and personal communication.
11 December 2016

Opinion Definition• From triple to quintuple– Triple:( ej, soij, hi )– Quintuple: (Bin Liu, NLP handbook, 2010)( ej, ajk, soijkl, hi, tl )ej: target entity jhi: holder iajk: aspect k (or sometimes called feature) of target
entity j
tl: time lso: sentiment value of the opinion
11 December 2016

5

Sentiment Representation• Categorical– Sentiment, non-sentiment– Positive, neutral, negative– Stars– Emotions categories like Joy, Angry, Sadness…

• Dimensional– Valence Arousal

11 December 2016

6

Sentiment Data Construction• Sentiment labels are subjective: moreannotators could make them more reliable.
• Manual gold data
– Annotated by at least 3 annotators– Crowdsourcing• User generated data (automatically generated)– User review scores (stars)– User generated text with emoticons (noisy)– Labels available from social platform
11 December 2016

7

8

Annotation Consideration• Granularity : Word, Sentence, Passage, Document?
– Sentences are natural units but their labels are rarely
found.
– Detecting emotions from sentences is the most difficult(some are of complex semantic but very few words).

• Data Management– Explicit answer vs. majority answer– w/ context vs. w/o context– Data segmentation
11 December 2016

Annotation Quality• Agreement– Raw agreement– Kappa value, weighted kappa value

11 December 20169

Now we get some ideas of sentiment
analysis…let’s see what the recent
research is about!

11 December 201610

Overall Introduction
Related Work

11 December 2016

11

Widely known early work
• Thumbs up? Sentiment classification using
machine learning techniques (Pang and Lee,EMNLP 2002): binary SVM classifier on
documents.

11 December 201612

A good start to get the idea ofsentiment analysis• Survey: Opinion Mining and Sentiment
Analysis, Bo Pang and Lillian Lee,
Foundations and Trends in Information
Retrieval, 2008. (135 pages)• Book: Sentiment Analysis and Opinion Mining,
Bing Liu, Morgan & Claypool Publishers,
2012. (168 pages)
11 December 201613

Recent One Year’s Research…
ACL• Sentiment Domain Adaptation with Multiple
Sources• Connotation Frames: A Data-Driven Investigation• Bi-Transferring Deep Neural Networks for Domain
Adaptation• Document-level Sentiment Inference with Social,
Faction, and Discourse Context

11 December 201614

Recent One Year’s Research…
NAACL• Ultradense Word Embeddings by Orthogonal Transformation• Separating Actor-View from Speaker-View Opinion
Expressions using Linguistic Features• Clustering for Simultaneous Extraction of Aspects andFeatures from Reviews• Opinion Holder and Target Extraction on Opinion Compounds-- A Linguistic Approach• Capturing Reliable Fine-Grained Sentiment Associations byCrowdsourcing and Best–Worst Scaling

11 December 201615

Recent One Year’s Research…
EMNLP• Aspect Level Sentiment Classification with Deep
Memory Network• Lifelong-RL: Lifelong Relaxation Labeling for
Separating Entities and Aspects in Opinion Targets• Learning Sentence Embeddings with Auxiliary
Tasks for Cross-Domain Sentiment Classification• Attention-based LSTM Network for Cross-Lingual
Sentiment Classification

11 December 201616

Recent One Year’s Research…
• Aspect• Domain Adaptation for Cross-Domain/Lingual• Deep Neural Network vs. Linguistic Features• Fine-Grained• Crowdsourcing

11 December 201617

Overall Introduction
Chinese Text Processing

11 December 201618

Chinese Language• Has no space between words• The finest granularity of most sentiment tools
is word : need word segmentation• Part of speech tagging and syntactic
information (parse tree) are nice to have.
• Two major Chinese writing forms: simplified
Chinese and traditional Chinese

11 December 201619

Chinese Language Processing Tools• The most widely used tool for Chinese isStanford CoreNLP1 (simplified Chinese)• Other popular ones:– LTP Cloud (simplified Chinese)– CKIP Parser2 (traditional Chinese)– jieba (segmentation, both simplified/traditional
Chinese)1 http://nlp.stanford.edu/software/
2 http://godel.iis.sinica.edu.tw/CKIP/parser.htm

11 December 2016

20

CKIP Parser• Its tag set is different from Stanford
CoreNLP’s
我們都喜歡蝴蝶
我們(Nh) 都(D) 喜歡(VK) 蝴蝶(Na)#1:1.[0] S(experiencer:NP(Head:Nhaa:我們)|quantity:Dab:都|Head:VK1:
喜歡|goal:NP(Head:Nab:蝴蝶))#。(PERIODCATEGORY)• We provide a tag mapping file (for sentiment
analysis)
11 December 201621

CSentiPackage@NLPSA

11 December 201622

CSentiPackage• Datasets– Chinese Morphological Dataset Cmorph (former
version of ACiBiMA)*– Chinese Opinion Treebank• Resources– NTUSD/ANTUSD• Tools– CopeOpi + Tag Mapping File
– UTCNN*https://github.com/windx0303/ACBiMA
11 December 201623

Statistics• NTUSD: Sentiment Dictionary (with 10,371
words): free for research, 400+ applications• ANTUSD: Augmented NTUSD (with 27,221
words, now integrating with e-Hownet)• Cmorph (with 8,000+ words) -> ACBiMA(with 11,000+ words)• Chinese Opinion Treebank: labels on Chinese
Treebank 5.1
11 December 201624

Materials:
From Words to Sentences• NTUSD: words (binary sentiment)• ANTUSD: words (annotation features)• Chinese Morphological Dataset: words(morphological structures)• Chinese Opinion Treebank: phrases (sentence
structure)• Chinese Opinion Treebank: sentences (binary
sentiment)
11 December 201625

Tools:From Words to Sentences,Documents, and Beyond• CopeOpi Sentiment Scoring Tool: words,
sentences, documents, documents+ (text)• UTCNN: posts and users (text and social
media)

11 December 2016

26

NTUSD• Simplified Chinese and traditional Chineseversions• A positive word collection of 2,812 words• A negative word collection of 8,276 words• No degree, no estimated scores and otherinformation.

11 December 201627

ANTUSD• 6 Fields– CopeOpi Score– Number of positive annotation– Number of neutral annotation– Number of negative annotation– Number of non-sentiment annotation– Number of not-a-word annotation
開心
酣聲
憤怒

0.434168
0-0.80011

1
0
0

0
0
0

0
1
5

0
3
0

0
0
0

• Not-a-word: useful as they are collected from real
segmentated data
11 December 201628

ANTUSD• Contains also short phrases like一昧要求, 一路過關斬將,備受外界期待…

11 December 201629

ANTUSD and E-HOWNET
E-How N e t.., A frame-based entity-relation model extended from HowNet.., Define lexical senses (concepts) in a hierarchical manner.., Now integrated with ANTUSD and covers 47.7% words in

ANTUSD• An integration of two resources which may help us play withsentiment and semantics.
• Related English resource: SentiWordnet– Refer to Wordnet– With PosScore and NegScore added– ObjScore = 1-(PosScore+NegScore)
11 December 201630

ANTUSD in E-HOWNET

11 December 201631

11 December 201632

Chinese Morphological Structure• Parallel type: 財富 (rich wealth)• Substantive-Modifier type: 痛哭 (bitterly cry)
• Subjective-Predicate type: 山崩 (land slip; landslide)• Verb-Object type: 避暑 (escape from summer)• Verb-Complement type: 提高 (increase: raise up)• Negation type: 無情 (no feelings)• Confirmation type: 有心 (have heart)• Others
11 December 201633

Chinese Opinion Treebank• Based on Chinese Treebank 5.1.
• Including the opinion labels of each sentences.
• Including the word-pairs and their composing
type in opinionated sentences.
• To avoid copyright issue, you need to have
Chinese Treebank 5.1 by yourself in order to
use Chinese Opinion Treebank!

11 December 201634

Chinese Opinion Treebank
S ID=230: 黄河“金三角”成为新的投资热点.node file
Node ID, POS, nodecontent, node depth.tree file
FieldsNode ID: children.trio file
Trio ID, trio head, trio left
node, trio right node, trio
type

Content
0,,,0
1,IP-HLN,,1
2,NP-SBJ,,2
3,NP-PN,,3
4,NR,黄河,4
5,NP,,3
6,PU,“,4
7,NN,金三角,48,PU,”,4
9,VP,,2
10,VV,成为,311,NP-OBJ,,312,CP,,4
13,WHNP-1,,5
14,-NONE-,*OP*,6
15,CP,,516,IP,,6
17,NP-SBJ,,718,-NONE-,*T*-1,8
19,VP,,7
20,VA,新,8
21,DEC,的,6
22,NP,,4
23,NN,投资,524,NN,热点,5

0:1,
1:2,9,
2:3,5,
3:4,
4:
5:6,7,8,6:
7:
8:
9:10,11,10:
11:12,22,
12:13,15,
13:14,
14:
15:16,21,
16:17,19,
17:18,
18:
19:20,
20:
21:
22:23,24,
23:
24:

2,1,2,9,3
3,22,23,24,2Opinion labels of three annotators(filename, SID, opinion, polarity, opinion type)chtb_020.raw,230,N,,
chtb_020.raw,230,Y,POS,STATUS
chtb_020.raw,230,Y,POS,STATUS
Opinion gold standard
chtb_020.raw,230,Y,POS,STATUS

11 December 201635

Notation (Parsing Tree)

Tri(S)=
1, IP, 活动, VP, Subjective-Predicate
2, VP, 取得, NP-OBJ, Verb-Object
3,NP-OBJ, 圆 满 , 成 功 , SubstantiveModifier
11 December 2016

• T: the parsing tree of asentence S• O = {o1, o2, …}: in-ordered setof tree nodes• tri=: an opinion trio•
: a syntactic interword relation
Rpt є {Substantive-Modifier,
Subjective-Predicate, VerbObject, Verb-Complement,
Other}
36

Chinese Opinion Treebank• Align the opinion labels of sentences toChinese Treebank 5.1 by sentence IDs.
• Align Opinion trios to Chinese Treebank 5.1
by node IDs.
• Can be used to do opinion cause analysis.

11 December 201637

CopeOpi• A statistical sentiment analysis tool
• Can be used without any training• Users can update character weights or add any
sentiment words• It runs fast.

11 December 201638

The First Idea
• Chinese characters are mostly morphemes and they
bear sentiment, too.
• Simple example: some characters are preferred fornaming, but some are not.
• For example, 德(ethic) 胜(win) 高(high) good fornames; 笨(stupid) 悲(sorrow) 惨(terrible) are notgood choices for names.
• With some exceptions, but still quite reliable if thesentiment of character is acquired statistically from alarge naming corpus (or just sentiment dictionaries.)
Exceptions like 徐悲鸿.
11 December 2016

39

Bag of Unitm

N ci 

nfnci /  fnc j
j 1

n

m

j 1

j 1

fpci /  fpc j  fnci /  fnc jPci fpci /  fpc j
j 1

n

m

j 1

j 1

fpci /  fpc j  fnci /  fnc j

S ci  ( Pci  N ci )1 pSw    Sc j
p j 1[仇 (-1.0) + 視 (0.0)] / 2 = -1/2 = -0.5 (NEG)[富(1.0) + 貴(0.936)] / 2 = 0.968 (POS)
好人、美麗、憤怒、弱小…
11 December 201640

Aggregation• Word sentiment– Summing up opinion scores of characters• Sentence sentiment– Summing up opinion scores of words

So is there any way we can give them weights?
11 December 201641

Weighted by Structures• Linguistic Information:– Morphological structures• Intra-word structures– Sentence syntactic structures• Inter-word structures

11 December 201642

Morphological Structure
Get types by SVM, CRF, handcraft…
Linguistic Morpho. Type

Example1. Parallel

財富、打罵

2. Substantive-Modifier

低級、痛哭

3. Subjective-Predicate

心疼、氣虛

4. Verb-Object

失控、免職

5. Verb-Complement

看清、擊潰

Opinion Morpho. Type

Example6. Negation

無法、不慎

7. Confirmation

有賴、有愧

8. Others

姪子、薄荷

11 December 201643

Example of Sentiment Trios in
Chinese Opinion Treebank
Linguistic Morpho. Type

ExampleParallel (Skip)

美麗而聰慧

1. Substantive-Modifier

高大的樓房

2. Subjective-Predicate

學習認真

3. Verb-Object

恢復疲勞

4. Verb-Complement收拾乾淨Morpho. Type Opinion

Examplen. Others

為…/以…

11 December 201644

Compositional
Chinese Sentiment Analysis
Sentiment Scoring Formulafor Each Morphological Type:• Parallel type
S (C1C 2 ) 

S (C1 )  S (C 2 )2• Substantive-Modifier type

•
•
•
•
•

Example:氣虛
Subjective-Predicate type
氣 0.5195
虛 -0.8178
Score(氣虛) = -0.8178

if ( S (C1 )  0 and S (C2 )  0) thenif ( S (C1 )  0 and S (C2 )  0) then S (C1C2 )  S (C1 )else S (C1C2 )  - 1  S (C1 )else S (C1C2 )  S (C1 )  S (C2 )• Subjective-Predicate
if ( S (C 2 )  0) then S (C1C 2 )  S (C 2 )else S (C1C 2 )  S (C1 )11 December 201645

Compositional
Chinese Sentiment Analysis
Sentiment Scoring Formulafor Each Morphological Type:

•
•
•
•
•
•
•

Example:看清、看壞
Verb-Complement type
看: 0.1
清: 0.8032
壞: -0.9
Score(看清) = 0.8072
Score(看壞) = -0.9

• Verb-Object typeif ( S (C1 )  0 and S (C 2 )  0)then S (C1C 2 )  S (C1 )  SIGN ( S (C1 ))  SIGN ( S (C 2 ))else S (C1C 2 )  S (C1 )  S (C 2 )• Verb-Complement type
= Subjective-Predicate type• Negation typeif (C1  NC ) then S (C1C 2 )   1  S (C 2 )else S (C1C 2 )   1  S (C1 )• Confirmation typeif (C1  PC) then S (C1C2 )  S (C2 ) else S (C1C2 )  S (C1 )• Others = Parallel type
11 December 201646

Example of Using Sentiment Trios•

Score: 0.6736
0.3018
Verb-Object typeif ( S (C1 )  0 and S (C 2 )  0)then S (C1C 2 )  S (C1 )  SIGN ( S (C1 ))  SIGN ( S (C 2 ))0.6736else S (C1C 2 )  S (C1 )  S (C 2 )Substantive-Modifier typeif ( S (C1 )  0 and S (C2 )  0) thenif ( S (C1 )  0 and S (C2 )  0) then S (C1C2 )  S (C1 )else S (C1C2 )  - 1  S (C1 )else S (C1C2 )  S (C1 )  S (C2 )0.67360.410911 December 201647

Performance of CopeOpi(Dataset w/o Structure)
Level

CorpusBy

Precision

Recallf-measureWord

836 words

Annotator

0.81

0.80

0.80

Sentence

CIRB010-OP

Annotator

0.75

0.65

0.66

Document CIRB010-OP

Annotator

0.73

0.69

0.72

Word

836 words

Machine

0.61

0.79

0.68

Sentence

CIRB010-OP

Machine

0.38

0.65

0.48

Sentence

CIRB020-OP

Machine

0.33

0.45

0.38

Sentence

CIRB020-OP-R

Machine

0.66

0.89

0.76

Document

CIRB010-OP

Machine

0.40

0.55

0.46

*NTCIR MOAT Corpus as materials
11 December 201648

Performance of CopeOpi(Dataset w/ Structure)
Setting Word

Sentence Opinion Polarity

Desc

1

bag

bag

0.70730.4988

2

struc

bag

0.71620.5117

CRF

3

bag

struc

0.80000.5361

Manual

4

struc

struc

0.79220.5297

CRF+Manual

5

struc

struc

0.79930.5187

CRF+Auto*Chinese Opinion Treebank as materials

11 December 201649

Performance of CopeOpi(FB Stance Classification)

11 December 201650

Deep Neural Network Example
Word• Morphological structure
for a better
word representation.
• Same idea but
for *Chinese sentiment
analysis*•

Luong, Thang, Richard Socher, and Christopher D. Manning. "Better Word Representations with Recursive Neural Networks
for Morphology." CoNLL. 2013.

11 December 201651

Deep Neural Network Example
Sentence•

Learned composition function (of semantics): Richard Socher (RNN, series
work from 2011)11 December 201652

Learning by Neural Network• Word Sentiment• Sentence Sentiment• Document Sentiment• Social Media Post Sentiment

11 December 201653

Learning by Deep Neural Network• Word Sentiment: CNN + ANTUSD• Sentence Sentiment• Document Sentiment• Social Media Post Sentiment: Text + User
Context– Not yet consider structures!

11 December 201654

Word Sentiment NN:
CNN + ANTUSDA Demonstrative Experiment
ANTUSD: A Large Chinese Sentiment Dictionary, Shih-Ming Wang and Lun-Wei Ku, inProceedings of LREC 201611 December 201655

Experiment S e t t i n g

Demonstrat i v e Experiment.., Dataset: ANTUSD ∩ E-hownet, a total 12995 words.., Classifier: support vector machine (SVM) with linear kernel.., Average over 10-fold validation scores

Three sentiment a n a lysis t asks.., Opinion extraction: identify opinion words({ POS,NEG} v.s. NONOP).., Polarity classification: classify opinion words (POS v.s. NEG).., Combined tasks (POS, NEG, NONOP)..,

P = correct(opinion)∩correct(polarity)
proposed(opinion)..,

R = correct(opinion)∩correct(polarity)
gold(opinion)

..,

2PR
F score = P+R

11 December 201656

P r e p rocessing

E x t r a c t single l a b e l f o r each wo r d
1. NOT: Count(Not)> 0
2. NONOP: Count(Non)> 0
3. POS: Count(Pos)> 0 and Count(Neg)=0
4. NEG: Count(Neg)> 0 and Count(Pos)=0
5. NEU: Count(Pos)=0, Count(Neg)=0 and Count(Neu)> 0

11 December 201657

P r e p rocessing
E x t r a c t single l a b e l f o r each wo r d
1. NOT: Count(Not)> 0
2. NONOP: Count(Non)> 0
3. POS: Count(Pos)> 0 and Count(Neg)=0
4. NEG: Count(Neg)> 0 and Count(Pos)=0
5. NEU: Count(Pos)=0, Count(Neg)=0 and Count(Neu)> 0.., NOT words are not used.., NEU words are dropped since there are only 16 of them

11 December 201658

Feat u r e s

ANTUSD & E-how n e t.., CopeOpi score in ANTUSD.., Synonym-Set index (SSI)

Concept frame index of a word.., Each word might belong to many concepts.., Represented as a binary vector..,

11 December 201659

Feat u r e s
ANTUSD & E-how n e t.., CopeOpi score in ANTUSD.., Synonym-Set index (SSI)

Concept frame index of a word.., Each word might belong to many concepts.., Represented as a binary vector..,

W o r d Embedding.., Corpus: LDC2009T14 (Chinese news).., Word vectors.., Summation of char vectors..,

11 December 2016

Very high coverage rate
60

Opinion E x t r a c t i o n.., COP, SSI has lower precision

opinion extraction is moresemantic-oriented.., Many concept frame
contain only one word..,

11 December 2016

Feature(s) Precision Recall f-score

COP
SSI
WV
CV

0.686
0.693
0.784
0.765

1.000 0.814
0.993 0.816
0.936 0.854
0.919 0.835

COP+SSI
COP+WV
COP+CV
SSI+WV
SSI+CV
WV+CV

0.740
0.785
0.764
0.789
0.772
0.808

0.914
0.933
0.917
0.937
0.920
0.921

0.818
0.853
0.833
0.856
0.840
0.861

61

Opinion E x t r a c t i o n.., COP, SSI has lower precision

opinion extraction is moresemantic-oriented.., Many concept frame
contain only one word..,.., Character vectors lead toslightly worse performance

11 December 2016

Feature(s) Precision Recall f-score
COP
0.686 1.000 0.814
SSI
0.693 0.993 0.816
WV
0.784 0.936 0.854
CV
0.765 0.919 0.835
COP+SSI 0.740 0.914 0.818
COP+WV 0.785 0.933 0.853
COP+CV 0.764 0.917 0.833
SSI+WV 0.789 0.937 0.856
SSI+CV
0.772 0.920 0.840
WV+CV
0.808 0.921 0.861

62•

Opinion E x t r a c t i o n.., COP, SSI has lower

precision
opinion extraction ismore semantic-oriented• .., Many concept
frame contain only
one word•..,• .., Character vectors leadto slightly worse
performance• .., Features are

Feature(s) Precision Recall f-score
COP
0.686 1.000 0.814
SSI
0.693 0.993 0.816
WV
0.784 0.936 0.854
CV
0.765 0.919 0.835
COP+SSI 0.740 0.914 0.818
COP+WV 0.785 0.933 0.853
COP+CV 0.764 0.917 0.833
SSI+WV 0.789 0.937 0.856
SSI+CV
0.772 0.920 0.840
WV+CV
0.808 0.921 0.861

complemented; combinedfeatures leads toimprovement
11 December 201663

P o l a r i t y Classificat i o n.., COP leads to a significant

better result, reflecting issentiment-oriented nature

11 December 2016

Feature(s) POS f1 NEG f1 Average f1
COP
0.973 0.976
0.974
SSI
0.792 0.842
0.817
0.882
WV
0.870 0.895
0.840
CV
0.829 0.851
0.980
COP+SSI 0.979 0.982
0.982
COP+WV 0.981 0.984
0.970
COP+CV 0.967 0.972
0.907
SSI+WV 0.898 0.915
0.877
SSI+CV 0.868 0.886
0.908
WV+CV 0.899 0.916

64

P o l a r i t y Classificat i o n.., COP leads to a significant

better result, reflecting issentiment-oriented nature.., Combining COP & otherfeatures still leads toimprovement

11 December 2016

Feature(s) POS f1 NEG f1 Average f1
COP
0.973 0.976
0.974
SSI
0.792 0.842
0.817
WV
0.870 0.895
0.882
CV
0.829 0.851
0.840
COP+SSI 0.979 0.982 0.980
COP+WV 0.981 0.984 0.982
COP+CV 0.967 0.972
0.970
SSI+WV 0.898 0.915
0.907
SSI+CV 0.868 0.886
0.877
WV+CV 0.899 0.916
0.908

65

P o l a r i t y Classificat i o n.., COP leads to a significant

better result, reflecting issentiment-oriented nature.., Combining COP & otherfeatures still leads toimprovement.., Combining word vectors and

SSI also leads toimprovement

11 December 2016

Feature(s) POS f1 NEG f1 Average f1
COP
0.973 0.976
0.974
SSI
0.792 0.842 0.817
WV
0.870 0.895 0.882
CV
0.829 0.851
0.840
COP+SSI
0.979 0.982
0.980
COP+WV 0.981 0.984 0.982
COP+CV 0.967 0.972 0.970
SSI+WV 0.898 0.915 0.907
SSI+CV 0.868 0.886
0.877
WV+CV 0.899 0.916
0.908

66

Combined Task.., COP outperforms the others

11 December 2016

Feature(s) Precision Recall f-score
COP
0.912 0.927 0.920
SSI
0.706 0.679 0.692
WV
0.737 0.767 0.752
CV
0.689 0.721 0.705
COP+SSI 0.864 0.945 0.903
COP+WV 0.850 0.902 0.875
COP+CV 0.840 0.869 0.854
SSI+WV 0.764 0.796 0.779
SSI+CV
0.732 0.755 0.743
WV+CV
0.764 0.813 0.787

67

Combined Task.., COP outperforms the others

Precision & Recal l.., Both the numerator of

P = correct(opinion)∩correct(polarity)
proposed(opinion)

precision and recall areaffected by COP’s better
polarity classification ability.., Only the denominator of

precision is affected by
COP’s worse opinion
extraction ability

11 December 2016

R = correct(opinion)∩correct(polarity)
gold(opinion)
Feature(s) Precision Recall f-score
COP
0.912 0.927 0.920
SSI
0.706 0.679 0.692
WV
0.737 0.767 0.752
CV
0.689 0.721 0.705
COP+SSI 0.864 0.945 0.903
COP+WV 0.850 0.902 0.875
COP+CV 0.840 0.869 0.854
SSI+WV 0.764 0.796 0.779
SSI+CV
0.732 0.755 0.743
WV+CV
0.764 0.813 0.787
68

Combined Task.., COP outperforms the others.., Both the numerator of

precision and recall areaffected by COP’s better
polarity classification ability.., Only the denominator of

precision is affected by
COP’s worse opinion

extraction ability.., WV+CV outperforms WV
due to coverage issue

11 December 2016

Feature(s) Precision Recall f-score
COP
0.912 0.927 0.920
SSI
0.706 0.679 0.692
WV
0.737 0.767 0.752
CV
0.689 0.721 0.705
COP+SSI
0.864 0.945 0.903
COP+WV
0.850 0.902 0.875
COP+CV 0.840 0.869 0.854
SSI+WV
0.764 0.796 0.779
SSI+CV
0.732 0.755 0.743
WV+CV
0.764 0.813 0.787

69

Inject More Semantics:
ANTUSD and E-Hownet
E-How N e t.., A frame-based entity-relation model extended from HowNet.., Define lexical senses (concepts) in a hierarchical manner.., Now integrated with ANTUSD and covers 47.7% words in

ANTUSD

11 December 201670

Wrapup• CSentiPackage– NTUSD/ANTUSD/ANTUSD+e-HowNet– Chinese Morphological Dataset Cmorph– Chinese Opinion Treebank– CopeOpi + Tag Mapping File
– An demonstrative exp of ANTUSD======== We are here ==============– UTCNN (next session)• Hand-on
11 December 2016

71

Future Release Tool
in CSentiPackage• EmotionPushCore: short message emotion
detector (ongoing)

11 December 201672

10:20-10:40

11 December 2016

73

CSentiPackage:
UTCNN

11 December 201674

Learning by Deep Neural Network• Word Sentiment: CNN + ANTUSD• Sentence Sentiment• Document Sentiment• Social Media Post Sentiment: Text + User
Context

11 December 201675

Outline• CSentiPackage: UTCNN– Introduction– Model– Results• Hands on real data
– Environment– Data preprocessing– Tools• NTUSD and ANTUSD• Cmorph and Chinese Opinion Treebank• CopeOpi• UTCNN
11 December 201676

Outline• CSentiPackage: UTCNN– Introduction– Model– Results• Hands on real data
– Environment– Data preprocessing– Tools• NTUSD and ANTUSD• Cmorph and Chinese Opinion Treebank• CopeOpi• UTCNN
11 December 201677

User Topic Comment Neural Network(UTCNN)• A deep learning model of stance classification
on social media text
LikersAuthors
TopicsDeep Learning Model
Post content

Commenters
Comment content

11 December 2016

78

UTCNN• Stance tendency– Author– Liker– Topic– Commenter• Semantic preference– Author– Liker– Topic– Commenter
11 December 2016

(post)We should reject the re-constructionof the Nuclear power plant.
Great! ()

NO! ……
(comment)
79

Document Composition• From word representation to document
representation
– CNN
– RNN
• LSTM

11 December 2016

80

CNN architecture••• Capture n-gramfeatures

11 December 201681

User- and Topic-dependent
document composition• Uk models the user
reading preference
for certain semantics• Tj models the topic
semantics

11 December 201682

User- and topic-dependent
stance tendency• uk models the user stance preference• tj models the topic stance tendency

11 December 201683

Authors and Likers
Moderator matrix embedding
Maximum pooling

11 December 2016

Moderator vector embedding
Maximum pooling84

Topics
Topic matrix embedding

Maximum pooling

11 December 2016

Topic vector embedding

Maximum pooling85

Comment model• Short document with only author

11 December 2016

86

UTCNN – full view

11 December 201687

Dataset• Facebook fan groups– Author/liker/comment/commenter– Single topic (learn latent topics by LDA)
– Unbalance– Chinese• Create Debate– Author– Four topics– Balance– English
11 December 201688

Dataset

Annotation results of FBFans and CreateDebate dataset

11 December 201689

Experiment settings• Convolution filter window sizes: 1, 2, 3
• Word embedding dimension: 50• User/topic matrix embedding size: 250 (5X50)• User/topic vector embedding size: 10• Latent topics: 100• Maximum topics per document: 3

11 December 201690

Results - FBFans
MethodF-scoreFeatures
Content User Topic Comment

Majority

Sup

Neu

Uns

F1SNU

.000 .841 .000

.280

SVM -UniBiTrigram

V

V.610 .938 .156

.621

SVM -AvgWordVec

VV.526 .100 .165

.336

SVM -AvgWordVec (transformed)

VV.597 .963 .210

.642

CNN (Kim, 2014)

V

V.726 .964 .222

.648

RCNN (Lai et al., 2015)

V

V.628 .944 .096

.605

UTCNN – user

VV

.748 .973 .000

.580

UTCNN – topic

V

V

V

.643 .944 .476

.706

UTCNN – comment

V

V

V

.632 .940 .480

.707

UTCNN shared user embedding

V

V

V

V

.625 .969 .531

.732

UTCNN (full)

V

V

V

V

.698 .957 .571 .755*

11 December 2016

V

V

V

91

Results - CreateDebate
MethodFeatures
Text

User

Majority

Topics

AVG

ABO

GAY

OBA

MAR

.549

.634

.539

.695

.604

SVM -UniBiTrigram

V

.592

.569

.565

.673

.600

SVM -AvgWordVec

V

.559

.637

.548

.708

.613

SVM -AvgWordVec (transformed)

V

.859

.830

.800

.741

.808

CNN (Kim, 2014)

V

.553

.636

.557

.709

.614

RCNN (Lai et al., 2015)

V

.553

.637

.534

.709

.608

ILP (Hasan and Ng, 2013a)

V

.614

.626

.581

.669

.623

ILP (Hasan and Ng, 2013a)

V

V

.749

.709

.727

.754

.735

CRF (Hasan and Ng, 2013b)

V

V

.747

.699

.711

.754

.728

PSL (Sridhar et al., 2015)

V

V

.668

.727

.635

.690

.680

UTCNN – topic

V

V

.824

.851

.743

.814

.808

UTCNN – user

V

.617

.627

.599

.685

.632

UTCNN (full)

V

.878

.850

.857

.782

.842*

11 December 2016

V

V

92

Conclusion
•

•
•
•
•

We have proposed UTCNN incorporating user, topic, content
and comment information for stance classification on social
media texts.
UTCNN learns user embeddings for all users with minimum
active degree.
Topic information obtained from the topic model or the predefined labels further improves the UTCNN model.
Comment information provides additional clues for stanceclassification.
We have shown that UTCNN achieves promising andbalanced results.

11 December 201693

Hand-on Session
11 December 201694

Outline• CSentiPackage: UTCNN– Introduction– Model– Results• Hands on real data
– Environment– Data preprocessing– Tools• NTUSD and ANTUSD• Cmorph and Chinese Opinion Treebank• CopeOpi• UTCNN
11 December 201695

Environment• Software– OS: Linux– Programming language• Java 6 or higher• python 2.7– Theano 0.8.2– Keras 1.0.3– sklearn• Hardware– Graphic cards (deep learning)
11 December 201696

Demo Environment• CPU– Intel Xeon E5-2630 v3 ×2• RAM– 64 GB• OS– Ubuntu 14.04 LTS• Graphic cards– Nvidia Tesla K40 ×2
11 December 201697

Preprocessing• Tokenize– Jieba– CKIP– Stanford parser• Part-of-speech tagging– CKIP– Stanford parser

11 December 201698

NTUSD• National Taiwan University Sentiment Dictionary• Release date: 2006
• Language: Traditional/ Simplified Chinese• Data: 11,088 sentiment words– 2,812 positive words– 8,276 negative words

11 December 201699

NTUSD – package•

NTUSD_negative_unicode.txt

11 December 2016

NTUSD_positive_unicode.txt

100NTUSD - reference• Ku, L. W., Liang, Y. T., & Chen, H. H. (2006, March). Opinion
Extraction, Summarization and Tracking in News and Blog
Corpora. In AAAI spring symposium: Computational
approaches to analyzing weblogs.
••

http://doraemon.iis.sinica.edu.tw/coling2016_tutorial/downloads/NTUSD_traditional.zip
http://doraemon.iis.sinica.edu.tw/coling2016_tutorial/downloads/NTUSD_simplified.zip

11 December 2016101

ANTUSD• Augmented NTUSD• Release date: 2016
• Language: Traditional/ Simplified Chinese• Data: 27,221 words– 9,382 positive words– 16 neutral words– 11,224 negative words– 5,415 non-opinion words– 612 negation words
11 December 2016102

ANTUSD - example
•

Score

Pos

Neu

Neg

Nonop

支持
(support)

0.0381147

1

0

0

0

全力支持
(fully support)

0.2870457

1

0

0

0

不支持
(not support)-0.1949018 0

0

1

0

11 December 2016103

ANTUSD - package•
readme.txt

opinion_words.zip

opinion_word.csv

11 December 2016

104

ANTUSD - reference• Wang, Shih-Ming, and Lun-Wei Ku. "ANTUSD: A Large
Chinese Sentiment Dictionary." in LREC 2016.
••

http://doraemon.iis.sinica.edu.tw/coling2016_tutorial/downloads/ANTUSD_traditional.zip
http://doraemon.iis.sinica.edu.tw/coling2016_tutorial/downloads/ANTUSD_unicode.zip

11 December 2016

105

Cmorph• Cmorph.txt: morphological types are labeled
by numbers:– 1:Parallel– 2: Substantive-Modifier– 3: Subjective-Predicate– 4: Verb-Object– 5: Verb-Complement– 8: Others*6: Negation and 7: Confirmation are detected by rules*Huang, Ting-Hao, Ku, Lun-Wei and Chen, Hsin-Hsi, Predicting Morphological Types of Chinese BiCharacter Words by Machine Learning Approaches, LREC 2010, pages 844-850,
11 December 2016106

Chinese Opinion Treebank• Excel file:sentence.csv*Ku, Lun-Wei, Huang, Ting-Hao and Chen, Hsin-Hsi, Construction of Chinese Opinion Treebank, LREC 2010,
pages 1315-1319.
11 December 2016107

Chinese Opinion Treebank

trio

002

12

002_12.tri(docID) (senID)

11 December 2016108

Chinese Opinion Treebank• 外商 投資 企業
成為 中國 外貿
重要 增長點

002_12.tree <S ID=12>( (IP-HLN (NP-SBJ (NN外商)(NN投資)(NN 企業))(VP (VV成為)(NP-OBJ (NP (NP-PN (NR中國))(NP (NN外貿)))(ADJP (JJ重要))(NP (NN增長點))))) )</S>
11 December 2016109

CopeOpi - intro• Unsupervised Chinese Sentiment scoring tool
• Dictionary: ANTUSD• Language: Traditional Chinese• Preprocessing– Tokenization– POS tagging (CKIP format)

11 December 2016110

CopeOpi – empirical usage
支持

支持

核四

Support nuclear power ,supportLungmen nuclear power plant

VC

VC

Nc

•

核能

，

Na

COMMACATEGORY

，

享受

相對

便宜

,

enjoy

relatively

cheaper

VH

VH

COMMAVJ
CATEGORY

11 December 2016

的

電價

。

power rate .
DE Na

PERIODCATEGORY

111

CopeOpi – empirical usage
支持核能Support
0.0381147

•

，

支持

核四

nuclear power ,supportLungmen nuclear power plant

0.0

0.0381147

0.0

0.0

，

享受

相對

便宜

,

enjoy

relatively

cheaper

0.0

0.0340755-0.042713-0.3732

的電價。

power rate .
0.0 0.0

0.0Document Score = 0.0675917
11 December 2016112

CopeOpi – transition process+

Supportive Word,
embracing nuclear power plant

Post

Neutral Word,
nuclear power,
nuclear power electricity ,Unsupportive Word,
anti-nuclear,abandon nuclear power plant

CopeOpi+-

Score = Sup-Uns+Neu
11 December 2016113

CopeOpi• Package including– CopeOpi program, written in Java– CopeOpi source code
– ANTUSD– A demo text
– Read me

11 December 2016114

CopeOpi - package• dic: dictionary files• out: output folder• CopeOpi.class (.java): interface• OpinionCore_Enhanced.class (.java): core
• readme.txt: readme file• file.lst: input file list
• test.txt: example input file• run.sh: running script
11 December 2016115

CopeOpi – example• $ ./run.sh– Run the CopeOpi with the files in the list “file.lst”
test.txt 0001

• Check the results in out/0001.txt

11 December 2016116

CopeOpi – example• Result summary in ./out.csv

11 December 2016117

CopeOpi – reference• CopeOpi– Ku, L. W., Ho, H. W., & Chen, H. H. (2009). Opinion mining andrelationship discovery using CopeOpi opinion analysis system. Journalof the American Society for Information Science and Technology, 60(7),1486-1503.

• CopeOpi with transition process– Chen, W. F., Ku, L. W., & Lee, Y. H. (2015). Mining Supportive andUnsupportive Evidence from Facebook Using Anti-Reconstruction of
the Nuclear Power Plant as an Example. In 2015 AAAI Spring
Symposium Series.
•

http://doraemon.iis.sinica.edu.tw/coling2016_tutorial/downloads/CopeOpi_EnhancedVersion.zip

11 December 2016

118

UTCNN - intro• Aim– Stance Classification on Social Media• Features– Information of social network platforms• Authorship• Likings• Topics• Comments

11 December 2016

119

UTCNN - data
•

Field

Author andliker IDs

Topic IDs Label

Delimiter

space

space

Tokenize

11 December 2016

Content Commenters
space
space

Comments
comma
space

120

UTCNN - data
• 3 46 57 … 573
49 61 4-1
<sssss>福 島 核電廠 的熔 毀 核 燃料棒 到底 有沒有 掉到 地下水層 …..<sssss>詳見 俄國 時報 電視 專訪 <sssss>
544 490 565 … 428
危機 ,如果 安全 你 家 借放 ,事實 是 沒有 人 知道 真相 這
些 都 只是 推論 就 看 誰 的 推論 有 根據 合理 奇怪 的 是
擁核 五 毛 只 根據 東京 電力 的 說法 而 東京 電力 是 最有 利益 關係 最 有 企圖 掩藏 事實 的 事主 貼 此 文 是 提供 大家 獨立 沒有 核電 利益 纏身 的 核工 專家 與 小出裕
章 的 推論 僅 供 參考

11 December 2016

121

UTCNN - package
•

•
•
•
•
•

dataset: data required for this tutorial– data.train– data.dev– data.test– data.readme– vectors.50d.txt
h5: parameters saved here
pickle: results saved here
config.ini: configuration file
UTCNN_release.py: main program
readme: readme file

11 December 2016

122

UTCNN - example• Package including– UTCNN model, written in python– Chinese word embeddings by GloVe– Demo data
• 1000 training samples• 100 development samples• 100 testing samples

11 December 2016

123

UTCNN - example• $ python UTCNN_release.py config.ini

11 December 2016

124

UTCNN - example• Parameters: ./h5/– Best: UTCNN_best.h5– Others: UTCNN_itr[00].h5• Prediction results: ./pickle/predict.pickle
11 December 2016

125

UTCNN - example• config.ini

11 December 2016

126

UTCNN - demo

11 December 2016

127

UTCNN - demo

http://doraemon.iis.sinica.edu.tw/wordforce/

11 December 2016

128

UTCNN - demo

http://doraemon.iis.sinica.edu.tw/wordforce/

11 December 2016

129

UTCNN - reference• Wei-Fan Chen and Lun-Wei Ku. (2016). UTCNN: a Deep
Learning Model of Stance Classification on Social Media Text.
In COLING 2016 main track.
• Wei-Fan Chen, Fang-Yu Lin and Lun-Wei Ku. (2016).
WordForce: Visualizing Controversial Words in Debates. In
COLING 2016 demo track.
•

http://doraemon.iis.sinica.edu.tw/coling2016_tutorial/downloads/UTCNN_release_161114.zip

11 December 2016130

Conclusion• Chinese sentiment dictionaries• Lexicon-based and deep learning-basedmodels for sentiment analysis• The utilization of these resources and tools

11 December 2016131

Final Wrap Up• Basic concepts of sentiment analysis andChinese text processing• Introduction of CSentiPackage• Hand-on CSentiPackage
Now you should be able to work with yourChinese texts and detect sentiment from them!
11 December 2016132

Something Important About
CSentiPackage• CSentiPackage you obtained here is only for yourgroup to use for the research purpose.
• Part of it has been officially released so they can be
downloaded any time.
• To obtain the other, join the next CSentiPackage
tutorial or check what’s new @http://academiasinicanlplab.github.io/
11 December 2016133

Join Our Three Demos Here
December 15th,10:30–12:30 Demo Session 3
1. Sensing Emotions in Text Messages: An Application and
Deployment Study of EmotionPush
December 16th,14:00–15:30 Demo Session 6
2. WordForce: Visualizing Controversial Words in Debates
3. Automatically Suggesting Example Sentences of NearSynonyms for Language Learners

11 December 2016

134

from
Lun-Wei Ku & Wei-Fan ChenNLPSA Lab, Academia Sinica
11 December 2016135