The 1st module (A1, A2) performs the pre‐processing andlinguistic analysis of each sentence of the text. It is in turn
composed of two blocks.
The first block extracts the document information that ispertinent to each topic: identifier, title and content ofeach sentence.
The second block filters out the stopwords and performs
the stemming of the resulting terms. Freeling and Mecab
have been used for stemming.

MOAT poses interesting challenges for us in two research aspects:
On one hand, there exist different techniques focused on the sentiment analysis problem. Some of them are based on
extracting the subjectivity of a given text by means of the application of semantic tagging and then performing a probabilisticclassification with the Bayes algorithm or similar. Other methods try to build supervised classification models based on different
lexical and syntactic features that are present in the text.
On the other hand,, East Asian languages
g g have some differential characteristics with respect
p to European
planguages
g g which makethem very appealing: a complex writing system made up of a mixture of scripts, a morphological structure which makes it hardto perform an accurate segmentation and conflation, lack of a standard orthography and the presence of numerous
orthographic variants, etc.
The main idea behind our approach to MOAT is to research on statistical sentiment classification techniques based on semanticfeatures extracted from the text. These techniques are in nature independent of the lexical and syntactical knowledge about thespecific language. Our objective is to compare their performance when applied to European and East Asian languages and draw
some conclusions about the results.
We submitted runs for monolingual English and Japanese in (1) the main mandatory opinionated sentence judgment subtask andthe optional (2) relevance and (3) polarity judgment subtasks.

The 2nd module (A3, A4) is the core of the system andincludes a supervised classification model associated to eachof the previously described subtasks.
The 3rd module (A5) is in charge of collecting the outputs ofeach classification block, and, based on these values,
determining the most adequate global output value.

The Semantic Knowledge Base contains a set of terms (specifically, nouns, adjectives and verbs) annotated with their semanticorientation (positive/negative polarity and normal/strong degree).
Built up from the linguistic resources provided by General Inquirer1:For English, 4,543 terms were directly extracted from the “Positive”, “Negative”, “Strong“ and “Weak” categories of Harvard IV‐4
dictionary.
For Japanese, English terms have been machine‐translated using JMDict2, resulting in over 55,000 terms.

not
In charge of deciding whether a sentence expresses an opinion or not,and, for those sentences that in fact express an opinion, annotating theirpolarity (positive, neutral or negative).
The global polarity annotation of a given sentence is based on aweighted semantic addition of the individual polarities that have been
identified for each fragment (opinion unit) that is part of the wholesentence (the average value):First, terms are semantically tagged using a Semantic Knowledge
Base (SKB).
Then, a weighted vector is built for each opinion unit, representing
the term frequency of the main opinionated features (i.e., SKBterms).
Finally, two k‐Nearest ‐Neighbour classifiers decide whether thesentence is opinionated or not, and indicate the opinion polarity.
Classifiers are trained using the feature matrix composed of thevectors of the sentences in the corpus provided in the NTCIR‐6Opinion Analysis Pilot task.

PEnglish
terms
1158

Negative term

N

1289

Positive terms implying strength

P+

443

358

Positive terms implying weakness

P‐1713005

Negative Strong

Negative terms implying strength

N+

273

3839

Negative Weak

Negative terms implying weakness

N‐

419

4878

Category

Description

Positive

Positive term

Negative
Positive Strong
Positive WeakTagJapanese
terms
13111
5816

Table1: Semantic Knowledge Base (SKB)
1 General Inquirer <www.wjh.harvard.edu/~inquirer>
2 JMDict: Japanese Multi‐lingual Dictionary <www.csse.monash.edu.au/~jwb/jmdict.html>

Our approach to solve the relevance judgment problem is based on the semantic
distance between the vectors that represent sentences and topics. If the distanceis lower than a given threshold, the sentence is positively classified as relevant(and, on the contrary, as non‐relevant):
A weighted vector is built for each sentence and topic, representing the term
frequency of the main most significant terms.
Then, a new vector is built with the degree of semantic similarity (the cosinedistance)) amongg those vectors (sentences(and topic).
p )Finally, a rule‐based classifier (tree‐chart) is used to find the optimum
discrimination threshold for the semantic similarity between topic and thegiven fragment.
normTerms <= 0,8333 [Mode: N ] (support:707)
similarity <= 0,0514 [Mode: N ] => N (support:680; 0,938%)
similarity > 0,0514 [Mode: N ] => N (support:27; 0,704%)
normTerms > 0,8333 and normTerms <= 1,4000 [Mode: N ] (support:791)
similarity <= 0 [Mode: N ] => N (support:714; 0,849%)
similarity > 0 [Mode: N ] => N (support:77; 0,61%)
normTerms > 1,400 and normTerms <= 2,2500 [Mode: N ] (support:1.358)
similarity <= 0,0514 [Mode: N ] => N (support:1.186; 0,718%)
similarity > 0,0514 [Mode: Y ] => N (support:172; 0,5%)

Lenient
StrictLenient
Relevance
StrictOpinionatedPrecision
0.595
0.286
0.374
0.085

Recall
0.012
0.012
0.319
0.304

F‐Measure
0.023
0.022
0.344
0.133

Table 3: Results for English Language

Lenient
StrictLenient
Relevance
StrictLenient
Polarity
StrictOpinionatedPrecision
0.316
0.241
0.455
0.223
0.247
0.240

Recall
0.090
0.094
0.082
0.082
0.018
0.017

F‐Measure
0.140
0.135
0.138
0.120
0.034
0.031

Table 4: Results for Japanese Language

Values for polarity judgement in English were notsignificant due to a bug in the polarity classifier.
We wrongly considered that a sentence would be judged as
relevant or not‐relevant independently of the fact that it
was opinionated or non‐opinionated (as in NTCIR‐6). Table
5 shows an adapted relevance evaluation for Japanese,
kindly provided by the task organizers.
Results are not good, mainly because recall is very low.

Relevance

Lenient

Precision
0.136

Recall
0.864

F‐Measure
0.234

Table 5: Evaluation of relevance, for Japanese, considering
all sentences

Table 2: Excerpt of the rule set for relevance classification

EnglishJapanese

MIRACLE RunOpinionatedYN
0.97% 99.03%POS0.00%

Polarity
NEG
NEU
16.67% 83.33%

Relevance
YN
27.03% 72.97%

Lenient NTCIR‐7

25.20%

74.80%

25.00%

48.00%

6.00%

99.40%

0.60%

MIRACLE Run

7.76%

92.24%

10.69%

68.76%

20.55%

78.40%

21.60%

Lenient NTCIR‐7

28.90%

71.10%

5.50%

15.30%

79.20%

43.20%

56.80%

Table 6: Opinion percentage in NTCIR‐MOAT test collection and MIRACLE runs

Our runs generate a high percentage of “No” classes for both opinionated and relevance judgments, i.e., tend to answer “Non‐opinionated” and “Non‐relevant” for the majority of the sentences.
This fact is derived from the unbalanced distribution of each class in the training data (NTCIR‐6 corpus)
Especially noticeable in the case of relevance judgment: only 27.41% of the NTCIR‐6 examples belong to the “Yes” class,whereas in the NTCIR‐7 judgment set, 99.40% of the instances are “Yes”.
Regarding the polarity, results for English are conditioned by the default rule of the polarity classifier, which returns “NEU” if facedwith ambiguous or unknown situations
Results of the opinionated judgment for English language are better than those for Japanese.
This can be explained because the Semantic Knowledge Base for Japanese contains a remarkably higher number of terms, andthus more terms are semantically tagged. The reason is that the translation process from English adds additional terms(synonyms or related words) apart from the literal word‐by‐word translation, that inherit the same semantic tags (Positive,Negative, Strong and/or Weak) as the original English terms, which introduce ambiguity (noise) in the classification step.
Some problems were also observed for a considerable number of semantically‐tagged terms, due to the lack of a disambiguation
process:
For example, the English verb “to increase” was tagged both “Positive” (increase happiness) and “Negative” (increaseunemployment), which may mean added noise to the classification step.

As this was our first participation in a task focused on sentiment analysis, our main effort was mainly dedicated to study and learn
the basics of the techniques and the distinctive semantic features of the languages involved and also to build the entire necessary
linguistic infrastructure to be able to submit our experiments in time.
Results of our runs are disappointing as compared to the results of the other participants.
However, starting from scratch is always difficult, so there are many aspects that we could not address in our runs this year, due toevident limitations of computing resources, time and expertise.
We are interested in going on with this line of research and continue our participation in future editions of MOAT.
More effort has to be invested in improving the preprocessing step (better stemmers and stopword lists) and also the Semantic
Knowledge Base used for tagging, specifically resources for Japanese. The translation from English to Japanese produces many
Japanese terms that are semantically‐related to the original English term, and this fact seems to cause trouble in the classification
process. Thus, a filtering process should be considered.
In addition, classifiers must be completely redesigned, improving the training dataset and considering the uneven distribution of
the output classes
classes. Also,
Also some semantic disambiguation techniques could also be applied
applied, perhaps using similar techniques as forautomatic topic semantic expansion in a general‐purpose information retrieval context.

Yohei Seki, David Kirk Evans, Lun‐Wei Ku, Le Sun, Hsin‐Hsi Chen and Noriko Kando. Overview of Multilingual Opinion Analysis Task at
NTCIR‐7, National Institute of Informatics, October 24, 2008.
Yohei Seki, David Kirk Evans, Lun‐Wei Ku, Hsin‐Hsi Chen, Noriko Kando and C. Y. Lin. Overview of Opinion Analysis Pilot Task at
NTCIR‐6. In Proc. of the Sixth NTCIR Workshop Meeting on Evaluation of Information Access Technologies: Information Retrieval,Question Answering, and Cross‐Lingual Information Access, pages 265–278, NII, Japan, May 2007.
Julio Villena‐Román, Sara Lana‐Serrano, José C. González‐Cristóbal. MIRACLE at NTCIR‐7 MOAT: First Experiments on MultilingualOpinion Analysis. In Proc. of the Seventh NTCIR Workshop Meeting on Evaluation of Information Access Technologies: Information
Retrieval, Question Answering, and Cross‐Lingual Information Access, December 16‐19, 2008, NII, Tokyo, Japan.

MIRACLE team is a Spanish R&D consortium formed by research groups of three different universities in Madrid (UPM1, UAM2 and UC3M3) along with DAEDALUS4, asmall/medium size enterprise founded in 1998 as a spin‐off of two of these groups and a leading company in the field of linguistic technologies in Spain.
Our area of research is focused on Natural Language Processing and Multilingual and Multimedia Information Retrieval.
We have participated in CLEF5 (Cross Language Evaluation Forum, the European homologue of NTCIR), since 2003 in many different tasks, including bilingual,monolingual and cross lingual retrieval, image, video, web and geographic information retrieval, question answering and interactive task, and this is our 2ndparticipation in NTCIR.
1 Universidad Politécnica de Madrid <www.upm.es>
2 Universidad Autónoma de Madrid <www.uam.es>
3 Universidad Carlos III de Madrid <www.uc3m.es>
4 DAEDALUS <www.daedalus.es>
5 CLEF <www.clef‐campaign.org>

This work has been partially supported by the Spanish R+D National Plan, project BRAVO (Multilingual and Multimodal Answers Advanced Search – Information Retrieval), TIN2007‐67407‐C03‐
03 and by Madrid R+D Regional Plan, project MAVIR (Enhancing the Access and the Visibility of Networked Multilingual Information for the Community of Madrid), S‐0505/TIC/000267.