Aligning Opinions: Cross-Lingual Opinion Mining with Dependencies
Mariana S. C. Almeida∗†
Cláudia Pinto∗
Helena Figueira∗Pedro Mendes∗
André F. T. Martins∗†∗
Priberam Labs, Alameda D. Afonso Henriques, 41, 2o , 1000-123 Lisboa, Portugal†
Instituto de Telecomunicações, Instituto Superior Técnico, 1049-001 Lisboa, Portugal{mla,atm}@priberam.pt

Abstract
We propose a cross-lingual framework forfine-grained opinion mining using bitext
projection. The only requirements are arunning system in a source language andword-aligned parallel data. Our method
projects opinion frames from the source tothe target language, and then trains a system on the target language using the automatic annotations. Key to our approach isa novel dependency-based model for opinion mining, which we show, as a byproduct, to be on par with the current stateof the art for English, while avoiding theneed for integer programming or reranking. In cross-lingual mode (English to Portuguese), our approach compares favorably to a supervised system (with scarcelabeled data), and to a delexicalized modeltrained using universal tags and bilingual
word embeddings.

1

Introduction

The goal of opinion mining is to extract opinions
and sentiments from text (Pang and Lee, 2008;
Wilson, 2008; Liu, 2012). With the advent of social media and the increasing amount of data available on the Web, this has become a very active
area of research, with applications in summarization of customer reviews (Hu and Liu, 2004; Wu etal., 2011), tracking of newswire and blogs (Ku etal., 2006), question answering (Yu and Hatzivassiloglou, 2003), and text-to-speech synthesis (Almet al., 2005).
While early work has focused on determining
sentiment at document and sentence level (Panget al., 2002; Turney, 2002; Balog et al., 2006),
research has gradually progressed towards finegrained opinion mining, where rather than determining global sentiment, the goal is to parse text

into opinion frames, identifying opinion expressions, agents, targets, and polarities (Ding et al.,2008), or addressing compositionality (Socher etal., 2013b). Since the release of the MPQA corpus1 (Wiebe et al., 2005; Wilson, 2008), a standard corpus for fine-grained opinion mining ofnews documents, a long string of work has been
produced (reviewed in §2). Despite the large volume of prior work, opinion mining has by andlarge been limited to monolingual approaches inEnglish.2 This is explained by the heavy effort
of annotation necessary for current learning-based
approaches to succeed, which delays the deployment of opinion miners for new languages.
We bridge the existing gap by proposing across-lingual approach to fine-grained opinion
mining via bitext projection. This technique has
been quite effective in several NLP tasks, suchas part-of-speech (POS) tagging (Täckström etal., 2013), named entity recognition (Wang andManning, 2014), syntactic parsing (Yarowsky andNgai, 2001; Hwa et al., 2005), semantic role labeling (Padó and Lapata, 2009), and coreference resolution (Martins, 2015). Given a corpus of parallel
sentences (bitext), the idea is to run a pre-trained
system on the source side and then to use wordalignments to transfer the produced annotations to
the target side, creating an automatic training corpus for the impoverished language.
To alleviate the complexity of the task, westart by introducing a lightweight representation—
called dependency-based opinion mining—and
convert the MPQA corpus to this formalism (§3).
We propose a simple arc-factored model that permits easy decoding (§4) and we show that, despite1
http://mpqa.cs.pitt.edu/corpora/mpqa_
corpus.
2
Besides English, monolingual systems have also beendeveloped for Chinese and Japanese (Seki et al., 2007), German (Clematide et al., 2012) and Bengali (Das and Bandyopadhyay, 2010).

its simplicity, this model is on par with state-ofthe-art opinion mining systems for English (§5).
Then, through bitext projection, we transfer thesedependency-based opinion frames to Portuguese(our target language), and train a system on the
resulting corpus (§6).
As part of this work, a validation corpus in Portuguese with subjectivity annotations was created,along with a translation of the MPQA Subjectivity lexicon of Wilson et al. (2005).3 Experimental
evaluation (§7) shows that our cross-lingual approach surpasses a supervised system trained ona small corpus in the target language, as well as adelexicalized baseline trained using universal POS
tags, bilingual word embeddings and a projected
lexicon.

2

Related WorkA considerable amount of work on fine-grained
opinion mining is based on the MPQA corpus.
Kim and Hovy (2006) proposed a method for finding opinion holders and topics, with the aid of a semantic role labeler. Choi et al. (2005) and Breck etal. (2007) used CRFs for finding opinion holders
and recognizing opinion expressions, respectively.
The two things are predicted jointly by Choi et al.
(2006), with integer programming, and Johansson
and Moschitti (2010), via reranking. The same
method was applied later for joint prediction of
opinion expressions and their polarities (Johansson and Moschitti, 2011). The advantage of a
joint model was also shown by Choi and Cardie(2010) and Yang and Cardie (2014). Yang andCardie (2012) classified expressions with a semiMarkov decoder, outperforming a B-I-O tagger; inlater work, the same authors proposed an ILP decoder to jointly retrieve opinion expressions, holders, and targets (Yang and Cardie, 2013). A more
recent work (İrsoy and Cardie, 2014) proposes arecurrent neural network to identify opinion spans.
All the approaches above rely on a span-based
representation of the opinion elements. This
makes joint decoding procedures more complicated, since they must forbid overlap of opinion
elements or add further constraints, leading to integer programming or reranking strategies. Besides, there is little consensus about what shouldbe the correct span boundaries, the inter-annotator
agreement being quite low (Wiebe et al., 2005). In

constrast, we use dependencies to model opinion
elements and relations, leading to a compact representation that does not depend on spans and which
is tractable to decode. A dependency scheme wasalso used by Wu et al. (2011) for fine-grained
opinion mining. Our work differs in which we
mine opinions in news articles instead of productreviews, a considerably different task. In addition,
the approach of Wu et al. (2011) relies on “spannodes” (instead of head words), requiring solving
an ILP followed by an approximate heuristic.
Query-based multilingual opinion mining wasaddressed in several NTCIR shared tasks (Sekiet al., 2007; Seki et al., 2010).4 However, toour best knowledge, a cross-lingual approach has
never been attempted. Some steps were taken by
Mihalcea et al. (2007) and Banea et al. (2008),
who translated an English lexicon and the MPQA
corpus to Romanian and Spanish, but for the much
simpler task of sentence-level subjectivity analysis. Cross-lingual sentiment classification wasaddressed by Wan (2009), Prettenhofer and Stein(2010) and Wei and Pal (2010) at document level,and by Lu et al. (2011) at sentence level. Recently,
Gui et al. (2013) applied projection learning for
opinion mining in Chinese. However, this work
only addresses agent detection and requires translating the MPQA corpus. While all these works
are relevant, none addresses fine-grained opinion
mining in its full generality, where the goal is topredict full opinion frames.

3

Dependency-Based Opinion Mining

This work addresses various elements of subjectivity annotated in the MPQA corpus, namely:• direct-subjective expressions (henceforth, opinions) that are direct mentions of a private state,e.g. opinions, beliefs, emotions, sentiments,
speculations, goals, etc.;• the opinion agent, i.e., the holder of the opinion;• the opinion target, i.e., what is being argued
about;• the opinion polarity, i.e., the sentiment (positive, negative or neutral) towards the target.
As an example, consider the sentence in Figure 1, which has two opinions, expressed by the
43

The Portuguese corpus and the lexicon are available at
http://labs.priberam.com/Resources.

NTCIR-8 had a cross-lingual track but in a very different sense: there, queries and documents are in different languages; in contrast, we transfer a model accross languages.

spans “is believed” (O1 ) and “are against” (O2 ).
The first opinion has an implicit agent and a neutral polarity toward the target “the rich elites”(T1 ). This target is also the agent (A2 ) of the second opinion, which has a negative polarity toward“Hugo Chávez” (T2 ).
3.1

Motivation

As noted in prior work (Choi et al., 2005; Kim andHovy, 2006; Johansson and Moschitti, 2010), one
source of difficulty when learning opinion miners on MPQA is with the boundaries of the entity spans. The fact that no criterion for choosing
these boundaries is explicitly defined in the annotation guidelines (Wiebe et al., 2005) leads to a
low inter-annotator agreement. To circumvent thisproblem and make the learning task easier, we depart from the classical span-based approaches toward dependency-based opinion mining. Thisdecision is inspired by the success of dependency
models for syntax and semantics (Buchholz andMarsi, 2006; Surdeanu et al., 2008). These dependency relations can be further converted to opinion
spans (as described in §3.3), or directly used asfeatures in downstream applications. As we willsee, a compact representation based on dependencies can achieve state-of-the-art results and has theadvantage of being easily transferred to other languages through a parallel corpus.
3.2

Dependency Graph

Figure 1 depicts a sentence-level dependency representation for fine-grained opinion mining. The
overall structure is a graph whose nodes are headwords (plus two special nodes, root and null),connected by labeled arcs, as outlined below.
Determining head nodes. The three opinion elements that we want to detect (opinions, agentsand targets) are each represented by a head node,
which corresponds to a single word (underlined in
Figure 1). When converting the MPQA corpus
to dependencies, we determine this “representative” word automatically, by using the following
simple heuristic: we first parse the sentence using the Stanford dependency parser (Socher et al.,2013a); then, we pick the last word in the span
whose syntactic parent is outside the span (if the
span is a syntactic phrase, there is only one word
whose parent is outside the span, which is the lexical head). The same heuristic has been used for

identifying the heads of mention spans in coreference resolution (Durrett and Klein, 2013).
Defining labeled arcs. The opinion relations are
represented as labeled arcs that link these headnodes. Two artificial nodes are added: a root
node, which links to all nodes that represent opinion words, with the label OPINION; and a null
node, which is used for representing implicit relations. To represent opinion-agent relations, wedraw an arc labeled AGENT toward the agent word.
For opinion-target relations, the arc is toward thetarget word and has one of the labels TARGET:0,
TARGET:+, or TARGET:-; this encodes the polarity
in addition to the type of relation. We also include
implicit arcs for opinion elements whose agent or
target is not mentioned inside the sentence—theseare modeled as arcs pointing to the null node.
Dependency opinion graph. We have the following requirements for a well-formed dependency opinion graph:
1. No self-arcs or arcs linking root to null.
2. An arc is labeled as OPINION if and only if itcomes from the root node.
3. Arcs labeled as AGENT or TARGET must comefrom an opinion node (i.e., a node with an incoming OPINION arc).
4. Every opinion node has exactly one AGENT and
one TARGET outgoing arcs (possibly implicit).5
Similarly to prior work (Choi and Cardie, 2010;
Johansson and Moschitti, 2011; Johansson andMoschitti, 2013), we map the MPQA’s polarityinto three levels: positive, negative and neutral,
where the latter includes spans without polarity
annotation or annotated as “both”. As in Johansson and Moschitti (2013), we also ignore the “uncertain” aspect of the annotated polarities.
3.3

Dependency-to-Span Conversion

To evaluate the opinion miner against manual annotations and compare with other systems, weneed a procedure to convert back from predicted
dependencies to spans. In this work, we useda very simple procedure that we next describe,
5

Even though this assumption is not always met in practice, it is typical in MPQA (only 10% of the opinions havemultiple agents, typically coreferent; and only 13% havemultiple targets). When multiple agents or targets exist, we
keep the ones that are closest to the opinion expression.

Figure 1: Example of an opinion mining graph in our dependency formalism. Heads are underlined.
which assumes the sentence was previously parsed
using a syntactic dependency parser.
To generate agent and target spans, we computethe largest span, containing the head word, whose
words are all descendants in the dependency parsetree and that are, simultaneously, not punctuations.
To generate opinion spans, we start with the head
word and expand the span by adding all neighbouring verbal words. In the case of English, we
also allow adverbs, adjectives, modal verbs andthe word to, when expanding to the left.
The application of this simple approach to thegold dependency graphs in the training partitionof the MPQA leads to oracle F1 scores of 86.0%,
95.8% and 93.0% in the reconstruction of opinion,
agent and target spans, respectively, according to
the proportional scores described in §5.2.

4

Arc-Factored Model

Under the assumptions stated in §3, this problem
decouples into independent maximization problems (one for each possible opinion word in thesentence). The detailed procedure is as follows,
where arcs a can take the form o → h (opinion toagent) and o → t (opinion to target). For every
candidate opinion word o:
1. Obtain the most compatible agent word, bh :=arg maxh fo→h (x, AGENT);
2. Obtain the best target word and its polarity,(bt, pb) := arg maxt,p fo→t (x, TARGET:p);
3. Compute the total score of this candidate
opinion as so := froot→o (x, OPINION) +fo→bh (x, AGENT) + fo→bt (x, TARGET:bp). Then,
if so ≥ 0, add the arcs root → o, o → bh, andbo → t to the dependency graph, respectively
with labels OPINION, AGENT, and TARGET:bp.

One of the advantages of the dependency representation is that we can easily decode opinion-agenttarget relations without the need of complicatedconstrained sequence models or integer programming, as done in prior work (Choi et al., 2006;
Yang and Cardie, 2012; Yang and Cardie, 2013).

For a sentence with L words, this decoding procedure takes O(L2 ) time. In practice, we speed
up this process by pruning from the candidate list
arcs whose connected POS were not observed in
the training set and whose length were larger thanthe ones observed in the training set.

4.1

4.2

Decoding

We model dependency-based opinion mining as astructured classification problem. Let x be a sentence and y ∈ Y(x) a set of well-formed dependency graphs, according to the constraints statedin §3. We define a score function that decomposes
as a sum of labeled arc scores,Xf (x, y) =fa (x, ya )(1)a∈ywhere ya is a labeled arc and the sum is over thearcs of the graph y. We use a linear model withweight vector w and local features φa (x, ya ):fa (x, ya ) = w · φa (x, ya ).

(2)

For making predictions, we need to computeyb = arg max f (x, y).
y∈Y(x)(3)

Features

We now describe our features φa , which arecomputed after processing the sentence to predict
POS tags, syntactic dependency trees, lemmas andvoice (active or passive) information. For English,
we used the Stanford dependency parser (Socher
et al., 2013a) for the syntactic annotations, thePorter stemmer to compute word stems, and a setof rules for computing the voice of each word. Our
Portuguese corpus include all these preprocessing
elements (§6.3), with the exception of the voice information (features depending on voice were onlyused for English).
We also used the Subjectivity Lexicon6 of Wilson et al. (2005) that we translated to Portuguese6
http://mpqa.cs.pitt.edu/lexicons/
subj_lexicon/(§6.3), and a set of negation words (e.g. not, never,nor) and quantity words (e.g. very, much, less)collected for both languages.
Our arc-factored features are described below;
they are inspired by prior work on dependency
parsing (Martins et al., 2013) and fine-grained
opinion mining (Breck et al., 2007; Johansson andMoschitti, 2013).
Opinion features. We define a set of featuresthat only look at the opinion word; special symbols are used if the opinion is connected to a rootor null node. The features below are also conjoined with the arc label.
• O PINION W ORD. The word itself, the lemma,
the POS, and the voice. Conjunction of the wordwith the POS, and of the lemma with the POS.
• B IGRAMS. Bigrams of words and POS corresponding to the opinion word conjoined with its
previous (and next) word.
• L EXICON ( BASIC ). Conjunction of the strengthand polarity of the opinion word in the Subjectivity Lexicon6 (e.g., “weaksubj+neg”).
• L EXICON ( COUNT ). Number of subjective
words (total, positive and negative) in a sentence, with and without being conjoined withthe polarity of the opinion word in the lexicon.
• L EXICON ( CONTEXT ). For each word that is in
the lexicon and within the 4-word context of theopinion, the form and the polarity of that wordin the lexicon, with and without being conjoinedwith the form and the polarity in the lexicon of
the opinion word. Besides the 4-word context,
we also used the next/previous word in the sentence which is in the lexicon.
• N EGATION AND Q UANTITY W ORDS . Within
the 4-word context, features indicating if a word
is a negation or quantity word, conjoined withthe word itself and the opinion word.
• S YNTACTIC PATH . The number of words up to
the top of the syntactic dependency tree, and thesequence of POS tags in that path.
Opinion-Argument features. In case of arcs
that neither connect to null nor root, the features above are also conjoined with the binned distance between the two words.For these arcs, we
did not use the LEXICON ( COUNT )/( CONTEXT )features, but we added features regarding the pairof opinion-argument words (below).

• O PINION -A RGUMENT W ORD PAIR. Several
conjunctions of word form, POS, voice and syntactic dependency relations corresponding to thepair opinion-argument.
• O PINION -A RGUMENT S YNTACTIC PATH. The
syntactic path from the opinion word to the argument, conjoined with the POS and the dependency relations in the path (in Figure 1, forthe agent “elites” headed by “are” with relation
nsuj, we have: “VBP↓NNS” and “nsuj↓”).
For arcs that neither connect to null or root,
we conjoin voice features with the label, distance,
and the direction of the arc. For these arcs, we
also include back-off features where the polarity
information is removed from the (target) labels.

5

English Monolingual Experiments

In a first set of experiments, we evaluated theperformance of our dependency-based model for
opinion mining (§3) in the MPQA English corpus.
5.1

Learning

We trained arc-factored models by running 25
epochs of max-loss MIRA (Crammer et al., 2006).
Our cost function takes into account mismatches
between predicted and gold dependencies, witha cost CP on labeled arcs incorrectly predicted(false positives) and a cost CR = 1 − CP on
missed gold labeled arcs (false negatives). The
cost CP , the regularization constant, and the number of epochs were tuned in the development set.
5.2

Evaluation Metrics

Opinion spans (Op.) are evaluated with F1 scores,
according to two matching criteria commonlyused in the literature: overlap matching (OM),
where a predicted span is counted as correct if
it overlaps a gold one, and proportional matching (PM), proposed by Johansson and Moschitti(2010). For the latter, we use the following formula for the recall, where we consider the sets ofgold (G) and predicted (P) opinion spans:7
TX|g p|/|p|R(G, P) =
max
;
(4)
g∈G
|P|
p∈P

7

This metric is slightly different from the PM metric of Johansson and Moschitti (2010), in which recall was computedPP
as R(G, P) = p∈P g∈G |g∩p|/|p|. The reason why we|P|replace the “sum” by a “max” is that each predicted span p in(4) could contribute to the recall with a value greater than 1.
Since most of the predicted spans only overlap a single gold
span, this fix has a very small effect in the final scores.

the precision is P (G, P) = R(P, G). We also report metrics based on a head matching (HM) criterion, where a predicted span is considered correct if its syntactic head matches the head of thegold span. We consider that a pair opinion-agent(Op-Ag.) or opinion-target (Op-Tg.) is correctly
extracted according to the OM or the HM criteria,
if both the elements satisfy these criteria and therelation holds in the gold data. We also computethe metric described in Johansson and Moschitti(2010) which measures how well agents of opinions are predicted based on a proportional matching (PM) criterion. This metric is applied to evaluate the extraction of both agents and targets. Finally, to evaluate the opinions’ polarities (Op-Pol.
metric) we consider as correct opinions where thespan and polarity both match the gold ones.
5.3

show that our dependency-based approach (§3.2)
followed by a simple dependency-to-span conversion (§3.3) is, despite its simplicity, on par witha top-performing opinion mining system. We conjecture that this is due to the ability to extract opinions, agents, and targets jointly using exact decoding. Note that our proposed dependency scheme
would also be able to include additional global features relating pairs of opinions (by adding scores
to pairs of opinion arcs) or two opinions having
the same agent (by adding scores to pairs of agent
arcs sharing its argument), similar to the reranking
features used by Johansson and Moschitti (2013).
Similar second-order scores have been used insyntactic and semantic dependency parsing (Martins et al., 2013; Martins and Almeida, 2014), butwith an increase in the complexity of the modeland of the decoder.

Results: Dependency-Based Model

We assess the quality of our monolingual
dependency-based model by comparing it to therecent state-of-the-art approach of Johansson andMoschitti (2013), whose code is available online.8
That paper reports the performance of a basic
span-based pipeline system (which extracts opinions with a CRF, followed by two separate classifiers to detect polarities and agents), and of a moresophisticated system that applies a reranking procedure to account for more complex features thatconsider interactions accross opinion elements.
We ran experiments using the same data and
MPQA partitions as Johansson and Moschitti(2013). However, since our system is designed forpredicting opinion, agents and targets together, weremoved the documents that were not annotatedwith targets. The final train/development/test sets
have a total of 6,774/1,404/2,559 sentences and3,834/881/1,426 opinions, respectively.
Table 1 reports the results; since the systemsof Johansson and Moschitti (2013) do not predict targets, Table 1 omits target scores.9 We observe that our dependency-based system achieves
results competitive with the best results of Johansson and Moschitti (2013) and clearly above theones reached by their basic system that does notuse re-ranking features. Though the two systems
are not fully comparable,10 the results in Table 18
http://demo.spraakdata.gu.se/richard/
unitn_opinion/details.html
9
We will report target scores later in §7.
10
Our system makes use of target annotations to predict
the opinion frames, while Johansson and Moschitti (2013)

6

Cross-Lingual Opinion Mining

We now turn to the problem of learning a opinion mining system for a resource-poor language(Portuguese), in a cross-lingual manner. We usea bitext projection approach (§6.1), whose only
requirements are a model for a resource-rich language (English) and parallel data (§6.2).
6.1

Bitext Projection

Our methodology is outlined as Algorithm 1. For
simplicity, we call the source and target languages
English (e) and “foreign” (f ), respectively. The
procedure is inspired by the idea of bitext projection (Yarowsky and Ngai, 2001). We start bytraining an English system on the labeled data Le(line 1), which in our case is the MPQA v.2.0 corpus. This system is then used to label the English
side of the parallel data, automatically identifying
opinion frames (line 2). The next step is to run
a word aligner on the parallel data (line 3). The
automatic alignments are then used to project theopinion frames to the target language (along withb (f )
some filtering), yielding an automatic corpus D(line 4), which finally serves to train a system forthe target language (line 5).
6.2

Parallel Data

We use an English-Portuguese parallel corpus
based on the scientific news Brazilian magazine
Revista Pesquisa FAPESP, collected by Aziz andhas access not only to direct subjective spans but also to subjective expressions annotations with their agents and polarity
information.

Op.
Op-Ag.
Op-Tg.
Op-Pol.

JM13, BASIC
HM
PM
OM
56.3 56.2 60.6
40.3 47.1 44.9
46.1 45.9 49.3

JM13, R ERANKING
HM
PM
OM
58.6
59.2
63.7
42.4
51.4
48.1
48.5 48.9* 52.5

HM
61.6*
45.7*
31.3*
47.9

O UR S YSTEM
PM
OM
59.8
65.1
51.4
50.3*
48.3* 48.3*
47.0
50.7

Table 1: Method comparison: F1 scores obtained in the MPQA corpus, for our dependency based method
and the approaches in Johansson and Moschitti (2013), with and without reranking. The symbol *indicates that the best system beats the other systems with statistical significance, with p < 0.05 andaccording to a bootstrap resampling test (Koehn, 2004).

Figure 2: Excerpt of a bitext document from FAPESP, with automatic opinion dependencies. The annotations are directly projected to Portuguese via automatic word alignments.
Algorithm 1 Cross-Lingual Opinion Mining
Input: Labeled data Le , parallel data De and Df .
Output: Target opinion mining system S f .
1: S e ← L EARN O PINION M INER(Le )b e ← RUN O PINION M INER(S e , De )2: D
3: D e↔f ← RUN W ORDA LIGNER(D e , D f )b f ← P ROJECTA ND F ILTER(De↔f , D
be)
4: D
f
f
b
5: S ← L EARN O PINION M INER(D )Specia (2011). Though this corpus is in Brazilian Portuguese (while our validation corpus is in
European Portuguese), we preferred FAPESP overother commonly used parallel corpora (such as theEuroparl and UN datasets), since it is closer toour newswire target domain, with a smaller prominence of direct speech. We computed word alignments using the Berkeley aligner (Liang et al.,2006), intersected them and filtered out all thealignments whose confidence is below 0.95.
After annotating the English side of FAPESPb e in Algorithm 1,
with the pre-trained system (Dwith a total of 166,719 sentences and 81,492 opinions), the high confidence alignments (De↔f ) areused to project the annotations to the Portugueseside of the corpus. The automatic annotations produced by our dependency-based system are easilytransferred at a word level (for words with highconfidence alignments), as illustrated in Figure 2.
To improve the quality of the resulting corpus,
we excluded sentences whose alignments coverless than 70% of the words in the target side of
the corpus, or sentences whose opinion elements
were not fully projected through high confidencealignments. At this point, we obtain an automatb f ), with
ically annotated corpus in Portuguese (D106,064 sentences and 32,817 opinions.
6.3

Portuguese Opinion Mining Corpus

For validation purposes, we also created a Portuguese corpus with manually annotated finegrained opinions. The corpus consists of a subset of the documents of the Priberam Compressive
Summarization Corpus11 (Almeida et al., 2014),
which contains 80 news topics with 10 documents
each, collected from several Portuguese newspapers, TV and radio websites in the biennia 2010–2011 and 2012–2013. In the scope of the current
work, we selected and annotated one document ofeach of the 80 topics. The first biennium was selected as the test set and the second biennium was
split into development and training sets (see Ta11
http://labs.priberam.com/Resources/
PCSC

ble 2 for statistics).

Train
Dev
Test

#doc.
20
20
40

#sent.
441
225
560

Op.
Op-Ag.
Op-Tg.
Op-Pol.

#opin.
240
197
391

Table 2: Number of documents, sentences andopinions in the Portuguese Corpus.

HM
77.0
69.1
61.9
49.4

PM
76.7
72.3
65.4
49.1

OM
79.2
73.5
71.4
50.7

Table 3: Inter-annotator agreement in the test partition (shown are F1 scores).
The corpus was annotated in a similar vein as
the MPQA (Wiebe et al., 2005), with the additionof the head node for each element of the opinion frame. It includes spans for direct-subjective
expressions with intensity and polarity information; agent spans; and target spans. The annotation
was carried out by three linguists, after reading the
MPQA annotation guidelines (Wiebe et al., 2005;
Wilson, 2008) and having a small practice periodusing the provided examples and some MPQA annotated sentences. Each document was annotatedby two of the three linguists and then revised by
the third linguist, who (in case of any doubts) discussed with the initial annotators to reach for the
final consensus. Scores for inter-annotator agreement are shown in Table 3.
The corpus was annotated with automatic
POS tags and dependency parse trees using
TurboParser (Martins et al., 2013).12 We used
an in-house lemmatizer to obtain lemmas foreach inflected word in the corpus. A Portuguese lexicon of subjectivity was created bytranslating the words in the Subjectivity Lexicon of Wilson et al. (2005). The annotatedcorpus and the translated subjectivity lexicon
are available at http://labs.priberam.com/
Resources/Fine-Grained-Opinion-Corpus,
and http://labs.priberam.com/Resources/
Subjectivity-Lexicon-PT, respectively.
12

http://www.ark.cs.cmu.edu/TurboParser

D ELEXICALIZEDHM PM OM
50.1 45.8 52.7
33.8 34.8 35.7
19.9 28.0 32.1
36.7 34.7 38.8

Table 4: F1 scores obtained in English (MPQA),
for our full system and the D ELEXICALIZED one.

7
Op.
Op-Ag.
Op-Tg.
Op-Pol.

O UR S YSTEMHM PM OM
65.7 63.5 69.8
47.6 48.8 51.1
34.9 44.8 50.3
51.5 50.2 54.4

Cross-Lingual Experiments

In a final set of experiments, we compare threesystems of fine-grained opinion mining for Portuguese. All were trained as described in §5.1.
7.1

System Description

Baseline #1: Supervised System. A S UPER VISED system was trained on the small Portuguese
training set described in §6.3. Though being asmall training corpus, this is, to the best of ourknowledge, the only existing corpus with finegrained opinions in Portuguese. We used the samearc-factored model and features described in §4.
Baseline #2: Delexicalized System with Bilingual Embeddings. This baseline consists of adirect model transfer: a D ELEXICALIZED system
is trained in the source language, without language specific features, so that it can be directly
applied to the target language. Despite its simplicity, this strategy managed to provide a fairly strong
baseline in several NLP tasks (Zeman and Resnik,
2008; McDonald et al., 2011; Søgaard, 2011).
To achieve a unified feature representation, we
mapped all language-specific POS tags to universal tags (Petrov et al., 2012), and removed allfeatures depending on the dependency relations,but maintained those depending on the syntactic
path (but not on the dependency relations themselves). In addition, we replaced the lexical features by 128-dimensional cross-lingual word embeddings.13 To obtain these bilingual neural embeddings, we ran the method of Hermann andBlunsom (2014) on the parallel data (§6.1). We
scaled the embeddings by a factor of 2.0 (selectedon the dev-set), following the procedure describedin Turian et al. (2010).
We trained the English delexicalized system on
the MPQA corpus, using the same test documents
13
A delexicalized system trained without the word embeddings had a worse performance.

Op.
Op-Ag.
Op-Tg.
Op-Pol.

BASELINE #1 (S UP.)
HM
PM
OM
49.4 48.7 50.8
23.5 27.2 31.5
23.0 24.9 30.6
24.1 23.8 24.7

BASELINE #2 (D ELEX .)HM
PM
OM
33.1 32.1
34.3
14.3 18.8
20.0
11.0 15.7
19.0
16.6 16.4
17.6

B ITEXT P ROJECTION
HM
PM
OM
58.0* 55.7* 58.0*
30.8* 31.2* 36.2*
29.4* 29.4* 35.6*
35.7* 34.1* 35.7*

Table 5: Comparison of cross-lingual approaches. F1 scores obtained in our Portuguese validation corpus
using: a S UPERVISED system trained on the small available data, a D ELEXICALIZED system trained with
universal POS tags and multilingual embeddings and our B ITEXT P ROJECTION OF D EPENDENCIES.
The symbol * indicates that the best system beats the other systems with statistical significance, withp < 0.05 and according to a bootstrap resampling test (Koehn, 2004).
as Riloff and Wiebe (2003) and whose list is available with the corpus, but selecting only documents
annotated with targets. We randomly split theremaining documents into train and development
sets, respectively with a total of 6,471 and 782 sentences.14 Table 4 shows the performance of thedelexicalized baseline in English, compared witha lexicalized system. We will see how this model
behaves in a cross-lingual setting in §7.2.
Our System: Bitext Projection of Opinion Dependencies. Finally, we implemented our crosslingual B ITEXT approach (§6). We trained the(lexicalized) English model on the MPQA corpus(the performance of this model is shown in Table 4). Then, we ran this model on the English
side of the parallel corpus, generating automatic
annotations, and projected these annotations to thePortuguese side, as described in §6.2. Finally, aPortuguese model was trained on these projected
annotations using the arc-factored model and features described in §4.
7.2

Comparison

Table 5 shows the F1 scores obtained by the threesystems on the Portuguese test partition. We observe that the B ITEXT approach outperformed theS UPERVISED and the D ELEXICALIZED ones in all
metrics with a considerable margin, which showsthe effectiveness of our proposed method. TheS UPERVISED system suffers from the fact that thetraining set is too small to allow good generalization; the bitext projection method, in contrast,
can create arbitrarily large training corpora without any annotation effort. The performance of
14
Note that this split is different from the one we used in§5. There we used the same split as Johansson and Moschitti(2013), for a fair comparison with their system; here, we follow the standard MPQA test partition.

the D ELEXICALIZED system is rather disappointing. This result is justified by a decrease of performance in English due to the delexicalization(cf. Table 4), followed by an extra loss of quality
due to language differences.
Though our B ITEXT approach scores the best,
the scores are behind the range of values obtained for English (Table 4), and far from the interannotator agreement numbers (Table 3), suggesting room for improvement. The polarity scores inTable 5 appear to be relatively low. This fact isprobably be justified with the annotator agreementscores (Table 3) which are considerably lower forthese metrics.

8

Conclusions

We presented a cross-lingual framework for finegrained opinion mining. We used a bitext projection technique to transfer dependency-based
opinion frames from English to Portuguese. Experimentally, our dependency model achieved
state-of-the-art results for English, and the Portuguese system trained with bitext projection outperformed two baselines: a supervised systemtrained on a small dataset, and a delexicalized
model with bilingual word embeddings.

9

Acknowledgements

We would like to thank the anonymous reviewers for their insightful comments, and Richard
Johansson for sharing his code and for answering several questions.
This work was partially supported by the EU/FEDER programme,
QREN/POR Lisboa (Portugal), under the Intelligo project (contract 2012/24803) and by a FCT
grants UID/EEA/50008/2013 and PTDC/EEISII/2312/2012.

References
Cecilia Ovesdotter Alm, Dan Roth, and Richard Sproat.
2005. Emotions from text: machine learning for textbased emotion prediction. In HLT and EMNLP.
Miguel B Almeida, Mariana SC Almeida, André FT Martins, Helena Figueira, Pedro Mendes, and Cláudia Pinto.
2014. Priberam compressive summarization corpus: A
new multi-document summarization corpus for european
portuguese. In LREC.

Karl Moritz Hermann and Phil Blunsom. 2014. Multilingual
Models for Compositional Distributional Semantics. In
ACL.
Minqing Hu and Bing Liu. 2004. Mining opinion features incustomer reviews. In AAAI.
Rebecca Hwa, Philip Resnik, Amy Weinberg, Clara Cabezas,
and Okan Kolak. 2005. Bootstrapping parsers via syntactic projection across parallel texts. Natural Language
Engineering, 11(3):311–325.

Wilker Aziz and Lucia Specia. 2011. Fully automatic compilation of a Portuguese-English parallel corpus for statistical machine translation. In STIL 2011.

Ozan İrsoy and Claire Cardie. 2014. Opinion mining withdeep recurrent neural networks. In EMNLP, pages 720–
728, Doha, Qatar, October.

Krisztian Balog, Gilad Mishne, and Maarten de Rijke. 2006.
Why are they excited?: Identifying and explaining spikes
in blog mood levels. In EACL.

Richard Johansson and Alessandro Moschitti. 2010. Reranking models in fine-grained opinion analysis. In COLING.

Carmen Banea, Rada Mihalcea, Janyce Wiebe, and Samer
Hassan. 2008. Multilingual subjectivity analysis using
machine translation. In EMNLP.
Eric Breck, Yejin Choi, and Claire Cardie. 2007. Identifying
expressions of opinion in context. In IJCAI, pages 2683–
2688.
S. Buchholz and E. Marsi. 2006. CoNLL-X shared task on
multilingual dependency parsing. In Int. Conf. on Natural
Language Learning.
Yejin Choi and Claire Cardie. 2010. Hierarchical sequential
learning for extracting opinions and their attributes. In
ACL, pages 269–274.
Yejin Choi, Claire Cardie, Ellen Riloff, and Siddharth Patwardhan. 2005. Identifying sources of opinions with conditional random fields and extraction patterns. In Proc. of
HLT and EMNLP.
Yejin Choi, Eric Breck, and Claire Cardie. 2006. Joint extraction of entities and relations for opinion recognition.
In EMNLP.
Simon Clematide, Stefan Gindl, Manfred Klenner, Stefanos Petrakis, Robert Remus, Josef Ruppenhofer, Ulli
Waltinger, and Michael Wiegand. 2012. MLSA A Multilayered Reference Corpus for German Sentiment Analysis. In LREC, Istanbul, Turkey, may.
K. Crammer, O. Dekel, J. Keshet, S. Shalev-Shwartz, and
Y. Singer. 2006. Online Passive-Aggressive Algorithms.
Journal of Machine Learning Research, 7:551–585.
Dipankar Das and Sivaji Bandyopadhyay. 2010. Labelingemotion in bengali blog corpus a fine grained tagging atsentence level. In In Proc. of the 8th Workshop on Asian
Language Resources (ALR8), COLING-2010, pages 47–
55.

Richard Johansson and Alessandro Moschitti. 2011. Extracting opinion expressions and their polarities: exploration ofpipelines and joint models. In ACL:HLT.
Richard Johansson and Alessandro Moschitti. 2013. Relational features in fine-grained opinion analysis. Computational Linguistics, 39(3):473–509.
Soo-Min Kim and Eduard Hovy. 2006. Extracting opinions,
opinion holders, and topics expressed in online news media text. In SST.
P. Koehn. 2004. Statistical signicance tests for machinetranslation evaluation. In ACL.
Lun-Wei Ku, Yu-Ting Liang, and Hsin-Hsi Chen. 2006.
Opinion extraction, summarization and tracking in news
and blog corpora. In AAAI Spring Symposium: Computational Approaches to Analyzing Weblogs.
Percy Liang, Ben Taskar, and Dan Klein. 2006. Alignmentby agreement. In NAACL.
Bing Liu. 2012. Sentiment analysis and opinion mining. Synthesis Lectures on Human Language Technologies, 5(1):1–167.
Bin Lu, Chenhao Tan, Claire Cardie, and Benjamin K. Tsou.
2011. Joint bilingual sentiment classification with unlabeled parallel corpora. In ACL:HLT.
André F. T. Martins and M. S. C. Almeida. 2014. Priberam:
A turbo semantic parser with second order features. In
SemEval, pages 471–476, August.
André F. T. Martins, Miguel B. Almeida, and Noah A.
Smith. 2013. Turning on the turbo: Fast third-order nonprojective turbo parsers. In ACL.
André F. T. Martins. 2015. Transferring coreference resolvers with posterior regularization. In ACL.

Xiaowen Ding, Bing Liu, and Philip S Yu. 2008. A holistic lexicon-based approach to opinion mining. In WSDM,
pages 231–240.

Ryan McDonald, Slav Petrov, and Keith Hall. 2011. Multisource transfer of delexicalized dependency parsers. In
EMNLP, pages 62–72.

Greg Durrett and Dan Klein. 2013. Easy victories and uphillbattles in coreference resolution. In EMNLP.

Rada Mihalcea, Carmen Banea, and Janyce Wiebe. 2007.
Learning multilingual subjective language via crosslingual projections. In ACL.

Lin Gui, Ruifeng Xu, Jun Xu, and Chenxiang Liu. 2013.
A cross-lingual approach for opinion holder extraction. Journal of Computational Information Systems,
9(6):2193–2200.

Sebastian Padó and Mirella Lapata. 2009. Cross-lingual annotation projection for semantic roles. Journal of Artificial Intelligence Research, 36(1):307–340.

Bo Pang and Lillian Lee. 2008. Opinion mining and sentiment analysis. Found. Trends Inf. Retr., 2(1-2):1–135.
Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002.
Thumbs up?: Sentiment classification using machinelearning techniques. In EMNLP.
Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012. A
universal part-of-speech tagset. In LREC.
Peter Prettenhofer and Benno Stein. 2010. Cross-language
text classification using structural correspondence learning. In ACL.
Ellen Riloff and Janyce Wiebe. 2003. Learning extraction
patterns for subjective expressions. In EMNLP.
Yohei Seki, David Kirk Evans, Lun-Wei Ku, Hsin-Hsi Chen,
Noriko Kando, and Chin-Yew Lin. 2007. Overview
of opinion analysis pilot task at NTCIR-6. In NTCIR-6
Workshop Meeting.

Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005.
Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39(2-3):165–
210.
Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005.
Recognizing contextual polarity in phrase-level sentiment
analysis. In HLT and EMNLP.
Theresa Wilson. 2008. Fine-Grained Subjectivity Analysis.
Ph.D. thesis, University of Pittsburgh.
Yuanbin Wu, Qi Zhang, Xuanjing Huang, and Lide Wu.
2011. Structural opinion mining for graph-based sentiment representation. In EMNLP.
Bishan Yang and Claire Cardie. 2012. Extracting opinion
expressions with semi-markov conditional random fields.
In EMNLP-CoNLL.
Bishan Yang and Claire Cardie. 2013. Joint inference forfine-grained opinion extraction. In ACL.

Yohei Seki, Lun-Wei Ku, Le Sun, Hsin-Hsi Chen, and Noriko
Kando. 2010. Overview of opinion analysis pilot task at
NTCIR-8: A Step Toward Cross Lingual Opinion Analysis. In NTCIR-8 Workshop Meeting.

Bishan Yang and Claire Cardie. 2014. Joint modeling of
opinion expression extraction and attribute classification.
Trans. of the Association for Computational Linguistics(TACL).

Richard Socher, John Bauer, Christopher D. Manning, andAndrew Y. Ng. 2013a. Parsing with compositional vector
grammars. In ACL.

David Yarowsky and Grace Ngai. 2001. Inducing multilingual pos taggers and np bracketers via robust projection
across aligned corpora. In NAACL.

Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang,
Christopher D Manning, Andrew Y Ng, and Christopher
Potts. 2013b. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, pages
1631–1642.

Hong Yu and Vasileios Hatzivassiloglou. 2003. Towards
answering opinion questions: Separating facts from opinions and identifying the polarity of opinion sentences. In
EMNLP.

Anders Søgaard. 2011. Data point selection for crosslanguage adaptation of dependency parsers. In ACL:HLT
short papers, pages 682–686. Association for Computational Linguistics.
M. Surdeanu, R. Johansson, A. Meyers, L. Màrquez, andJ. Nivre. 2008. The CoNLL-2008 Shared Task on Joint
Parsing of Syntactic and Semantic Dependencies. In
CoNLL.
Oscar Täckström, Dipanjan Das, Slav Petrov, Ryan McDonald, and Joakim Nivre. 2013. Token and type constraints
for cross-lingual part-of-speech tagging. Trans. of the Association for Computational Linguistics (TACL).
Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word
representations: a simple and general method for semisupervised learning. In ACL.
Peter D. Turney. 2002. Thumbs up or thumbs down?: Semantic orientation applied to unsupervised classificationof reviews. In ACL.
Xiaojun Wan. 2009. Co-training for cross-lingual sentimentclassification. In ACL-IJCNLP.
Mengqiu Wang and Chris Manning. 2014. Cross-lingual
projected expectation regularization for weakly supervised learning. TACL, 2:55–66.
Bin Wei and Christopher Pal. 2010. Cross lingual adaptation:
an experiment on sentiment classifications. In ACL, pages
258–262.

Daniel Zeman and Philip Resnik. 2008. Cross-language
parser adaptation between related languages. In IJCNLP,
pages 35–42.