# Distilling Opinion in Discourse: A Preliminary Study

Nicholas Asher, Farah Benamara, Yvette Yannick Mathieu

► To cite this version:

Nicholas Asher, Farah Benamara, Yvette Yannick Mathieu. Distilling Opinion in Discourse: A Preliminary Study. 22nd International Conference on Computational Linguistics (COLING 2008), Aug 2008, Manchester, United Kingdom. pp.7-10. halshs-00751060

HAL Id: halshs-00751060

https://shs.hal.science/halshs-00751060

Submitted on 9 Jun 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Distilling Opinion in Discourse: A Preliminary Study

Nicholas Asher and Farah Benamara
IRIT-CNRS Toulouse,
France

{asher, benamara}@irit.fr

Yvette Yannick Mathieu
LLF-CNRS Paris,
France

yannick.mathieu@linguist.jussieu.fr

## Abstract

In this paper, we describe a preliminary study for a discourse based opinion categorization and propose a new annotation schema for a deep contextual opinion analysis using discourse relations.

## 1 Introduction

Computational approaches to sentiment analysis eschew a general theory of emotions and focus on extracting the affective content of a text from the detection of expressions of sentiment. These expressions are assigned scalar values, representing a positive, a negative or neutral sentiment towards some topic. Using information retrieval, text mining and computational linguistic techniques together with a set of dedicated linguistic resources, one can calculate opinions exploiting the detected "bag of sentiment words". Recently, new methods aim to assign fine-grained affect labels based on various psychological theories–e.g., the MPQA project (Wiebe et al., 2005) based on literary theory and linguistics and work by (Read et al., 2007) based on the Appraisal framework (Martin and White, 2005).

We think there is still room for improvement in this field. To get an accurate appraisal of opinion in texts, NLP systems have to go beyond positive/negative classification and to identify a wide range of opinion expressions, as well as how they are discursively related in the text. In this paper, we describe a preliminary study for a discourse based opinion categorization. We propose a new annotation schema for a fine-grained contextual

opinion analysis using discourse relations. This analysis is based on a lexical semantic analysis of a wide class of expressions coupled together with an analysis of how clauses involving these expressions are related to each other within a discourse. The aim of this paper is to establish the feasibility and stability of our annotation scheme at the subsentential level and propose a way to use this scheme to calculate the overall opinion expressed in a text on a given topic.

## 2 A lexical semantic analysis of opinion expressions

We categorize opinion expressions using a typology of four top-level categories (see table 1): **REPORTING** expressions, which provide an evaluation of the degree of commitment of both the holder and the subject of the reporting verb, **JUDGMENT** expressions, which express normative evaluations of objects and actions, **ADVISE** expressions, which express an opinion on a course of action for the reader, and **SENTIMENT** expressions, which express feelings (for a more detailed description of our categories see (Asher et al, 2008)).

Our approach to categorize opinions uses the lexical semantic research of (Wierzbicka, 1987), (Levin, 1993) and (Mathieu, 2004). From these classifications, we selected opinion verb classes and verbs which take opinion expressions within their scope and which reflect the holder's commitment on the opinion expressed. We removed some verb classes, modified others and merged related classes into new ones. Subjective verbs were split into these new categories which were then extended by adding nouns and adjectives.

Our classification is the same for French and English. It differs from psychologically based classifications like Martin's Appraisal system : in ours

<footnote>© 2008. Licensed under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported license (http://creativecommons.org/licenses/by-nc-sa/3.0/). Some rights reserved.</footnote>
<table><caption>Table 1: Top-Level opinion categories.</caption><thead><tr><th>Groups</th><th>SubGroups</th><th>Examples</th></tr></thead><tbody><tr><td rowspan="6">Reporting</td><td>a) Inform</td><td>inform, notify, explain</td></tr><tr><td>b) Assert</td><td>assert, claim, insist</td></tr><tr><td>c) Tell</td><td>say, announce, report</td></tr><tr><td>d) Remark</td><td>comment, observe, remark</td></tr><tr><td>e) Think</td><td>think, reckon, consider</td></tr><tr><td>f) Guess</td><td>presume, suspect, wonder</td></tr><tr><td rowspan="3">Judgment</td><td>g) Blame</td><td>blame, criticize, condemn</td></tr><tr><td>h) Praise</td><td>praise, agree, approve</td></tr><tr><td>i) Appreciation</td><td>good, shameful, brilliant</td></tr><tr><td rowspan="3">Advise</td><td>j) Recommend</td><td>advise, argue for</td></tr><tr><td>k) Suggest</td><td>suggest, propose</td></tr><tr><td>l) Hope</td><td>wish, hope</td></tr><tr><td rowspan="10">Sentiment</td><td>m) Anger/CalmDown</td><td>irritation, anger</td></tr><tr><td>n) Astonishment</td><td>astound, daze, impress</td></tr><tr><td>o) Love, fascinate</td><td>fascinate, captivate</td></tr><tr><td>p) Hate, disappoint</td><td>demoralize, disgust</td></tr><tr><td>q) Fear</td><td>fear, frighten, alarm</td></tr><tr><td>r) Offense</td><td>hurt, chock</td></tr><tr><td>s) Sadness/Joy</td><td>happy, sad</td></tr><tr><td>t) Bore/entertain</td><td>bore, distraction</td></tr><tr><td>u) Touch</td><td>disarm, move, touch</td></tr></tbody></table>

the contents of the JUDGMENT and SENTIMENT categories are quite different, and more detailed for SENTIMENT descriptions with 14 sub-classes. Ours is also broader: the REPORTING and the ADVISE categories do not appear as such in the Appraisal system. In addition, we choose not to build our discourse based opinion categorization on the top of MPQA (Wiebe et al, 2005) for two reasons. First, we suggest a more detailed analysis of private states by defining additional sets of opinion classes such as HOPES and RECOMMENDATIONS. We think that refined categories are needed to build a more nuanced appraisal of opinion expressions in discourse. Second, text anchors which correspond to opinion in MPQA are not well defined since each annotator is free to identify expression boundaries. This is problematic if we want to integrate rhetorical structure into opinion identification task. MPQA often groups discourse indicators (but, because, etc.) with opinion expressions leading to no guarantee that the text anchors will correspond to a well formed discourse unit.

## 3 Towards a Discursive Representation of Opinion Expressions

Rhetorical structure is an important element in understanding opinions conveyed by a text. The following simple examples drawn from our French corpus show that discourse relations affect the strength of a given sentiment. S1: [I agree with you]a even if I was shocked and S2: Buy the DVD, [you will not regret it]b. Opinions in S1 and S2 are positive but the contrast introduced by even in S1 decreases the strength of the opinion expressed in (a) whereas the explanation provided by (b) in S2 increases the strength of the recommendation.

Using the discourse theory SDRT (Asher and Lascarides, 2003) as our formal framework, our four opinion categories are used to label opinion expressions within a discourse segment. For example, there are three opinion segments in the sentence S3: [[It's poignant]d, [sad]e]g and at the same time [horrible]f

We use five types of rhetorical relations: CONTRAST, CORRECTION, SUPPORT, RESULT and CONTINUATION (For a more detailed description see (Asher et al, 2008)). Within a discourse segment, negations were treated as reversing the polarities of the opinion expressions within their scope. Conditionals are hard to interpret because they affect the opinion expressed within the consequent of a conditional in different ways. For example, conditionals, expressions of ADVISE can block the advice or reverse it. Thus if you want to waste you money, buy this movie will be annotated as a recommendation not to buy it. On the other hand, conditionals can also strengthen the recommendation as in if you want to have good time, go and see this movie. We have left the treatment of conditionals as well as disjunctions for future work.

### 3.1 Shallow Semantic Representation

In order to represent and evaluate the overall opinion of a document, we characterize discourse segments using a shallow semantic representation using a feature structure (FS) as described in (Asher et al, 2008). Figure 1 shows the discursive representation of the review movie S4: [This film is amazing.]a. [[One leaves not completely convinced] b.1, but [one is overcome]b.2]. [[It's poignant]c.1, [sad]c.2] and at the same time [horrible]c.3].[Buy it]d. [You won't regret it]e.

<figure><img src="image_2.png" /><figcaption>Figure 1: Discursive representation of S4.</figcaption></figure>

Once we have constructed the discursive representation of a text, we have to combine the different FS in order to get a general representation
that goes beyond standard positive/negative representation of opinion texts. In this section, we first explain the combination process of FS. We then show how an opinion text can be summarized using a graphical representation.

The combination of low-level FS is performed in two steps: (1) combine the structures related by coordinating relations (such as CONTRAST and CONTINUATION). In figure 1, this allows to build from the segments b.1 and b.2 a new FS; (2) combine the structures related via subordinating relations (such as SUPPORT and RESULT) in a bottom up way. In figure 1, the FS of the segment a is combined with the structure deduced from step 1. During this process, a set of dedicated rules is used. The procedure is formalized as follows. Let a, b be two segments related by the rhetorical relation *R* such as: *R(a, b)*. Let *S*<sub>a</sub>, *S*<sub>b</sub> be the FS associated respectively to *a* and *b* i.e *S*<sub>a</sub> : [*category* : [*group*<sub>a</sub> : subgroup*<sub>a</sub>*], modality : [*polarity* : *p*<sub>a</sub>, strength : *s*<sub>a</sub>] ...] and *S*<sub>b</sub> : [*category* : [*group*<sub>b</sub> : subgroup*<sub>b</sub>*], modality : [*polarity* : *p*<sub>b</sub>, strength : *s*<sub>b</sub>] ...] and let *S* : [*category* : [*group*], modality : [*polarity* : *p*, strength : *s*] ...] be the FS deduced from the combination of *S*<sub>a</sub> and *S*<sub>b</sub>. Some of our rules are:

CONTINUATIONS strengthen the polarity of the common opinion. One of the rule used is: if (*group*<sub>*a*</sub> = *group*<sub>*b*</sub>) and (*subgroup*<sub>*a*</sub> ≠ *subgroup*<sub>*b*</sub>) then if (*p*<sub>*a*</sub> = *neutral*) and (*p*<sub>*b*</sub> ≠ *neutral*) then *group* = *group*<sub>*a*</sub> and *p* = *p*<sub>*b*</sub> and *s* = max(*s*<sub>*a*</sub>, *s*<sub>*b*</sub>), as in *moving and sad news*.

For CONTRAST, let *OW*<sub>*i*</sub> be the set of opinion words that belongs to a segment *S*<sub>*i*</sub>. We have for *OW*<sub>*a*</sub> = ∅ and *OW*<sub>*b*</sub> ≠ ∅: *group* = *group*<sub>*b*</sub>, *p* = *p*<sub>*b*</sub> and *s* = *s*<sub>*b*</sub> + 1, as in *I don't know a lot on Edith Piaf's life but I was enthralled by this movie*.

Finally, an opinion text is represented by a graph
$G = (\aleph, \Re)$ such as:

- $\aleph = H \cup T$ is the set of nodes where :
  $H = \{ho_i/ho_i \text{ is an opinion holder}\}$ and $T = \{to_i: \text{value}/to_i \text{ is a topic and value is a FS}\}$, such as :
  $\text{value} = [\text{Polarity} : p, \text{Strength} : s, \text{Advice} : a]$, where:
  $p = \{\text{positive, negative, neutral}\}$ and $s, a = \{0, 1, 2\}$.
- $\Re = \Re_H \cup \Re_T \cup \Re_{H-T}$ where: $\Re_H = \{(h_i, h_j)/h_i, h_j \in H\}$ means that two topics are related via an ELABORATION relation.
This holds generally between a topic and a subtopic, such as a movie and a scenario; $\Re_T = \{(t_i, t_j, type)/t_i, t_j \in T \text{ and type = support/contrast}\}$ means that two holders are related via a CONTRAST (holders $h_i$ and $h_j$ have a contrasted opinion

on the same topic) or a SUPPORT relation
(holders share the same point of view) ; and
$\Re_{H-T} = \{(h_i, t_j, type)/h_i \in H \text{ and } t_j \in T \text{ and } type = \text{attribution/commitment}$}^{}$ means that an opinion to-
wards a topic $t_j$ is attributed or committed to a
holder $h_i$. For example, in *John said that the film
was horrible*, the opinion is only attributed to John
because verbs from the TELL group do not con-
vey anything about the author view. However, in
*John infomed the commitee that the situation was
horrible*, the writer takes the information to be es-
tablished. The figure 2 below shows the general
representation of the movie review S4.

<figure><img src="image_3.png" /><figcaption>Figure 2: General representation of S4.</figcaption></figure>

4 Annotating Opinion Segments:
Experiments and Preliminary Results

We have analyzed the distribution of our categories
in three different types of digital corpora, each
with a distinctive style and audience : movie re-
views, Letters to the Editor and news reports in
English and in French. We randomly selected 150
articles for French corpora (around 50 articles for
each genre). Two native French speakers anno-
tated respectively around 546 and 589 segments.
To check the cross linguistic feasability of gener-
alisations made about the French data, we also an-
notated opinion categories for English. We have
annotated around 30 articles from movie reviews
and letters. For news reports, the annotation in En-
glish was considerably helped by using texts from
the MUC 6 corpus (186 articles), which were an-
notated independently with discourse structure by
three annotators in the University of Texas's DIS-
COR project (NSF grant, IIS-0535154); the anno-
tation for our opinion expressions involved a col-
lapsing of structures proposed in DISCOR.

The annotation methodology is described in (Asher et al, 2008). For each corpus, annotators first begin to annotate elementary discourse segments, define its shallow representation and finally, connect the identified segments using the set of rhetorical relations we have identified. A segment is annotated only if *it explicitly* contains an opinion word that belong to our lexicon or if it bears a rhetorical relation to an opinion segment.
The average distribution of opinion expressions in our corpus across our categories for each language is shown in table 2. The annotation of movie reviews was very easy. The opinion expressions are mainly adjectives and nouns. We found an average of 5 segments per review. Opinion words in Letters to the Editor are adjectives and nouns but also verbs. We found an average of 4 segments per letter. Finally, opinions in news documents involve principally reported speech. As we only annotated segments that clearly expressed opinions or were related via one of our rhetorical relations to a segment expressing an opinion, our annotations typically only covered a fraction of the whole document. This corpus was the hardest to annotate and generally contained lots of embedded structure introduced by REPORTING type verbs.

To compute the inter-annotator agreements (IAA) we did not take into account the opinion holder and the topic as well as the polarity and the strength because we chose to focus, at a first step, only on agreements on opinion categorization, segment identification and rhetorical structure detection. We computed the agreements only on the French corpus. The French annotators performed a two step annotation where an intermediate analysis of agreement and disagreement between the two annotators was carried out. This analysis allowed each annotator to understand the reason of some annotation choices. Using the Kappa measure, the IAA on opinion categorization is 95% for movie reviews, 86% for Letters to the Editors and 73% for news documents.

Annotators had good agreement concerning what the basic segments were (82%), which shows that the discourse approach in sentiment analysis is easier compared to the lexical task where annotators have low agreements on the identification of opinion tokens. The principal sources of disagreement in the annotation process came from annotators putting opinion expressions in different categories (mainly between PRAISE/BLAME group and APPRECIATION group, such as *shame*) and the choice of rhetorical relations. Nevertheless, by using explicit discourse connectors, we were able to get relatively high agreement on the choice of rhetorical relations. We also remained quite unsure how to distinguish between the reporting of neutral opinions and the reporting of facts. The main extension of this work are to (1) deepen our opinion typology, specifically to include modals

<table><caption>Table 2: Average distribution of our categories.</caption><thead><tr><th rowspan="2">Groups</th><th colspan="2">Movie (%)</th><th colspan="2">Letters (%)</th><th colspan="2">News (%)</th></tr><tr><th>French</th><th>English</th><th>French</th><th>English</th><th>French</th><th>English</th></tr></thead><tbody><tr><td>Reporting</td><td>2.67</td><td>2.12</td><td>14.80</td><td>13.34</td><td>43.91</td><td>42.85</td></tr><tr><td>a</td><td>0</td><td>0</td><td>0.71</td><td>1.33</td><td>4.02</td><td>4.76</td></tr><tr><td>b</td><td>0.53</td><td>0</td><td>0</td><td>4</td><td>5.83</td><td>0</td></tr><tr><td>c</td><td>0</td><td>0</td><td>1.79</td><td>0</td><td>4.51</td><td>35.71</td></tr><tr><td>d</td><td>0.88</td><td>0</td><td>2.17</td><td>0</td><td>11.82</td><td>0</td></tr><tr><td>e</td><td>1.33</td><td>0</td><td>10.12</td><td>6.67</td><td>5.89</td><td>1.34</td></tr><tr><td>f</td><td>0</td><td>2.12</td><td>0</td><td>1.34</td><td>11.77</td><td>0</td></tr><tr><td>Judgment</td><td>60.53</td><td>40.52</td><td>52.50</td><td>73.34</td><td>39.23</td><td>33.34</td></tr><tr><td>g</td><td>0.54</td><td>0</td><td>6.32</td><td>26.66</td><td>13.69</td><td>16.67</td></tr><tr><td>h</td><td>2.45</td><td>2.12</td><td>7.54</td><td>20</td><td>1.81</td><td>4.76</td></tr><tr><td>i</td><td>54.49</td><td>38.29</td><td>33.48</td><td>26.87</td><td>23.72</td><td>11.90</td></tr><tr><td>Advise</td><td>6.92</td><td>10.63</td><td>10.05</td><td>13.34</td><td>7.27</td><td>9.52</td></tr><tr><td>j</td><td>6.26</td><td>8.51</td><td>0.70</td><td>5.33</td><td>1.37</td><td>0</td></tr><tr><td>k</td><td>0.66</td><td>2.12</td><td>3.94</td><td>1.33</td><td>3.61</td><td>0</td></tr><tr><td>l</td><td>0</td><td>0</td><td>5.38</td><td>6.67</td><td>2.28</td><td>9.52</td></tr><tr><td>Sentiment</td><td>27.30</td><td>34.04</td><td>33.08</td><td>2.67</td><td>11.35</td><td>16.67</td></tr><tr><td>m</td><td>0.54</td><td>0</td><td>3.23</td><td>0</td><td>0.90</td><td>0</td></tr><tr><td>n</td><td>2.23</td><td>6.38</td><td>3.96</td><td>2.66</td><td>0.90</td><td>7.14</td></tr><tr><td>o</td><td>7.38</td><td>4.25</td><td>3.74</td><td>0</td><td>1.87</td><td>9.52</td></tr><tr><td>p</td><td>4.97</td><td>2.12</td><td>5.03</td><td>0</td><td>2.72</td><td>0</td></tr><tr><td>q</td><td>2.23</td><td>0</td><td>5.03</td><td>0</td><td>1.86</td><td>0</td></tr><tr><td>r</td><td>0.89</td><td>0</td><td>7.17</td><td>0</td><td>2.28</td><td>0</td></tr><tr><td>s</td><td>3.79</td><td>4.25</td><td>2.87</td><td>0</td><td>0.88</td><td>0</td></tr><tr><td>t</td><td>1.33</td><td>14.9</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>u</td><td>4.46</td><td>2.12</td><td>2.15</td><td>2.12</td><td>0</td><td>0</td></tr></tbody></table>

and moods like the subjunctive, and to (2) provide a deep semantic representation that associates for each category of opinion a lambda term involving the preferred content and a lambda term for the presuppositional content of the expression, if it has one. In terms of automatization, we plan to exploit a syntactic parser to get the argument structure of verbs and then a discourse segmenter like that developed in the DISCOR project, followed by the detection of discourse relations using cue words.

## References

* Asher N. and Benamara F. and Mathieu Y.Y. 2008. Categorizing Opinions in Discourse. ECAI08.

* Asher N. and Lascarides A. 2003. Logics of Conversation. Cambridge University Press.

* Levin B. 1993. English Verb Classes and Alterna-tions: A Preliminary Investigation. University of Chicago Press.

* Martin J.R and White P.R.R. 2005. Language of Evaluation: Appraisal in English. Palgrave Macmillan.

* Mathieu Y. Y. 2004. A Computational Semantic Lexicon of French Verbs of Emotion. In Shanahan, G., Qu, Y., Wiebe, J. (eds.): Computing Attitude and Affect in Text. Dordrecht.

* Read J., Hope D. and Carroll J. 2007. Annotating Expressions of Appraisal in English. The Linguistic Annotation Workshop, ACL 2007.

* Wiebe J., Wilson T. and Cardie C. 2005. Annotating Expressions of Opinions and Emotions in Language. Language Resources and Evaluation 1(2).

* Wierzbicka A. 1987. Speech Act Verbs. Sydney: Academic Press.