# An Empirical Investigation of Sentiment Analysis of the Bug Tracking Process in Libre Office Open Source Software

Apostolos Kritikos, Theodoros Venetis, Ioannis Stamelos

► To cite this version:

Apostolos Kritikos, Theodoros Venetis, Ioannis Stamelos. An Empirical Investigation of Sentiment Analysis of the Bug Tracking Process in Libre Office Open Source Software. 16th IFIP International Conference on Open Source Systems (OSS), May 2020, Innopolis, Russia. pp.36-46, 10.1007/978-3-030-47240-5_4. hal-03647264

HAL Id: hal-03647264

https://inria.hal.science/hal-03647264

Submitted on 20 Apr 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# An Empirical Investigation of Sentiment Analysis of the Bug Tracking process in Libre Office Open Source Software

Apostolos Kritikos¹, Theodoros Venetis¹ and Ioannis Stamelos¹

1. Aristotle University of Thessaloniki, School of Informatics, University Campus, 54124 Thessaloniki, Greece {akritiko, venetheo, stamelos}@csd.auth.gr WWW home page: http://www.csd.auth.gr/en/

**Abstract.** In this work we are studying the sentiment in Open Source Software projects and more specifically in the process of bug reporting, to investigate the human factor, namely, the feedback from the community (end-users, developers, testers, etc.). One of the characteristics for which Open Source Software has gained attention, over the years, is the fact that it is continuously being tested and maintained by its community of volunteers. Sentiment analysis, a rapidly growing field, can enrich software evaluation with a social aspect. Results suggest that FLOSS projects' bug reports can potentially constitute a rich emotionally - imbued information source.

## 1 Introduction

Since the beginning of Open Source Software, the community around a FLOSS project is one of the greatest strengths of open source software development over closed source development. Eric S. Raymond [1] highlights this strength; “*Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone*”. In the community of an Open Source Software project we find developers who usually contribute both in the development and test process. In addition, community members can also be people who do not possess development skills but use the project as end users and contribute to testing. Cerone et. al. [2] stress the importance of defining qualitative metrics for indicators of collaboration effectiveness within Open Source Software communities.

In this paper we will be analyzing the sentiment from bug reports posted to Bugzilla. The proposed methodology is being applied to various major versions of the Open Source Software project Libre Office.
The rest of the paper is organized as follows: Section 2 provides background information for the basic terms discussed in this work. Section 3 presents scientific work done by other researchers relevant with the aim of this paper. Section 4 analyzes our case study plan. Section 5 provides the statistical analysis conducted to the data we collected. In Section 6 we discuss the findings on our statistical analysis, while Section 7 speculates on threats to validity. Finally, in Section 8 we conclude our work by summarizing our findings and we refer to possible future work. Please note that all the original data for this scientific work are available at http://users.auth.gr/akritiko/datasets/kritikosOSS2020_dataset.zip.

## 2 Background

This section presents an overview of the research state of the art on sentiment analysis (also known as opinion mining).

Sentiment analysis (often referred to as opinion mining) is the process of determining the polarity of an input text. It can be performed mainly at three different levels of granularity [3]: *document-level analysis*, classifying the sentiment expressed in a whole document as positive, negative or neutral; *sentence-level analysis*, which determines the sentiment per sentence in a document; *aspect-level analysis*, which delivers more fine-grained results and is based on the idea that an opinion consists of a sentiment and a respective target, which typically refers to an entity or a specific aspect of it.

The approaches mainly used for deducing the sentiment expressed in a document is the lexicon-based [4] and the machine learning-based [5] approach. Besides these two, there is a third one that involves the deployment of ontology-based methodologies for performing aspect-level sentiment analysis [6].

## 3 Related Work

In the software engineering domain, there have been the following relevant approaches for studying emotions and related factors in Open Source Software projects. Guzman et al. [7] study emotions expressed in commit comments of several open source projects and analyze their relationship with different factors. Rousinopoulos et al. [8] attempt to analyse the evolution of the sentiment of developers of a Open Source Software project (openSUSE Factory) by applying sentiment analysis techniques on the e-mails sent to the mailing list of the project. Similarly, Garcia et al. [9] perform sentiment analysis on the e-mail archives of another FLOSS project (Gentoo) and attempt to investigate the relation between the emotions in the mailing list and the respective activity of contributors in the project. Finally, the authors of [10] and [11] analyse comments in an issue tracking system, in order to investigate whether a relationship exists between human affectiveness
(emotion, sentiment and politeness) and developer productivity (i.e. time needed for fixing an issue).

# 4 Case Study Plan

We have chosen Libre Office as a candidate to conduct our analysis for a variety of reasons: (1) It is an Open Source Software project with a rich code base (until the time of writing it has had 486,472 commits made by 1,880 contributors representing 9,517,407 lines of code [12]), (2) it has an active community that shows no sign of long-term decline and has attracted the long-term and most active committers in OpenOffice.org (from which Libre Office was originated as a fork).

In this work we will be studying sixteen (16) major versions of the Libre Office productivity suite, namely: 3.3, 3.4, 3.5, 3.6, 4.0, 4.1, 4.2, 4.3, 4.4, 5.0, 5.1, 5.2, 5.3, 5.4, 6.0, 6.1.

## 4.1 Data Collection – Bugs

We collected our data from the official Libre Office bug tracking tool which is implemented with Bugzilla [13]. We gathered a total of 740 bugs that are related to the versions. Each bug is a threaded discussion with comments related to the bug reported. These 740 bugs are consisted of 6960 comments in total. Apart from the text of the comments to which we have applied the sentiment analysis process we also gather the following information for each bug [14]:

* ASSIGNED_TO: The person to which the bug was assigned to be fixed.

* CREATED: The date of the creation of the bug.

* CREATOR: The creator of the bug.

* LAST_CHANGE: When was the last time that this bug was updated.

* OPEN: If the bug is open or closed.

* COMPONENT_ID: Which component does the bug refer to (i.e. Writer).

* VERSION_ID: Which version does the bug refer to (i.e. 3.3).

As far as the comments of a bug are concerned (threaded discussion) we have gathered the following information:

* CREATED: Date and time that the comment was created.

* CREATOR: The creator of the comment (can be different than the creator of the bug).

* COMMENT_TEXT: The text of the comment.

* COUNT_ORDER: The order of the comment in the discussion (needed to be able to follow the flow of the threaded discussion).
* **BUG CreAtOR:** Used to identify when a comment is an answer to the thread by the creator of the bug.

* **BY_SYSTEM:** Used to identify system automated responses (which show zero sentiment by default).

## 4.2 Data Collection – Sentiment Analysis

As far as the sentiment analysis part is concerned, we used two (2) different tools, namely VADER [15] and MeaningCloud [16].

We consider both tools valuable for our investigation. VADER (Valence Aware Dictionary and sEntiment Reasoner) is an academic product [17] that serves “as a lexicon and rule-based sentiment analysis tool”. MeaningCloud, on the other hand is a commercial solution for sentiment analysis that provides a free plan for 20.000 requests per month. Since we are interested in investigating if and how sentiment affects the bug tracking process in open source software, by using both tools we, without loss of generality, get safer results concerning the existence of sentiment in the bug tracking process of open source software whereas, at the same time, we are utilizing the different kinds of information that each tool provides with the aim of succeeding a wider scope in our investigation.

VADER sentiment analyzer provided us with the following data per comment:

* **COMPOUNDSCORE:** As the authors of the tool describe it, a “normalized, weighted composite score. This score is the sum of the valence scores of each word in the lexicon, adjusted according to the rules, and then normalized to be between -1 (most extreme negative) and +1 (most extreme positive).”.

* **NEGATIVE, NEUTRAL, POSITIVE:** These scores are ratios that show the percentage of the comment that is positive, negative and / or neutral.

MeaningCloud analyzer provided us with the following data per comment:

* **SCORE:** The sentiment score that can be one of the following values: none (no sentiment), N+ (very negative), N (negative), neutral, P (positgive), P+ (very positive).

* **CONFIDENCE:** Percentage to which the analysis is accurate [0 – 100].

* **IRONY:** If the comment is ironic (IRONIC, NONIRONIC).

* **SUBJECTIVITY:** If the opinion on the comment is considered subjective or objective (SUBJECTIVE, OBJECTIVE).

* **AGREEMENT:** If the sentences of the comment are in agreement with each other or there are ambiguous meanings (AGREEMENT, DISAGREEMENT).
# 5 Statistical Analysis

This section presents the statistical analysis of the data collected. As a first step we tried to identify possible outliers that might be misleading for our results to consider possible data reduction [18]. In the following image you can see a visualization of the number of comments distribution per version of Libre Office and the box plots regarding our variables.

<figure><img src="image_2.png" /><figcaption>Fig 1. NOF Comments Distribution</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Fig 3. Box plots for Objective, Subjective, Agreement and Disagreement Comments</figcaption></figure>
From Fig.2 and Fig.3 we can see that our means are being affected by low values of our population. This also becomes obvious by the descriptive statistics provided to Table 3 following.

<table><caption>**Table 3.** Discriptive analysis for variables.</caption><thead><tr><th>Variable</th><th>n</th><th>mean</th><th>sd</th><th>median</th><th>min</th><th>max</th><th>skew</th><th>kyrtosis</th></tr></thead><tbody><tr><td>Comments</td><td>16</td><td>435.00</td><td>390.32</td><td>336.50</td><td>2.0</td><td>1345.0</td><td>0.91</td><td>-0.26</td></tr><tr><td>Ironic</td><td>16</td><td>2.44</td><td>2.06</td><td>2.00</td><td>0.0</td><td>7.0</td><td>0.47</td><td>-0.78</td></tr><tr><td>NonIronic</td><td>16</td><td>432.56</td><td>389.11</td><td>334.50</td><td>2.0</td><td>1341.0</td><td>0.92</td><td>-0.24</td></tr><tr><td>Subjective</td><td>16</td><td>279.56</td><td>259.58</td><td>205.50</td><td>1.0</td><td>889.0</td><td>0.95</td><td>-0.20</td></tr><tr><td>Objective</td><td>16</td><td>155.44</td><td>131.34</td><td>131.00</td><td>1.0</td><td>456.0</td><td>0.83</td><td>-0.40</td></tr><tr><td>Agreement</td><td>16</td><td>299.94</td><td>282.60</td><td>212.50</td><td>1.0</td><td>973.0</td><td>0.99</td><td>-0.10</td></tr><tr><td>Disagreement</td><td>16</td><td>135.06</td><td>108.60</td><td>124.00</td><td>1.0</td><td>372.0</td><td>0.69</td><td>-0.67</td></tr><tr><td>Vader (Positive)</td><td>16</td><td>269.88</td><td>248.98</td><td>200.00</td><td>2.0</td><td>869.0</td><td>1.03</td><td>0.00</td></tr><tr><td>Vader (Neutral)</td><td>16</td><td>101.75</td><td>92.38</td><td>81.00</td><td>0.0</td><td>311.0</td><td>0.84</td><td>-0.45</td></tr><tr><td>Vader (Negative)</td><td>16</td><td>63.38</td><td>50.76</td><td>55.50</td><td>0.0</td><td>165.0</td><td>0.46</td><td>-1.13</td></tr><tr><td>MeaningCloud (Positive)</td><td>16</td><td>241.38</td><td>229.88</td><td>177.50</td><td>1.0</td><td>808.0</td><td>1.10</td><td>0.20</td></tr><tr><td>MeaningCloud (Neutral)</td><td>16</td><td>40.94</td><td>31.86</td><td>35.50</td><td>0.0</td><td>107.0</td><td>0.55</td><td>-0.87</td></tr><tr><td>MeaningCloud (Negative)</td><td>16</td><td>60.75</td><td>47.12</td><td>54.50</td><td>0.0</td><td>156.0</td><td>0.56</td><td>-0.93</td></tr><tr><td>MeaningCloud (No Sent.)</td><td>16</td><td>91.94</td><td>84.51</td><td>67.00</td><td>1.0</td><td>274.0</td><td>0.72</td><td>-0.80</td></tr></tbody></table>

Since this work studies the bugs reported to Libre Office per version, we decided not to proceed to data reduction since the versions that seem to act like outliers are provide valuable information. For example, the increased activity of comments in some of the 4.x versions is since 3.6 version was the version that followed the fork from Open Office after which redesign and refactor processes took place. We also chose to let versions with low concentration of comments, i.e. 6.1, to further investigate if this behavior is due to successful bug fixing or it was just a version with low bug tracking activity. In Table 3 you can see the descriptive analysis of the variables we gathered for the sentiment analysis using the tools described in Section 4.

<figure><img src="image_4.png" /><figcaption>**Fig 4.** Box plots for VADER sentiment analyzer</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>**Fig 5.** Box plots for MeaningCloud sentiment analyzer</figcaption></figure>
Finally, we conducted a descriptive analysis separately for the confidence variable per version of the Libre Office project.

<table><caption>**Table 4.** Descriptive analysis for confidence.</caption><thead><tr><th>Version</th><th>n</th><th>mean</th><th>sd</th><th>median</th><th>min</th><th>max</th><th>skew</th><th>kyrtosis</th></tr></thead><tbody><tr><td>v3.3</td><td>89</td><td>94.83</td><td>6.32</td><td>100</td><td>76</td><td>100</td><td>-0.75</td><td>-0.83</td></tr><tr><td>v3.4</td><td>145</td><td>95.03</td><td>6.26</td><td>100</td><td>76</td><td>100</td><td>-0.85</td><td>-0.52</td></tr><tr><td>v3.5</td><td>309</td><td>95.83</td><td>5.24</td><td>100</td><td>84</td><td>100</td><td>-0.87</td><td>-0.64</td></tr><tr><td>v3.6</td><td>129</td><td>96.05</td><td>5.26</td><td>100</td><td>86</td><td>100</td><td>-0.94</td><td>-0.62</td></tr><tr><td>v4.0</td><td>364</td><td>95.83</td><td>5.71</td><td>100</td><td>74</td><td>100</td><td>-1.06</td><td>-0.06</td></tr><tr><td>v4.1</td><td>1345</td><td>96.98</td><td>5.03</td><td>100</td><td>76</td><td>100</td><td>-1.41</td><td>0.53</td></tr><tr><td>v4.2</td><td>624</td><td>96.78</td><td>4.99</td><td>100</td><td>74</td><td>100</td><td>-1.48</td><td>1.47</td></tr><tr><td>v4.3</td><td>746</td><td>96.79</td><td>5.11</td><td>100</td><td>76</td><td>100</td><td>-1.42</td><td>0.96</td></tr><tr><td>v4.4</td><td>1126</td><td>97.05</td><td>4.83</td><td>100</td><td>83</td><td>100</td><td>-1.41</td><td>0.61</td></tr><tr><td>v5.0</td><td>624</td><td>96.73</td><td>4.84</td><td>100</td><td>73</td><td>100</td><td>-1.40</td><td>1.38</td></tr><tr><td>v5.1</td><td>128</td><td>96.02</td><td>4.94</td><td>100</td><td>76</td><td>100</td><td>-1.16</td><td>1.12</td></tr><tr><td>v5.2</td><td>615</td><td>96.66</td><td>5.05</td><td>100</td><td>73</td><td>100</td><td>-1.43</td><td>1.37</td></tr><tr><td>v5.3</td><td>435</td><td>96.65</td><td>5.23</td><td>100</td><td>84</td><td>100</td><td>-1.23</td><td>-0.06</td></tr><tr><td>v5.4</td><td>186</td><td>97.72</td><td>3.86</td><td>100</td><td>86</td><td>100</td><td>-1.57</td><td>1.63</td></tr><tr><td>v6.0</td><td>93</td><td>96.82</td><td>5.09</td><td>100</td><td>76</td><td>100</td><td>-1.67</td><td>2.52</td></tr><tr><td>v6.1</td><td>2</td><td>93.00</td><td>9.90</td><td>93</td><td>86</td><td>100</td><td>0.00</td><td>-2.75</td></tr></tbody></table>

Having concluded with the descriptive analysis of the chosen variables we then proceeded in studying our selected variables regarding their normality. Since the descriptive analysis is showing (Table 3) skeweness with values around or bigger than 1 we have the indication that some of our variables might not follow the normal distribution. Therefore, we proceeded in applying the Shapiro-Wilk normality test in order to identify which variables follow the normal distribution and which do not

<table><caption>**Table 5.** Shapiro-Wilk normality tests.</caption><thead><tr><th>Variable</th><th>W</th><th>p-value</th></tr></thead><tbody><tr><td>Comments</td><td>0.87727</td><td>0.0352</td></tr><tr><td>Ironic</td><td>0.90422</td><td>0.09393</td></tr><tr><td>NonIronic</td><td>0.87637</td><td>0.03408</td></tr><tr><td>Subjective</td><td>0.89079</td><td>0.05734</td></tr><tr><td>Objective</td><td>0.87399</td><td>0.03131</td></tr><tr><td>Agreement</td><td>0.86491</td><td>0.02275</td></tr><tr><td>Disagreement</td><td>0.90727</td><td>0.1051</td></tr></tbody></table>
<table><tr><td>VADER (Positive)</td><td>0.85854</td><td>0.01824</td></tr><tr><td>VADER (Neutral)</td><td>0.88603</td><td>0.04822</td></tr><tr><td>VADER (Negative)</td><td>0.92277</td><td>0.1869</td></tr><tr><td>MeaningCloud (Positive)</td><td>0.85216</td><td>0.01466</td></tr><tr><td>MeaningCloud (Neutral)</td><td>0.93041</td><td>0.2238</td></tr><tr><td>MeaningCloud (Negative)</td><td>0.92766</td><td>0.2238</td></tr><tr><td>MeaningCloud (No Sent.)</td><td>0.89165</td><td>0.05915</td></tr></table>

From the results we are seeing that there are variables with p-value > 0.05 therefore parametric statistical analysis cannot be carried for them in this study since their data might not follow the normal distribution. Therefore, we will be conducting our statistical tests with the use of nonparametric tests and more specifically using the Spearman's Rank Correlation Coefficient.

<table><caption>**Table 6.** Spearman's Rank Correlation for VADER versus MeaningCloud</caption><thead><tr><th></th><th>Vader POS</th><th>Vader NEU</th><th>Vader NEG</th></tr></thead><tbody><tr><th>MC POS</th><td>p-value = 2.2e-16<br/>p = 0.9617647</td><td></td><td></td></tr><tr><th>MC NEU</th><td></td><td>p-value = 2.2e-16<br/>p = 0.9588235</td><td></td></tr><tr><th>MC NEG</th><td></td><td></td><td>p-value = 1.252e-10<br/>p = 0.9757177</td></tr></tbody></table>

In Table 6 we have applied the Spearman's Rank Correlation to investigate possible statistical similarities in the distributions of VADER (Positive, Neutral, Negative) and Meaning Cloud (Positive, Neutral, Negative). Our results indicate that all three correlations are statistically significant (p-value < 0.05) and very strong correlated (p > 0.9).

# 6 Discussion

In this work we are studying the sentiment in Open Source Software projects and more specifically in the process of bug reporting in the Libre Office project via its bug tracking platform, Bugzilla. We conducted our experiment in 16 major versions of the software and gathered the respective data (as analyzed in Section 4) for 740 bugs consisted of threaded discussions with 6960 comments in total. We proceeded
in sentiment analysis of these comments using two different tools (Vader & Meaning
Cloud) and statistically analyzed the results.

Our first goal in this investigation was to get a result on whether the sentiment
analysis of comments in bug report is meaningful. Given that bug reporting is a
technical process, usually made by technical oriented people (developers, and so
forth) one might argue that the text of the reports could be standardized with
technical language and therefore, possible lack of sentiment. In Table 4, showing the
descriptive analysis of the confidence metric of the sentiment analysis across the
versions of Libre Office, we can see that the mean is > 90% for all the versions. This
gives us a first indicator that the confidence of the sentiment analysis (which is done
with the MeaningCloud tool) is strong across all our data. In Table 6 we performed a
nonparametric test to get results regarding the similarities for the positive, neutral
and negative groups of comments between the two tools we have used. We see that
the tests show significant similarities in the distributions of Positive, Neutral and
Negative groups of comments for both Vader and Meaning Cloud.

From Table 3 we see that the number of ironic comments (variable Irony) are
significantly fewer (min: 0, max: 7, mean: 2.44) than the non-ironic comments (min:
2, max: 1341, mean: 432.56). This seems to be intuitively logical if we consider that
Bugzilla is a tool to which a community of Libre Office users (many of them
voluntarily) report bugs. Having a closer look to random comments tagged as ironic
from MeaningCloud we observed that such comments usually contain capitalized
words and although the message does not seem to be ironic, however the tool scores
them as such (i.e. [...] *however, it's happened on EVERY release since it started
happening 2 years ago (I always move to the n+1.0.0 release when it comes out)*
).

Looking at the subjectivity related variables we are observing similarly
interesting comments as with the irony variable. Starting a bug report with a
reference to ourselves is common in bug reporting (i.e. *I have experienced this think
while I was doing that*). However, in some cases if we strip this first-person
reference in the beginning of the sentence, we are left with the bug itself (i.e. in the
previous example: this is happening when someone does that). In the comments
collected there are similar “misunderstandings” by the MeaningCloud which could
be considered objective comments otherwise (i.e. *In order to limit the confusion
between ProposedEasyHack and EasyHack and to make queries much easier we are
changing ProposedEasyHack to NeedsDevEval. Thank you and apologies for the
noise*).

# 7 Threats to Validity

We are conducting our experiment to Libre Office. This may be a threat to validity
since including all of the components of the project would resul in a much bigger
dataset with a bigger bariety of functionality and thus, variety in bugs.
A significant difficulty we came across during our work involves the domain-specific vocabulary used in the input text (i.e. bug reports) we analysed. This often may result to inaccurate ratings by the sentiment analyser, since most popular sentiment analysis tools have been trained on more generic texts (i.e. Social Media). However, this problem is not new; it constitutes a commonly recognized threat to the validity of sentiment analysis results, not only in the domain we are working on, but in any other specialized field.

Another difficulty lies in the fact that, besides specialized terminology, content like bug reports may often contain extensive use of “every-day” (i.e. Jargon) expressions, abbreviations and even emoticons (sequences of symbols representing an emotion). These expressions are typically very dynamic, changing constantly and are being frequently replaced by other expressions, following each time the popular trends. Thus, even training the sentiment analyser accordingly would unfortunately generate dubious results.

Moreover, attempting to perform sentiment analysis in informal text generated by numerous human users may often involve subjectivity and sarcasm, both of which lead to inaccurate sentiment measurements. A solution to the former problem may involve integrating a subjectivity classifier [19, 20]. The latter problem, on the other hand, is even more challenging and typically appears in online discussions (like e.g. bug reports), although it is not that frequent in reviews of products and services. There have been some initial investigations (e.g. [21] and [22]), although the topic has not been extensively studied yet.

Finally due to space limitation we decided to make this first attempt to run our experiment per version of the Libre Office. This means that our original dataset was “compressed” in groups and, inevitably some of the fields that would otherwise be part of the experiment were omitted. It is however our intention to extend the version in this direction as well, as mentioned in the future work section.

## 8 Conclusion & Future Work

In this work we investigated the semantic analysis of bugs reported to the Libre Office Open Source Software, a tool written in C++ programming language.

As future work, we plan to replicate our case study on projects written in various programming languages for both desktop and web development (i.e. java, php, and so forth). We also intent to include metrics related with both the structural properties of the source code of the Open Source project alongside with its quality and resilience characteristics.

Additionally, we would like to attempt to enrich the human based dataset (for the sentiment analysis part) by using, additionally to the text originated from the bug reports (which mainly tend to be technical reports), the user ratings (both score and the text of the review) that various FLOSS projects provide. This way we
believe we will be capturing the sentiment of both the developers and the end users
of the software.

Finally, it is our intention to study other sentiment analyzers found in
literature and possibly compare and contrast their application to our dataset.

8 References

[1] Raymond, E.S., 1999. The Cathedral and the Bazaar (1st ed.). Tim O'Reilly (Ed.). O'Reilly & Associates, Inc., Sebastopol, CA, USA.

[2] Cerone, A., Fong, S. and Shaikh, S.A., 2012. Analysis of collaboration effectiveness and individuals' contribution in FLOSS communities.

[3] Liu, B. (2012). Sentiment analysis and opinion mining. Synthesis Lectures on Human Language Technologies, 5(1), 1-167.

[4] Taboada, M., Brooke, J., Tofiloski, M., Voll, K., & Stede, M. (2011). Lexicon-based methods for sentiment analysis. Computational Linguistics, 37(2), 267–307.

[5] Pang, B., & Lee, L. (2008). Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1–2), 1–135.

[6] Kontopoulos, E., Berberidis, C., Dergiades, T., & Bassiliades, N. (2013). Ontology-based sentiment analysis of twitter posts. Expert systems with applications, 40(10), 4065-4074.

[7] Guzman, E., Azócar, D., & Li, Y. (2014, May). Sentiment analysis of commit comments in GitHub: an empirical study. In Proceedings of the 11th Working Conference on Mining Software Repositories (pp. 352-355). ACM.

[8] Rousinopoulos, A. I., Robles, G., & González-Barahona, J. M. (2014). Sentiment Analysis of Free/Open Source Developers: Preliminary Findings from a Case Study. Revista Eletrônica de Sistemas de Informação ISSN 1677-3071 doi: 10.5329/RESI, 13(2).

[9] Garcia, D., Zanetti, M. S., & Schweitzer, F. (2013, September). The role of emotions in contributors activity: a case study on the GENTOO community. In Cloud and Green Computing (CGC), 2013 Third International Conference on (pp. 410-417). IEEE.

[10] Ortu, M., Adams, B., Destefanis, G., Tourani, P., Marchesi, M., & Tonelli, R. Are Bullies more Productive? Empirical Study of Affectiveness vs. Issue Fixing Time.

[11] Murgia, A., Tourani, P., Adams, B., & Ortu, M. (2014, May). Do developers feel emotions? an exploratory analysis of emotions in software artifacts. In Proceedings of the 11th Working Conference on Mining Software Repositories (pp. 262-271). ACM.

[12] Libre Office Project on Open Hub [url]. https://www.openhub.net/p/libreoffice

[13] Libre Office Bugzilla [url]. https://bugs documentfoundation.org/
[14] Libre Office Bugzilla Fields Documentation [url].
https://wiki documentfoundation.org/ QA/ Bugzilla/ Fields

[15] VADER, Github Repository [url]
https://github.com/cjhutto/vaderSentiment

[16] MeaningCloud Tool [url]
https://www.m meaningcloud.com/

[17] Hutto, C.J. & Gilbert, E.E. (2014). VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text. Eighth International Conference on Weblogs and Social Media (ICWSM-14). Ann Arbor, MI, June 2014.

[18] Wohlin, C., Runeson, P., Host, M., Ohlsson, M.C., Regnell, B., Wesslen, A., 2000. Experimentation in Software Engineering, 1st ed. Kluwer Academic Publishers, Boston/Dordrecht/London.

[19] Wiebe, J., & Riloff, E. (2005). Creating subjective and objective sentence classifiers from unannotated texts. InComputational Linguistics and Intelligent Text Processing(pp. 486-497). Springer Berlin Heidelberg.

[20] Barbosa, L., & Feng, J. (2010). Robust sentiment detection on twitter from biased and noisy data. In Proceedings of the 23rd international conference on computational linguistics: Posters (COLING '10).

[21] González-Ibáñez, R., Muresan, S., & Wacholder, N. (2011, June). Identifying sarcasm in Twitter: a closer look. InProceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2 (pp. 581-586). Association for Computational Linguistics.

[22] Riloff, E., Qadir, A., Surve, P., De Silva, L., Gilbert, N., & Huang, R. (2013). Sarcasm as Contrast between a Positive Sentiment and Negative Situation. In EMNLP (pp. 704-714).