SlowGuess/NAACL_2021_OCR huggingface dataset:
In order to isolate the effects of negation and speculation on the results, we test all models trained on the original Laptop and Restaurant datasets on the  $\text{Laptop}_{\text{Neg}}$ ,  $\text{Restaurant}_{\text{Neg}}$ ,  $\text{Laptop}_{\text{Spec}}$ , and  $\text{Restaurant}_{\text{Spec}}$  test splits. Tables 5 and 6 show the results for negation and speculation, respectively. The results for the dev split and the  $F_{1}$ -s of the test split are shown in Appendix D.
Firstly, all models perform comparatively worse on the challenge datasets, dropping an average of 24 and 25 pp on  $F_{1}$ -i on the negation and speculation data, respectively. Nearly all of this drop comes from poorer classification (acc-s,  $F_{1}$ -s), while target extraction ( $F_{1}$ -a) is relatively stable. This demonstrates the importance of resolving negation and speculation for TSA and the usefulness of the annotated data to determine these effects.
On Laptop $_{Neg}$  and Restaurant $_{Neg}$  incorporating negation auxiliary tasks gives an average improvement of 3.8 pp on the  $F_{1}-i$  metric when using GloVe embeddings. More specifically, MTL with negation improves the sentiment classification scores, but does not help extraction. This makes sense conceptually, as negation has little effect on whether or not a word is part of a sentiment target. Instead,
These findings also generalise to the results on the development splits, shown in Tables 11 and 12 within Appendix C.
LaptopNeg	sentiment	NEGCD	DR	LEX	NEG\(SFU\)	SPEC	UPOS	STL
GloVe	42.80(2.48)	38.54*(0.98)	38.72*(3.00)	45.26(1.45)	41.23*(2.90)	38.92*(1.74)
TL	48.49(2.32)	45.90(3.54)	45.93(2.13)	47.04(2.93)	45.71(2.19)	46.29(2.03)
extraction	GloVe	75.36*(0.91)	76.05*(1.20)	78.68(0.97)	75.04*(1.92)	76.14(2.06)	77.98(1.41)
TL	82.39(1.34)	82.95(1.36)	83.47(1.26)	83.25(1.80)	82.24*(1.39)	82.58(1.58)
targeted	GloVe	32.28(2.23)	29.30*(0.54)	30.47*(2.45)	33.96(1.30)	31.36*(1.78)	30.36*(1.56)
TL	39.95(2.02)	38.08(3.13)	38.35(2.01)	39.18(2.88)	37.59(1.99)	38.23(1.89)
RestaurantNeg	sentiment	GloVe	53.41(4.28)	49.78*(2.10)	47.69*(1.19)	56.01(1.07)	48.86*(3.94)	50.58*(2.18)
TL	60.69(1.91)	62.61(2.11)	60.80(3.20)	60.45(2.04)	61.70(1.42)	60.06(2.13)
extraction	GloVe	80.97(1.47)	82.22(1.29)	82.15(0.74)	80.74(1.58)	81.53(0.32)	81.92(0.91)
TL	83.04(1.26)	82.94*(0.97)	84.10(0.86)	83.94(1.67)	83.48(1.59)	82.33*(1.37)
targeted	GloVe	43.28(3.95)	40.95*(2.31)	39.19*(1.23)	45.22(0.80)	39.85*(3.35)	41.43*(1.87)
TL	50.40(2.03)	51.92(1.64)	51.15(3.04)	50.75(2.10)	51.49(0.86)	49.45(2.01)
Table 5: Sentiment (acc-s), extraction  $(F_{1}\text{-} a)$  and full targeted  $(F_{1}\text{-} i)$  results for LaptopNeg and RestaurantNeg test split, where the values represent the mean (standard deviation) of five runs with a different random seeds. The bold values represent the best model, while highlighted models are those that perform better than the single task baseline. The  $\star$  represents the models that are significantly worse  $(\mathrm{p} < 0.05)$  than the best performing model on the respective dataset, metric, and embedding.
jointly learning dependency relations (DR) and full lexical analysis (LEX) improve extraction results. Furthermore, when using TL instead of GloVe embeddings, the best MTL model  $(\mathrm{NEG}_{SFU})$  does marginally beat the STL TL equivalent on average, indicating that multi-task learning is still able to contribute something to transfer learning.
On  $\mathrm{LaptopSpec}$  and  $\mathrm{RestaurantSpec}$  MTL models improve results when using GloVe embeddings, with the additional speculation (SPEC) and dependency relation (DR) data improving the  $F_{1} - i$  metric by 0.5 pp and 0.49 pp respectively on average. However, with TL, MTL only leads to benefits on the Restaurant dataset. Unlike the negation data results, the speculation results appear to be helped more by syntactic auxiliary tasks like DR than semantic tasks like  $\mathrm{NEG}_{CD}$  and to some extent  $\mathrm{NEG}_{SFU}$ .
The best MTL GloVe models on the original datasets (LEX $^{15}$  and DR, respectively) also outper
form the STL GloVe models on the challenge data, indicating that MTL leads to greater robustness. When comparing the STL model using GloVe and TL on average the model improves by 9.55 pp on the negation dataset compared to 3.65 pp for the speculation suggesting that transfer learning is less effective for speculation.
In this paper, we have compared the effects of MTL using various auxiliary tasks for TSA and have created a negation and speculation annotated challenge dataset $^{16}$  for TSA in order to isolate the effects of MTL. We show that TSA methods are drastically affected by negation and speculation effects in the data. These effects can be similarly reduced by either incorporating auxiliary task information into the model through MTL or through transfer learning. Additionally, MTL of negation
dataset is worse than STL by 0.05 but for all other  $F_{1}$ -i Laptop results LEX is better than STL.
$^{16}$ https://bit.ly/312kwpP
15The development  $F_{1}$  -i result for LEX on the Laptop
LaptopSpec	sentiment	NEGCD	DR	LEX	NEGSFU	SPEC	UPOS	STL
GloVe	34.32(1.86)	35.67(1.00)	36.75(1.91)	35.98(2.05)	36.74(1.64)	35.57(1.31)	34.67(1.40)
TL	35.42(3.54)	34.76(1.63)	35.06(1.97)	34.08*(0.40)	35.03(2.36)	35.01(1.04)	35.97(1.45)
extraction	GloVe	74.77*(1.54)	74.01*(1.93)	77.80(1.34)	75.99(2.48)	73.39*(1.74)	76.80(0.99)	75.01(1.93)
TL	80.11*(1.40)	80.77(1.23)	81.47*(0.50)	83.14(2.22)	81.49(1.24)	81.07(1.38)	79.84*(0.58)
targeted	GloVe	25.67*(1.62)	26.39*(0.60)	28.59(1.42)	27.33(1.70)	26.95(1.07)	27.31(0.82)	26.01*(1.26)
TL	28.36(2.81)	28.09(1.68)	28.56(1.60)	28.33(0.52)	28.54(1.83)	28.37(0.77)	28.72(1.20)
RestaurantSpec	sentiment	GloVe	62.38(3.75)	64.01(2.72)	63.44(2.21)	63.33(1.87)	64.30(3.14)	63.15(3.38)	63.94(1.84)
TL	67.23(1.08)	68.98(1.17)	69.70(2.51)	67.62(1.58)	66.93(1.79)	68.13(1.25)	68.17(2.44)
extraction	GloVe	75.53(1.03)	76.40(1.90)	75.75(1.18)	75.66(1.65)	75.29(0.77)	75.87(0.97)	75.58(1.48)
TL	77.92(1.36)	77.84(0.84)	79.10(1.48)	78.76(1.27)	78.20(1.80)	77.15(1.92)	77.61(1.87)
targeted	GloVe	47.14(3.24)	48.94(3.06)	48.07(2.22)	47.90(1.25)	48.41(2.48)	47.94(3.14)	48.35(2.32)
TL	52.39(1.18)	53.69(0.69)	55.15(2.70)	53.25(1.10)	52.34(1.85)	52.55(1.22)	52.94(2.99)
Table 6: Sentiment (acc-s), extraction  $(F_{1}-a)$  and full targeted  $(F_{1}-i)$  results for Laptop  $Spec$  and Restaurant  $Spec$  test split, where the values represent the mean (standard deviation) of five runs with a different random seeds. The bold values represent the best model, while highlighted models are those that perform better than the single task baseline. The  $\star$  represents the models that are significantly worse ( $p<0.05$ ) than the best performing model on the respective dataset, metric, and embedding.
can lead to small improvements when combined with transfer learning. Returning to the two original research questions, we can conclude that in general 1) MTL using negation (speculation) as an auxiliary task does make TSA models more robust to negated (speculative) samples and 2) transfer learning seems to incorporate much of the same knowledge. Additionally, incorporating syntactic information as an auxiliary task within MTL creates models that are more robust to both negation and speculation.
Neither MTL nor TL are currently guarantees for improved performance $^{17}$ . Additionally, the results from the challenge datasets indicate that different auxiliary tasks improve the performance of different subtasks of TSA. This may suggest that the target extraction and sentiment classification tasks should not be treated as a collapsed labelling task, as the sentiment and extraction tasks are too dissimilar (Hu et al., 2019). Future work should consider
using pipeline or joint approaches, where each subtask can be paired with the most beneficial auxiliary tasks. This decoupling could also allow MTL and transfer learning to compliment each other more.
Finally, in order to improve reproducibility and to encourage further work, we release the code $^{18}$ , dataset, and trained models associated with this paper, hyperparameter search details with compute infrastructure (Appendix E), number of parameters and runtime details (Appendix F), and further detailed dev and test results (appendices C and D), in line with the result checklist from Dodge et al. (2019).
This work has been carried out as part of the SANT project (Sentiment Analysis for Norwegian Text), funded by the Research Council of Norway (grant number 270908). Andrew has been funded by Lan
17 Compare the performance of LEX using GloVe (28.59) to when it uses TL (28.56) in Table 6 for the Laptop dataset.
$^{18}$ https://github.com/jerbarnes/multitask_negation_for_targeted_sentiment
caster University by an EPSRC Doctoral Training Grant. The authors thank the UCREL research centre for hosting the models created from this research.
Cecilia Ovesdotter Alm, Dan Roth, and Richard Sproat. 2005. Emotions from text: Machine learning for text-based emotion prediction. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 579-586, Vancouver, BC, Canada.
Isabelle Augenstein, Sebastian Ruder, and Anders Søgaard. 2018. Multi-task learning of pairwise sequence classification tasks over disparate label spaces. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1896-1906, New Orleans, USA.
Jeremy Barnes. 2019. Ltg-oslo hierarchical multi-task network: The importance of negation for document-level sentiment in spanish. In Proceedings of the Iberian Languages Evaluation Forum co-located with 35th Conference of the Spanish Society for Natural Language Processing, IberLEF@SEPLN 2019, Bilbao, Spain, September 24th, 2019., pages 378-389.
Jeremy Barnes, Lilja Øvrelid, and Erik Velldal. 2019. Sentiment analysis is not solved! Assessing and probing sentiment classification. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 12-23, Florence, Italy.
Jeremy Barnes, Erik Velldal, and Lilja Øvrelid. 2020. Improving sentiment analysis with multitask learning of negation. Natural Language Engineering, 27.
Konstantin Bauman, Bing Liu, and Alexander Tuzhilin. 2017. Aspect based recommendations: Recommending items with the most valuable aspects based on user reviews. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 717-725. ACM.
Richard Caruana. 1993. Multitask learning: A knowledge-based source of inductive bias. In Proceedings of the Tenth International Conference on Machine Learning, pages 41-48. Morgan Kaufmann.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillip Koehn, and Tony Robinson. 2014. One billion word benchmark for measuring progress in statistical language modeling. In *Fifteenth Annual Conference of the International Speech Communication Association*.
Lu Chen, Wenbo Wang, Meenakshi Nagarajan, Shao-jun Wang, and Amit Sheth. 2012. Extracting diverse sentiment expressions with target-dependent polarity from twitter. In Proceedings of the International AAAI Conference on Web and Social Media.
Zhuang Chen and Tieyun Qian. 2020. Relation-aware collaborative learning for unified aspect-based sentiment analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3685-3694, Online. Association for Computational Linguistics.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493-2537.
Isaac Councill, Ryan McDonald, and Leonid Velikovich. 2010. What's great and what's not: learning to classify the scope of negation for improved sentiment analysis. In Proceedings of the Workshop on Negation and Speculation in Natural Language Processing, pages 51-59, Uppsala, Sweden.
Noa P. Cruz, Maite Taboada, and Ruslan Mitkov. 2016. A machine-learning approach to negation and speculation detection for sentiment analysis. Journal of the Association for Information Science and Technology, 67(9):2118-2136.
Sanjiv R. Das and Mike Y. Chen. 2007. Yahoo! for Amazon: Sentiment extraction from small talk on the web. Management Science, 53(9):1375-1388.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019. Show your work: Improved reporting of experimental results. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2185-2194, Hong Kong, China. Association for Computational Linguistics.
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383-1392, Melbourne, Australia. Association for Computational Linguistics.
Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34-48.
Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating nlp models via contrast sets.
Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. 2017. A joint many-task model: Growing a neural network for multiple NLP tasks. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1923-1933, Copenhagen, Denmark. Association for Computational Linguistics.
Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier. 2019. An interactive multi-task learning network for end-to-end aspect-based sentiment analysis. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 504-515, Florence, Italy. Association for Computational Linguistics.
Minghao Hu, Yuxing Peng, Zhen Huang, Dongsheng Li, and Yiwei Lv. 2019. Open-domain targeted sentiment analysis via span-based extraction and classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 537-546, Florence, Italy. Association for Computational Linguistics.
Qingnan Jiang, Lei Chen, Ruifeng Xu, Xiang Ao, and Min Yang. 2019. A challenge dataset and effective models for aspect-based sentiment analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6279-6284, Hong Kong, China. Association for Computational Linguistics.
Wei Jin and Hung Hay Ho. 2009. A novel lexicalized hmm-based learning framework for web opinion mining. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, page 465-472, New York, NY, USA. Association for Computing Machinery.
Nora Kassner and Hinrich Schütze. 2020. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7811-7818, Online. Association for Computational Linguistics.
Arzoo Katiyar and Claire Cardie. 2016. Investigating LSTMs for joint extraction of opinion entities and
relations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 919-929, Berlin, Germany. Association for Computational Linguistics.
Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2020. Learning the difference that makes a difference with counterfactually-augmented data. In International Conference on Learning Representations.
Evgeny Kim and Roman Klinger. 2019. Frowning Frodo, wincing Leia, and a seriously great friendship: Learning to classify emotional relationships of fictional characters. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 647-653, Minneapolis, Minnesota. Association for Computational Linguistics.
Jin-Dong Kim, Yue Wang, and Yamamoto Yasunori. 2013. The Genia event extraction shared task, 2013 edition - overview. In Proceedings of the BioNLP Shared Task 2013 Workshop, pages 8-15, Sofia, Bulgaria. Association for Computational Linguistics.
Roman Klinger and Philipp Cimiano. 2013. Bidirectional inter-dependencies of subjective expressions and targets and their value for a joint model. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 848-854.
Natalia Konstantinova, Sheila C.M. de Sousa, Noa P. Cruz, Manuel J. Maña, Maite Taboada, and Ruslan Mitkov. 2012. A review corpus annotated for negation, speculation and their scope. In Proceedings of the 8th International Conference on Language Resources and Evaluation, pages 3190-3195, Istanbul, Turkey.
Meghana Kshirsagar, Sam Thomson, Nathan Schneider, Jaime Carbonell, Noah A. Smith, and Chris Dyer. 2015. Frame-semantic role labeling with heterogeneous annotations. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 218–224, Beijing, China.
Emanuele Lapponi, Jonathon Read, and Lilja Øvrelid. 2012. Representing and resolving negation for sentiment analysis. In Proceedings of the 2012 IEEE 12th International Conference on Data Mining Workshops, pages 687-692, Washington, DC, USA.
Xin Li, Lidong Bing, Piji Li, and Wai Lam. 2019a. A unified model for opinion target extraction and target sentiment prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6714-6721.
Xin Li, Lidong Bing, Wenxuan Zhang, and Wai Lam. 2019b. Exploiting BERT for end-to-end aspect-based sentiment analysis. In Proceedings of the 5th
Workshop on Noisy User-generated Text (W-NUT 2019), pages 34-41, Hong Kong, China. Association for Computational Linguistics.
Bing Liu. 2015. Sentiment analysis: Mining Opinions, Sentiments, and Emotions. Cambridge University Press, Cambridge, United Kingdom.
Dehong Ma, Sujian Li, and Houfeng Wang. 2018. Joint learning for targeted sentiment analysis. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4737-4742, Brussels, Belgium. Association for Computational Linguistics.
Héctor Martínez Alonso and Barbara Plank. 2017. When is multitask learning effective? Semantic sequence prediction under varying data conditions. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 44-53, Valencia, Spain.
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. 2015. Image-based recommendations on styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '15, page 43-52, New York, NY, USA. Association for Computing Machinery.
Tomas Mikolov, Greg Corrado, Kai Chen, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. Proceedings of the International Conference on Learning Representations, pages 1-12.
Margaret Mitchell, Jacqui Aguilar, Theresa Wilson, and Benjamin Van Durme. 2013. Open domain targeted sentiment. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1643-1654, Seattle, Washington, USA. Association for Computational Linguistics.
Roser Morante and Walter Daelemans. 2012. ConanDoyle-neg: Annotation of negation cues and their scope in Conan Doyle stories. In Proceedings of the 8th International Conference on Language Resources and Evaluation, Istanbul, Turkey.
Lilja Øvrelid, Petter Mæhlum, Jeremy Barnes, and Erik Velldal. 2020. A fine-grained sentiment dataset for norwegian. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 5025-5033, Marseille, France. European Language Resources Association.
Nanyun Peng and Mark Dredze. 2017. Multi-task domain adaptation for sequence tagging. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 91-100, Vancouver, Canada.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1532-1543, Doha, Qatar.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018b. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1499-1509, Brussels, Belgium. Association for Computational Linguistics.
Barbara Plank. 2016. Keystroke dynamics as signal for shallow syntactic parsing. In Proceedings of the 26th International Conference on Computational Linguistics, pages 609-619, Osaka, Japan.
Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, Mohammad AL-Smadi, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphée De Clercq, Véronique Hoste, Marianna Apidianaki, Xavier Tannier, Natalia Loukachevitch, Evgeniy Kotelnikov, Nuria Bel, Salud María Jiménez-Zafra, and Gúlşen Eryigit. 2016. SemEval-2016 task 5: Aspect based sentiment analysis. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 19-30, San Diego, California. Association for Computational Linguistics.
Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Suresh Manandhar, and Ion Androutsopoulos. 2015. SemEval-2015 task 12: Aspect based sentiment analysis. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 486-495, Denver, Colorado. Association for Computational Linguistics.
Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. SemEval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 27-35, Dublin, Ireland. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2017. Reporting score distributions makes a difference: Performance study of LSTM-networks for sequence tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 338-348, Copenhagen, Denmark. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2018. Why comparing single performance scores does not allow to draw conclusions about machine learning approaches. arXiv preprint arXiv:1803.09578.