# Negation and Speculation in NLP: A Survey, Corpora, Methods, and Applications
## 2 Corpora and Annotation Process
### Corpora
#### 2.2.1 English Corpora
This expert followed the'min-max' strategy during the annotation process, and modified it several times due to ambiguous cases arising between annotators. As a result, the guidelines followed in annotating the BioScope corpus were adapted to multiple domains [7]. Around 13% of the entire corpus contains negating expressions and more than 16% has speculative sentences. The reliability level of the annotation process was evaluated using the IAA rate, defined as the F\(\beta\)--1 measure of one annotation, considering the second to be the 'gold standard'. The average IAA of annotation was 0.85 for negation scope and 0.81 for speculation scope. The corpus is freely available for the purposes of academic and research ([[https://rgai.sed.hu/sites/rgai.sed.hu/files/bioscope.zip](https://rgai.sed.hu/sites/rgai.sed.hu/files/bioscope.zip)]([https://rgai.sed.hu/sites/rgai.sed.hu/files/bioscope.zip](https://rgai.sed.hu/sites/rgai.sed.hu/files/bioscope.zip)) (accessed on 17 September 2021)). The Computational Natural Language Learning (CoNLL) 2010 shared task was dedicated to identifying the speculation cues and their scope in two sub-corpora: biological publications and Wikipedia articles [50]. The first consists of 14,541 sentences, the biological part of the BioScope corpus representing the training set with an evaluation dataset formed of 790 of 5003 sentences from the PMC database. Wikipedia documents, the second sub-corpus, include 2484 of 11,111 sentences as a training set, with an evaluation set of 2346 of 9634 sentences. The corpus was annotated by two independent annotators who followed the'min-max' strategy used on the BioScope corpus. The chief linguist who wrote the annotation guidelines resolved any disagreement between the annotators. The corpus is publicly available for research purposes ([[http://www.inf.u-szeged.hu/rgai/conll2010](http://www.inf.u-szeged.hu/rgai/conll2010) st]([http://www.inf.u-szeged.hu/rgai/conll2010](http://www.inf.u-szeged.hu/rgai/conll2010) st) (accessed on 2 February 2022)). The research community has investigated other domains, and [PERSON] et al. highlight the need for corpora covering domains other than the biomedical [54]. The annotation guidelines of the biomedical domain should be adapted to new domains, such as product reviews [44] and short stories by [PERSON] [55]. The Product Review corpus is the first corpus in the review domain to have been annotated for negation [44]. It consists of 2111 sentences written as reviews for products, as extracted from Google Product Search. The number of negated sentences in this corpus is 679, and each sentence was annotated manually to define its cues and scope. Unlike the BioScope guidelines [14], the authors of this corpus included no negation cues in its scope. The IAA between two annotators on a dataset sample was 0.91, which is high. As a result, to complete the annotation of the entire corpus, each annotator applied the guidelines to a separate part of the reviews. [PERSON] and [PERSON] introduced the negation focus using the PropBank Corpus [51], the focus of negation in a sentence is the part of the scope that is most directly or explicitly negated. This corpus was selected for its semantic annotation, as it is not restricted to the biomedical domain [56]. The authors worked on 3779 sentences marked with MNEG, annotating the negation focus. Half of the PropBank FOC sentences were annotated by two annotators, achieving an IAA of 0.72. Any disagreements were then carefully examined and resolved, and the annotators were given a revised version of the guidelines to annotate the remaining half. The Simon Fraser University (SFU) Review Corpus is a large annotated English corpus comprising 400 documents extracted from the Epinions.com website, belonging to various domains such as books, hotels, movies, and consumer product reviews [57]. Each sentence is labelled according to whether it is a positive or negative review, where a different person submits every review. [PERSON] et al.