# Negation and Speculation in NLP: A Survey, Corpora, Methods, and Applications
## 2 Corpora and Annotation Process
### Corpora
#### 2.2.1 English Corpora
The annotation projection offers imperfect help in the annotation process, where the word-level F1 measure was 0.39, 0.45, and 0.24 for negation cue, scope, and event, respectively. The corpus, with its annotation guidelines, has been published for public use ([[https://github.com/qianchu/NegPar](https://github.com/qianchu/NegPar)]([https://github.com/qianchu/NegPar](https://github.com/qianchu/NegPar)) (accessed on 25 January 2022)). French: ESSAI is a corpus of French clinical trials obtained mainly from the National Cancer Institute registry website [13; 24]. The French protocol for such clinical trials has two parts: a summary of the trial, which presents its purpose and the methods applied, and its detailed description. ESSAI comprises 6547 sentences; 1025 sentences are negating, while 630 are speculative. The IAA measure of the negation annotation is 0.80, which is not high compared to English corpora [13]. French: [PERSON] et al. introduced another French corpus called CAS. This consists of 3811 sentences for clinical cases adapted from scientific literature and training materials [70]. These clinical cases were published in various journals and websites in French-speaking countries such as Belgium, Switzerland, and Canada, and relate to medical specialties [53]. The purpose of clinical cases is to describe clinical situations for real (de-identified) or fake patients. This corpus is automatically annotated with negation using different supervised learning techniques, trained on the ESSAI corpus. Uncertainty cues and their scope were annotated for CAS on the basis of heuristic rules. Of the 3811 sentences, 804 are negating and 226 include speculation. Later in 2021, two annotators manually verified CAS for the negation annotation, achieving a [PERSON]'s IAA of 0.84 [13]. Brazilian Portuguese: [PERSON] et al. presented the Brazilian Portuguese clinical trial protocols provided by Brazilian registration of clinical trials (REBEC) [13]. Each protocol includes a sample title, scientific title, description, and inclusion and exclusion criteria. Negation in this corpus provides valuable information on the specification of the target and the patient's recruitment. Three students annotated different parts of these protocols and came up with 643 negating sentences out of 3228, which is considered relatively low in this corpus. The authors compiled another Brazilian Portuguese corpus, collected from three hospitals in Brazil, covering medical specialties such as cardiology and nephrology [13]. This corpus contains 9808 sentences on clinical narratives. The negation cues and their scopes were manually annotated, and include 1751 negating cases with a Cohen's kappa coefficient of 0.74. Spanish: Negation and Uncertainty annotations in biomedical texts in the Spanish (NUBS) corpus make it the largest publicly available Spanish corpus for negation. It is the first to have annotations of speculative cues and scope and speculative events [18]. NUBES consists of 29,682 sentences from a Spanish private hospital's anonymized health records. The corpus was extracted from several sections, namely Surgical History, Physical Examination, and Diagnostic Tests, and divided into 10 batches. Together, two linguists drafted the initial annotation guidelines sourced from IULA [9] then extended them to include uncertainty. One batch was annotated using the initial guidelines, then improved after a medical expert resolved the disagreements between the annotators. The Cohen kappa IAA was calculated at 0.8 based on the sample batch. Later, the same batch was annotated by a third annotator to resolve all disagreements and create a 'gold standard'. Consequently, the other nine batches could be annotated by a single annotator using the final version of the annotation guidelines. This corpus of 29,682 sentences contains 7567 negated and 2219 uncertain instances.