Question: Who has annotated the dataset?
Evidence: The dataset consists of 66,723 sentences with 2,157,048 tokens (incl. punctuation), see Table . The sizes of the seven court-specific datasets varies between 5,858 and 12,791 sentences, and 177,835 to 404,041 tokens. The distribution of annotations on a per-token basis corresponds to approx. 19–23 %. The Federal Patent Court (BPatG) dataset contains the lowest number of annotated entities (10.41 %).
The dataset includes two different versions of annotations, one with a set of 19 fine-grained semantic classes and another one with a set of 7 coarse-grained classes (Table ). There are 53,632 annotated entities in total, the majority of which (74.34 %) are legal entities, the others are person, location and organization (25.66 %). Overall, the most frequent entities are law GS (34.53 %) and court decision RS (23.46 %). The other legal classes (ordinance VO, European legal norm EUN, regulation VS, contract VT, and legal literature LIT) are much less frequent (1–6 % each). Even less frequent (less than 1 %) are lawyer AN, street STR, landscape LDS, and brand MRK.
The classes person, lawyer and company are heavily affected by the anonymisation process (80 %, 95 % and 70 % respectively). More than half of city and street, about 55 %, have also been modified. Landscape and organization are affected as well, with 40 % and 15 % of the occurrences edited accordingly. However, anonymisation is typically not applied to judge, country, institution and court (1–5 %).
The dataset was originally annotated by the first author. To evaluate and potentially improve the quality of the annotations, part of the dataset was annotated by a second linguist (using the annotation guidelines specifically prepared for its construction). We selected a small part that could be annotated in approx. two weeks. For the sentence extraction we paid special attention to the anonymised mentions of person, location or organization entities, because these are usually explained at their first mention. The resulting sample consisted of 2005 sentences with a broad variety of different entities (3 % of all sentences from each federal court). The agreement between the two annotators was measured using Kappa on a token basis. All class labels were taken into account in accordance with the IOB2 scheme BIBREF18. The inter-annotator agreement is 0.89, i. e., there is mostly very good agreement between the two annotators. Differences were in the identification of court decision and legal literature. Some unusual references of court decision (consisting only of decision type, court, date, file number) were not annotated such as `Urteil des Landgerichts Darmstadt vom 16. April 2014 – 7 S 8/13 –'. Apart from missing legal literature annotations, author names and law designations were annotated according to their categories (i. e., `Schoch, in: Schoch/Schneider/Bier, VwGO § 123 Rn. 35', `Bekanntmachung des BMG gemäß §§ 295 und 301 SGB V zur Anwendung des OPS vom 21.10.2010').
The second annotator had difficulties annotating the class law, not all instances were identified (`§ 272 Abs. 1a und 1b HGB', `§ 3c Abs. 2 Satz 1 EStG'), others only partially (`§ 716 in Verbindung mit' in `§ 716 in Verbindung mit §§ 321 , 711 ZPO'). Some titles of contract were not recognised and annotated (`BAT', `TV-L', `TVÜ-Länder' etc.).
This evaluation has revealed deficiencies in the annotation guidelines, especially regarding court decision and legal literature as well as non-entities. It would also be helpful for the identification and classification to list well-known sources of law, court decision, legal literature etc.
The dataset was originally annotated by the first author. To evaluate and potentially improve the quality of the annotations, part of the dataset was annotated by a second linguist (using the annotation guidelines specifically prepared for its construction).
Answer: The dataset was originally annotated by the first author and part of the dataset was annotated by a second linguist.