Language resources are an essential component in entire R&D domains. From the humble but vast repositories of monolingual texts that are used by the newest language modeling approaches like BERT and GPT, to parallel corpora that allows our machine translation systems to inch closer to human performance, to the more specialized resources like WordNets that encode semantic relations between nodes, these resources are necessary for the general advancement of Natural Language Processing, which eventually evolves into real apps and services we are (already) taking for granted.We introduce RONEC - the ROmanian Named Entity Corpus, a free, open-source resource that contains annotated named entities in copy-right free text.A named entity corpus is generally used for Named Entity Recognition (NER): the identification of entities in text such as names of persons, locations, companies, dates, quantities, monetary values, etc. This information would be very useful for any number of applications: from a general information extraction system down to task-specific apps such as identifying monetary values in invoices or product and company references in customer reviews.We motivate the need for this corpus primarily because, for Romanian, there is no other such corpus. This basic necessity has sharply arisen as we, while working on a different project, have found out there are no usable resources to help us in an Information Extraction task: we were unable to extract people, locations or dates/values. This constituted a major road-block, with the only solution being to create such a corpus ourselves. As the corpus was out-of-scope for this project, the work was done privately, outside the umbrella of any authors' affiliations - this is why we are able to distribute this corpus completely free.The current landscape in Romania regarding language resources is relatively unchanged from the outline given by the META-NET project over six years ago. The in-depth analysis performed in this European-wide Horizon2020-funded project revealed that the Romanian language falls in the "fragmentary support" category, just above the last, "weak/none" category (see the language/support matrix in BIBREF3). This is why, in 2019/2020, we are able to present the first NER resource for Romanian.We note that, while fragmentary, there are a few related language resources available, but none that specifically target named entities:ROCO BIBREF4 is a Romanian journalistic corpus that contains approx. 7.1M tokens. It is rich in proper names, numerals and named entities. The corpus has been automatically annotated at word-level with morphosyntactic information (MSD annotations).Released in 2016, ROMBAC BIBREF5 is a Romanian text corpus containing 41M words divided in relatively equal domains like journalism, legalese, fiction, medicine, etc. Similarly to ROCO, it is automatically annotated at word level with MSD descriptors.The much larger and recently released CoRoLa corpus BIBREF6 contains over 1B words, similarly automatically annotated.In all these corpora the named entities are not a separate category - the texts are morphologically and syntactically annotated and all proper nouns are marked as such - NP - without any other annotation or assigned category. Thus, these corpora cannot be used in a true NER sense. Furthermore, annotations were done automatically with a tokenizer/tagger/parser, and thus are of slightly lower quality than one would expect of a gold-standard corpus.The corpus, at its current version 1.0 is composed of 5127 sentences, annotated with 16 classes, for a total of 26377 annotated entities. The 16 classes are: PERSON, NAT_REL_POL, ORG, GPE, LOC, FACILITY, PRODUCT, EVENT, LANGUAGE, WORK_OF_ART, DATETIME, PERIOD, MONEY, QUANTITY, NUMERIC_VALUE and ORDINAL.It is based on copyright-free text extracted from Southeast European Times (SETimes). The news portal has published “news and views from Southeast Europe” in ten languages, including Romanian. SETimes has been used in the past for several annotated corpora, including parallel corpora for machine translation. For RONEC we have used a hand-picked selection of sentences belonging to several categories (see table TABREF16 for stylistic examples).The corpus contains the standard diacritics in Romanian: letters ș and ț are written with a comma, not with a cedilla (like ş and ţ). In Romanian many older texts are written with cedillas instead of commas because full Unicode support in Windows came much later than the classic extended Ascii which only contained the cedilla letters.The 16 classes are inspired by the OntoNotes5 corpus BIBREF7 as well as the ACE (Automatic Content Extraction) English Annotation Guidelines for Entities Version 6.6 2008.06.13 BIBREF8. Each class will be presented in detail, with examples, in the section SECREF3 A summary of available classes with word counts for each is available in table TABREF18.The corpus is available in two formats: BRAT and CoNLL-U Plus.As the corpus was developed in the BRAT environment, it was natural to keep this format as-is. BRAT is an online environment for collaborative text annotation - a web-based tool where several people can mark words, sub-word pieces, multiple word expressions, can link them together by relations, etc. The back-end format is very simple: given a text file that contains raw sentences, in another text file every annotated entity is specified by the start/end character offset as well as the entity type, one per line. RONEC is exported in the BRAT format as ready-to-use in the BRAT annotator itself. The corpus is pre-split into sub-folders, and contains all the extra files such as the entity list, etc, needed to directly start an eventual edit/extension of the corpus.Example (raw/untokenized) sentences:Tot în cadrul etapei a 2-a, a avut loc întâlnirea Vardar Skopje - S.C. Pick Szeged, care s-a încheiat la egalitate, 24 - 24.I s-a decernat Premiul Nobel pentru literatură pe anul 1959.Example annotation format:T1 ORDINAL 21 26 a 2-aT2 ORGANIZATION 50 63 Vardar SkopjeT3 ORGANIZATION 66 82 S.C. Pick SzegedT4 NUMERIC_VALUE 116 118 24T5 NUMERIC_VALUE 121 123 24T6 DATETIME 175 184 anul 1959The CoNLL-U Plus format extends the standard CoNLL-U which is used to annotate sentences, and in which many corpora are found today. The CoNLL-U format annotates one word per line with 10 distinct "columns" (tab separated):nolistsepID: word index;FORM: unmodified word from the sentence;LEMMA: the word's lemma;UPOS: Universal part-of-speech tag;XPOS: Language-specific part-of-speech tag;FEATS: List of morphological features from the universal feature inventory or from a defined language-specific extension;HEAD: Head of the current word, which is either a value of ID or zero;DEPREL: Universal dependency relation to the HEAD or a defined language-specific subtype of one;DEPS: Enhanced dependency graph in the form of a list of head-deprel pairs;MISC: Miscellaneous annotations such as space after word.The CoNLL-U Plus extends this format by allowing a variable number of columns, with the restriction that the columns are to be defined in the header. For RONEC, we define our CoNLL-U Plus format as the standard 10 columns plus another extra column named RONEC:CLASS. This column has the following format: nolistsep[noitemsep]each named entity has a distinct id in the sentence, starting from 1; as an entity can span several words, all words that belong to it have the same id (no relation to word indexes)the first word belonging to an entity also contains its class (e.g. word "John" in entity "John Smith" will be marked as "1:PERSON")a non-entity word is marked with an asterisk *Table TABREF37 shows the CoNLL-U Plus format where for example "a 2-a" is an ORDINAL entity spanning 3 words. The first word "a" is marked in this last column as "1:ORDINAL" while the following words just with the id "1".The CoNLL-U Plus format we provide was created as follows: (1) annotate the raw sentences using the NLP-Cube tool for Romanian (it provides everything from tokenization to parsing, filling in all attributes in columns #1-#10; (2) align each token with the human-made entity annotations from the BRAT environment (the alignment is done automatically and is error-free) and fill in column #11.For the English language, we found two "categories" of NER annotations to be more prominent: CoNLL- and ACE-style. Because CoNLL only annotates a few classes (depending on the corpora, starting from the basic three: PERSON, ORGANIZATION and LOCATION, up to seven), we chose to follow the ACE-style with 18 different classes. After analyzing the ACE guide we have settled on 16 final classes that seemed more appropriate for Romanian, seen in table TABREF18.In the following sub-sections we will describe each class in turn, with a few examples. Some examples have been left in Romanian while some have been translated in English for the reader's convenience. In the examples at the end of each class' description, translations in English are colored for easier reading.Persons, including fictive characters. We also mark common nouns that refer to a person (or several), including pronouns (us, them, they), but not articles (e.g. in "an individual" we don't mark "an"). Positions are not marked unless they directly refer to the person: "The presidential counselor has advised ... that a new counselor position is open.", here we mark "presidential counselor" because it refers to a person and not the "counselor" at the end of the sentence as it refers only to a position.Locul doi i-a revenit româncei Otilia Aionesei, o elevă de 17 ani.green!55!blueThe second place was won by Otilia Aionesei, a 17 year old student.Ministrul bulgar pentru afaceri europene, Meglena Kuneva ...green!55!blueThe Bulgarian Minister for European Affairs, Meglena Kuneva ...These are nationalities or religious or political groups. We include words that indicate the nationality of a person, group or product/object. Generally words marked as NAT_REl_POL are adjectives.avionul americangreen!55!bluethe American airplaneGrupul olandezgreen!55!bluethe Dutch groupGrecii iși vor alege președintele.green!55!blueThe Greeks will elect their president.Companies, agencies, institutions, sports teams, groups of people. These entities must have an organizational structure. We only mark full organizational entities, not fragments, divisions or sub-structures.Universitatea Politehnica București a decis ...green!55!blueThe Politehnic University of Bucharest has decided ...Adobe Inc. a lansat un nou produs.green!55!blueAdobe Inc. has launched a new product.Geo-political entities: countries, counties, cities, villages. GPE entities have all of the following components: (1) a population, (2) a well-defined governing/organizing structure and (3) a physical location. GPE entities are not sub-entities (like a neighbourhood from a city).Armin van Buuren s-a născut în Leiden.green!55!blueArmin van Buuren was born in Leiden.U.S.A. ramane indiferentă amenințărilor Coreei de Nord.green!55!blueU.S.A. remains indifferent to North Korea's threats.Non-geo-political locations: mountains, seas, lakes, streets, neighbourhoods, addresses, continents, regions that are not GPEs. We include regions such as Middle East, "continents" like Central America or East Europe. Such regions include multiple countries, each with its own government and thus cannot be GPEs.Pe DN7 Petroșani-Obârșia Lotrului carosabilul era umed, acoperit (cca 1 cm) cu zăpadă, iar de la Obârșia Lotrului la stațiunea Vidra, stratul de zăpadă era de 5-6 cm.green!55!blueOn DN7 Petroșani-Obârșia Lotrului the road was wet, covered (about 1cm) with snow, and from Obârșia Lotrului to Vidra resort the snow depth was around 5-6 cm.Produsele comercializate în Europa de Est au o calitate inferioară celor din vest.green!55!blueProducts sold in East Europe have a lower quality than those sold in the west.Buildings, airports, highways, bridges or other functional structures built by humans. Buildings or other structures which house people, such as homes, factories, stadiums, office buildings, prisons, museums, tunnels, train stations, etc., named or not. Everything that falls within the architectural and civil engineering domains should be labeled as a FACILITY. We do not mark structures composed of multiple (and distinct) sub-structures, like a named area that is composed of several buildings, or "micro"-structures such as an apartment (as it a unit of an apartment building). However, larger, named functional structures can still be marked (such as "terminal X" of an airport).Autostrada A2 a intrat în reparații pe o bandă, însă pe A1 nu au fost încă începute lucrările.green!55!blueRepairs on one lane have commenced on the A2 highway, while on A1 no works have started yet.Aeroportul Henri Coandă ar putea sa fie extins cu un nou terminal.green!55!blueHenri Coandă Airport could be extended with a new terminal.Objects, cars, food, items, anything that is a product, including software (such as Photoshop, Word, etc.). We don't mark services or processes. With very few exceptions (such as software products), PRODUCT entities have to have physical form, be directly man-made. We don't mark entities such as credit cards, written proofs, etc. We don't include the producer's name unless it's embedded in the name of the product.Mașina cumpărată este o Mazda.green!55!blueThe bought car is a Mazda.S-au cumpărat 5 Ford Taurus și 2 autobuze Volvo.green!55!blue5 Ford Taurus and 2 Volvo buses have been acquired.Named events: Storms (e.g.:"Sandy"), battles, wars, sports events, etc. We don't mark sports teams (they are ORGs), matches (e.g. "Steaua-Rapid" will be marked as two separate ORGs even if they refer to a football match between the two teams, but the match is not specific). Events have to be significant, with at least national impact, not local.Războiul cel Mare, Războiul Națiunilor, denumit, în timpul celui de Al Doilea Război Mondial, Primul Război Mondial, a fost un conflict militar de dimensiuni mondiale.green!55!blueThe Great War, War of the Nations, as it was called during the Second World War, the First World War was a global-scale military conflict.This class represents all languages.Românii din România vorbesc română.green!55!blueRomanians from Romania speak Romanian.În Moldova se vorbește rusa și româna.green!55!blueIn Moldavia they speak Russian and Romanian.Books, songs, TV shows, pictures; everything that is a work of art/culture created by humans. We mark just their name. We don't mark laws.Accesul la Mona Lisa a fost temporar interzis vizitatorilor.green!55!blueAccess to Mona Lisa was temporarily forbidden to visitors.În această seară la Vrei sa Fii Miliardar vom avea un invitat special.green!55!blueThis evening in Who Wants To Be A Millionaire we will have a special guest.Date and time values. We will mark full constructions, not parts, if they refer to the same moment (e.g. a comma separates two distinct DATETIME entities only if they refer to distinct moments). If we have a well specified period (e.g. "between 20-22 hours") we mark it as PERIOD, otherwise less well defined periods are marked as DATETIME (e.g.: "last summer", "September", "Wednesday", "three days"); Ages are marked as DATETIME as well. Prepositions are not included.Te rog să vii aici în cel mult o oră, nu mâine sau poimâine.green!55!bluePlease come here in one hour at most, not tomorrow or the next day.Actul s-a semnat la orele 16.green!55!blueThe paper was signed at 16 hours.August este o lună secetoasă.green!55!blueAugust is a dry month.Pe data de 20 martie între orele 20-22 va fi oprită alimentarea cu curent.green!55!blueOn the 20th of March, between 20-22 hours, electricity will be cut-off.Periods/time intervals. Periods have to be very well marked in text. If a period is not like "a-b" then it is a DATETIME.Spectacolul are loc între 1 și 3 Aprilie.green!55!blueThe show takes place between 1 and 3 April.În prima jumătate a lunii iunie va avea loc evenimentul de două zile.green!55!blueIn the first half of June the two-day event will take place.Money, monetary values, including units (e.g. USD, $, RON, lei, francs, pounds, Euro, etc.) written with number or letters. Entities that contain any monetary reference, including measuring units, will be marked as MONEY (e.g. 10$/sqm, 50 lei per hour). Words that are not clear values will not be marked, such as "an amount of money", "he received a coin".Primarul a semnat un contract în valoare de 10 milioane lei noi, echivalentul a aproape 2.6m EUR.green!55!blueThe mayor signed a contract worth 10 million new lei, equivalent of almost 2.6m EUR.Measurements, such as weight, distance, etc. Any type of quantity belongs in this class.Conducătorul auto avea peste 1g/ml alcool în sânge, fiind oprit deoarece a fost prins cu peste 120 km/h în localitate.green!55!blueThe car driver had over 1g/ml blood alcohol, and was stopped because he was caught speeding with over 120km/h in the city.Any numeric value (including phone numbers), written with letters or numbers or as percents, which is not MONEY, QUANTITY or ORDINAL.Raportul XII-2 arată 4 552 de investitori, iar structura de portofoliu este: cont curent 0,05%, certificate de trezorerie 66,96%, depozite bancare 13,53%, obligațiuni municipale 19,46%.green!55!blueThe XII-2 report shows 4 552 investors, and the portfolio structure is: current account 0,05%, treasury bonds 66,96%, bank deposits 13,53%, municipal bonds 19,46%.The first, the second, last, 30th, etc.; An ordinal must imply an order relation between elements. For example, "second grade" does not involve a direct order relation; it indicates just a succession in grades in a school system.Primul loc a fost ocupat de echipa Germaniei.green!55!blueThe first place was won by Germany's team.The corpus creation process involved a small number of people that have voluntarily joined the initiative, with the authors of this paper directing the work. Initially, we searched for NER resources in Romanian, and found none. Then we looked at English resources and read the in-depth ACE guide, out of which a 16-class draft evolved. We then identified a copy-right free text from which we hand-picked sentences to maximize the amount of entities while maintaining style balance. The annotation process was a trial-and-error, with cycles composed of annotation, discussing confusing entities, updating the annotation guide schematic and going through the corpus section again to correct entities following guide changes. The annotation process was done online, in BRAT. The actual annotation involved 4 people, has taken about 6 months (as work was volunteer-based, we could not have reached for 100% time commitment from the people involved), and followed the steps:nolistsepEach person would annotate the full corpus (this included the cycles of shaping up the annotation guide, and re-annotation). Inter-annotator agreement (ITA) at this point was relatively low, at 60-70%, especially for a number of classes.We then automatically merged all annotations, with the following criterion: if 3 of the 4 annotators agreed on an entity (class&start-stop), then it would go unchanged; otherwise mark the entity (longest span) as CONFLICTED.Two teams were created, each with two persons. Each team annotated the full corpus again, starting from the previous step. At this point, class-average ITA has risen to over 85%.Next, the same automatic merging happened, this time entities remained unchanged if both annotations agreed.Finally, one of the authors went through the full corpus one more time, correcting disagreements.We would like to make a few notes regarding classes and inter-annotator agreements:nolistsep[noitemsep]Classes like ORGANIZATION, NAT_REL_POL, LANGUAGE or GPEs have the highest ITA, over 98%. They are pretty clear and distinct from other classes.The DATETIME class also has a high ITA, with some overlap with PERIOD: annotators could fall-back if they were not sure that an expression was a PERIOD and simply mark it as DATETIME.WORK_OF_ART and EVENTs have caused some problems because the scope could not be properly defined from just one sentence. For example, a fair in a city could be a local event, but could also be a national periodic event.MONEY, QUANTITY and ORDINAL all are more specific classes than NUMERIC_VALUE. So, in cases where a numeric value has a unit of measure by it, it should become a QUANTITY, not a NUMERIC_VALUE. However, this "specificity" has created some confusion between these classes, just like with DATETIME and PERIOD.The ORDINAL class is a bit ambiguous, because, even though it ranks "higher" than NUMERIC_VALUE, it is the least diverse, most of the entities following the same patterns.PRODUCT and FACILITY classes have the lowest ITA by far (less than 40% in the first annotation cycle, less than 70% in the second). We actually considered removing these classes from the annotation process, but to try to mimic the OntoNotes classes as much as possible we decided to keep them in. There were many cases where the annotators disagreed about the scope of words being facilities or products. Even in the ACE guidelines these two classes are not very well "documented" with examples of what is and what is not a PRODUCT or FACILITY. Considering that these classes are, in our opinion, of the lowest importance among all the classes, a lower ITA was accepted.Finally, we would like to address the "semantic scope" of the entities - for example, for class PERSON, we do not annotate only proper nouns (NPs) but basically any reference to a person (e.g. through pronouns "she", job position titles, common nouns such as "father", etc.). We do this because we would like a high-coverage corpus, where entities are marked as more semantically-oriented rather than syntactically - in the same way ACE entities are more encompassing than CoNLL entities. We note that, for example, if one would like strict proper noun entities, it is very easy to extract from a PERSON multi-word entity only those words which are syntactically marked (by any tagger) as NPs.We have presented RONEC - the first Named Entity Corpus for the Romanian language. At its current version, in its 5127 sentences we have 26377 annotated entities in 16 different classes. The corpus is based on copy-right free text, and is released as open-source, free to use and extend.We hope that in time this corpus will grow in size and mature towards a strong resource for Romanian. For this to happen we have released the corpus in two formats: CoNLL-U PLus, which is a text-based tab-separated pre-tokenized and annotated format that is simple to use, and BRAT, which is practically plug-and-play into the BRAT web annotation tool where anybody can add and annotate new sentences. Also, in the GitHub repo there are automatic alignment and conversion script to and from the two formats so they could easily be exported between.Finally, we have also provided an annotation guide that we will improve, and in time evolve into a full annotation document like the ACE Annotation Guidelines for Entities V6.6 BIBREF8.

Q: Did they experiment with the corpus?

Q: What writing styles are present in the corpus?
A: current news, historical news, free time, sports, juridical news pieces, personal adverts, editorials.

Q: How did they determine the distinct classes?