Introduction
Nowadays, a large amount of information is produced and shared in unstructured form, mostly unstructured text BIBREF0 , BIBREF1 . This information can be exploited in decision making processes but, to be useful, it should be transformed and presented in ways that make its intrinsic knowledge more readily intelligible. For that, we need efficient methods and tools that quickly extract useful information from unstructured text collections. Such demand can be observed, for instance, in Biology, where researchers, in order to be abreast of all developments, need to analyse new biomedical literature on a daily basis BIBREF2 . Another application is on fraud and corruption studies where the network information — the set of actors and their relationships — is implicitly stored in unstructured natural-language documents BIBREF3 . Hence, text mining and information extraction are required to pre-process the texts in order to extract the entities and the relations between them.
Information extraction is a challenging task mainly due to the ambiguous features of natural-language. Moreover, most tools need to be adapted to different human languages and to different domains BIBREF4 . In fact, the language of the processed texts is still the decisive factor when choosing among existing information extraction technologies. This is also true for the task of entity extraction (Named Entity Recognition - NER).
For several reasons, text mining tools are typically first developed for English and only afterwards extended to other languages. Thus, there are still relatively few text mining tools for Portuguese and even less that are freely accessible. In particular, for the named entities recognition task in Portuguese texts, we find three extractors available: Alchemy, Zemanta and Rembrandt BIBREF5 . We also find some studies where the measures ( INLINEFORM0 , INLINEFORM1 and INLINEFORM2 ) for those extractors are computed and compared BIBREF6 , but their comparative effectiveness remains domain and final purpose dependent.
In this work, we present PAMPO (PAttern Matching and POs tagging based algorithm for NER), a new method to automatically extract named entities from unstructured texts, applicable to the Portuguese language but potentially adaptable to other languages as well. The method relies on flexible pattern matching, part-of-speech tagging and lexical-based rules. All steps are implemented using free software and taking advantage of various existing packages.
The process has been developed using as case-study a specific book written in Portuguese, but it has since been used in other applications and successfully tested in different text collections. In this paper, we describe the evaluation procedures on independent textual collections, and produce a comparative study of PAMPO with other existing tools for NER.
Related Work
In 1991, Lisa F. Rau presented a paper describing an algorithm, based on heuristics and handcrafted rules, to automatically extract company names from financial news BIBREF7 . This was one of the first research papers on the NER field BIBREF8 . NER was first introduced as an information extraction task but since then its use in natural language text has spread widely through several fields, namely Information Retrieval, Question Answering, Machine Translation, Text Translation, Text Clustering and Navigation Systems BIBREF9 . In an attempt to suit the needs of each application, nowadays, a NER extraction workflow comprises not only analysing some input content and detecting named entities, but also assigning them a type and a list of URIs for disambiguation BIBREF10 . New approaches have been developed with the application of Supervised machine Learning (SL) techniques BIBREF6 and NER evolved to NERC — Named Entity Recognition and Classification. The handicap of those techniques is the requirement of a training set, i.e., a data set manually labelled. Therefore, the NER task depends also on the data set used to train the NER extraction algorithm.
Currently, many existing approaches for NER/NERC are implemented and available as downloadable code, APIs or web applications, i.e., as tools or services available on the web. A thorough search produces the following list: AIDA, AlchemyAPI, Apache Stanbol, CiceroLite, DBpedia Spotlight, Evri, Extractiv, FOX, FRED, Lupedia, NERD, Open Calais, PoolParty Knowledge Discoverer, Rembrandt, ReVerb, Saplo, Semiosearch Wikifier, Wikimeta, Yahohh! Content Analysis (YCA), Zemanta. More detailed information may be found in BIBREF10 , BIBREF11 , BIBREF12 , BIBREF5 , where the authors compare the services' strengths and weaknesses and compute some measures for their performance.
Nadeau et al. in A survey of named entity recognition and classification BIBREF8 point out three factors that distinguish the NERC algorithms: the language, the textual genre or domain, and the entity type. Regarding the third one, based on the Grishman et al. definition BIBREF13 , named entity refers to the name of a person or an organization, a location, a brand, a product, a numeric expression (including time, date, money and percentage), found in a sentence, but generally, the most studied types consider the enamex designation — proper names of `persons', `locations' and `organizations' — the `miscellaneous' category for the proper names that fall outside the classic enamex). In recent research , the possible types to extract are open and include subcategories BIBREF8 .
The language is an important factor to be taken in consideration in the NER task. Most of the services are devoted to English and few support NER on Portuguese texts. The first reference to work developed in Portuguese texts was published in 1997 BIBREF14 ; the authors perform the NER task and compute some measures in a Portuguese corpus and other five corpora. Until now, we have only identified the Rembrandt tool as a service developed and devoted to extract named entities in Portuguese texts. Other tools (AlchemyAPI, NERD and Zemanta) have been adapted to work and accept Portuguese texts but were not specifically developed for that purpose. As recently pointed out by Taba and Caseli BIBREF15 , the Portuguese language still lacks high quality linguistic resources and tools.
NER is not only one task of the text mining process but also an initial step in the performance of other tasks, such as relation extraction, classification and/or topic modelling BIBREF0 . This makes the quality of the NER process particularly important. In the light of the related works and taking in consideration that most of the approaches optimize INLINEFORM0 but not INLINEFORM1 , we propose PAMPO to extract named entities in Portuguese texts. In this work we do not classify neither disambiguate the entity. Our major concern is to increase the INLINEFORM2 without decreasing the INLINEFORM3 of the named entity extractor.
The entity extraction algorithm
In this work, we consider the enamex definition of entities plus the miscellaneous named entities where we include events like, for instance, `Jogos Olímpicos' (`Olympic Games'). To identify those entities, an information extraction procedure was designed using regular expressions and other pattern matching strategies, along with part-of-speech tagging, i.e., employing a Part-of-Speech Tagger (POST) tool. The extraction of the named entities from Portuguese unstructured texts is composed of two phases: candidate generation, where we generate a superset of candidate entities, and entity selection, where only relevant candidates are kept. The two phases are described in Algorithms SECREF3 and SECREF3 , respectively.
PAMPO - Candidate Generation In this phase, we provide a customizable base of regular expressions that gathers common candidate entities. Typical expressions capture capitalized words, personal titles (president, deputy, etc.) and other common words (assembly). This patterns' base is extendable and the aim of the process in this phase is to identify all good candidates.
PAMPO - Candidate Generation
Input: INLINEFORM0 , INLINEFORM1 :Term Pattern Base INLINEFORM2 INLINEFORM3 is the set of candidate entities each sentence INLINEFORM4 in INLINEFORM5 each term pattern INLINEFORM6 in TPB INLINEFORM7 sub-sequences of INLINEFORM8 that match INLINEFORM9 Output: INLINEFORM10
PAMPO - Entity Selection Here, all candidate entities of the previous phase are part-of-speech tagged. The POST process tags tokens with their corresponding word type (lexical category). Based on the tagging of the terms in candidate entities, we can identify some that can be discarded. This is done by applying a second level of regular expressions. In the entity selection phase, the regular expressions are defined on the lexical categories instead of terms themselves. For example, if the first word type is a `pron-det' (POS tag meaning determiner pronoun) the word is removed. Another example is the removal of candidate entities that do not have at least one tag `prop' or `n' (POS tag meaning a proper noun and a noun).
PAMPO - Entity selection
Input: INLINEFORM0 : candidate entities, INLINEFORM1 : category clipping patterns, INLINEFORM2 : category pruning patterns, INLINEFORM3 : term pruning pattern base each candidate entity INLINEFORM4 in INLINEFORM5 INLINEFORM6 POST of the candidate entity INLINEFORM7 each clipping pattern INLINEFORM8 in INLINEFORM9 INLINEFORM10 matches prefix of INLINEFORM11 remove matching prefix from INLINEFORM12 remove corresponding prefix from INLINEFORM13 each pruning pattern INLINEFORM14 in INLINEFORM15 INLINEFORM16 matches INLINEFORM17 INLINEFORM18 each pruning pattern INLINEFORM19 in INLINEFORM20 INLINEFORM21 = INLINEFORM22 INLINEFORM23 Output: modified INLINEFORM24
Implementation
The program was developed in R BIBREF16 and makes use of some specific text mining packages. We have implemented our method using the following R packages: tm BIBREF17 , cwhmisc BIBREF18 , memoise BIBREF19 , openNLP BIBREF20 , Hmisc BIBREF21 . The OpenNLP POS Tagger uses a probability model to predict the correct POS tag and, for Portuguese language, it was trained on CoNLL_X bosque data.
An application
The INLINEFORM0 , INLINEFORM1 , INLINEFORM2 and INLINEFORM3 bases adopted, for Portuguese texts, and used in this application are described in this section. As a first approach, and to test the PAMPO algorithm, we selected a book about the Portuguese Freemasonry BIBREF22 . Despite being on a specific topic, it contains a rich variety of situations to test our extractor. As an example, the piece of text shown in Figure FIGREF25 was scanned from the book with current OCR software and will be used here to highlight the contribution of each phase to the final result. The five named entities manually identified in this piece of text are `Irmandade do Bairro Ut O', `Parlamento do G', `Jorge Silva', `Ian' and `ministro Miguel Relvas'.
Applying Algorithm 1 to the paragraph of Figure FIGREF25 , the set of `candidate entities' found are `Irmandade do Bairro Ut O', `Conhecemos', `Parlamento do G', `L', `K', `Jorge Silva', `Ian' and `ministro Miguel Relvas'. Although most of the words in the extracted `candidate entities' list start with capital letter, with this algorithm we were able to extract also other important words that are not capitalized like the first word in the last named entity (ministro). This is possible because the INLINEFORM0 base includes a set of patterns that captures not only words (or sequence of words) starting with capital letters but also words that are associated to some entity's name like the ones in list1 on Appendix A.
Having collected the `candidate entities' in the previous step, we now proceed by removing from that list the ones that do not correspond to named entities. For that purpose, we use list2 (see Appendix A) as INLINEFORM0 base, all the tags that are not a noun ( INLINEFORM1 ) or a proper noun ( INLINEFORM2 ) are included in the INLINEFORM3 base and, finally, some terms that are not named entities but that were not excluded by previous actions (see list3 on Appendix A), are used as INLINEFORM4 base. Applying Algorithm 2 with those lists to the set of `candidate entities', from Figure FIGREF25 , we obtain as named entities `Irmandade do Bairro Ut O', `Parlamento do G', `Jorge Silva', `Ian' and `ministro Miguel Relvas'. In fact, these five terms are the only named entities in the paragraph.
Analysis of results
Table TABREF27 shows the most frequent `candidate entities' from the whole book, as extracted by Algorithm 1 and which of those candidate entities were considered as actual `named entities' by Algorithm 2.
To give an idea of the improvement introduced by each phase, we represent the `candidate entities' set in a word cloud where words with higher frequency have larger font size. As it can be observed in Figure FIGREF28 , after phase 1 some words that do not refer to entities, such as `Idem'(`Idem'), `Entre' (`Between') and `Nas' (`At the'), are present in the cloud, but, as expected, they disappear in phase 2.
From this book, a total of 12120 named entities were extracted by PAMPO, corresponding to 5159 unique named entities. To assess the quality of this process, the first 125 pages of the book were manually labelled (1/3 of the text book). The values of the computed measures are shown in Table TABREF29 . This part of the book contains 3836 named entities. INLINEFORM0 and INLINEFORM1 are estimated for the two phases based on the results obtained on the 125 pages of the book. A total of 5089 terms were labelled `candidate entities' in the first phase and 3075 were identified as `named entities' in the second phase. The true positives were 3205 in the first phase and 2982 in the second phase (partial identifications count as 1/2). This means that the INLINEFORM2 , given by Equation ( EQREF30 ), decreases from 0.84 to 0.78, and the INLINEFORM3 , given by Equation ( EQREF31 ), increases from 0.63 to 0.97. DISPLAYFORM0 DISPLAYFORM1
Equation ( EQREF32 ) defines another measure commonly used to assess the quality of the process, INLINEFORM0 . This measure allows interpreting the global quality, taking into account the decrease of INLINEFORM1 and the increase of INLINEFORM2 . The second phase of the PAMPO process increases the value of INLINEFORM3 from 0.72 to 0.87. DISPLAYFORM0
After these illustrative results of the PAMPO algorithm, the following section presents the results of a comparison between PAMPO and other approaches to extract named entities from texts in Portuguese.
Comparing PAMPO with other NER tools
In this work, we evaluate our NER approach using two news corpora. One corpus is a set of 227 texts published on December 31, 2010 by the Lusa agency (portuguese agency of news) and will be referred to as `News'. The other corpus (named here `Sports news') is a set of 881 sports news. The texts were manually annotated according to the enamex designation and the type `miscellaneous'.
Each of the corpora used for evaluation has a considerable number of texts but with different characteristics. The `Sports news' corpus has text from only one domain, while the `News' presents a diversity of topics. This fact allows evaluating if the domain/topic factor can significantly affect the quality of the algorithm. Some features of the two corpora are present in Table TABREF33 . The minimum text length in words is 24 for the `News' corpus and 59 for `Sports news'. The maximum lengths are 770 and 445 respectively. The total named entities manually found for each type range between 798 and 7051 with an average of 16.4 entities (without type distinction) per text.
In this work we not only study the quality of the PAMPO NER extractor for Portuguese texts but we also compare the results with three other extractors. Two of them, AlchemyAPI and Zemanta, are easily accessed with the tool developed by Bartosz Malocha in EURECOM and available on the web. The other one, Rembrandt, has to be downloaded and locally installed, which is not a straightforward task.
PAMPO output
Considering the Portuguese text represented in Figure FIGREF37 (a) the PAMPO algorithm identifies the `named entities' listed in Figure FIGREF37 (b).
As can be observed by this example, the algorithm extracts all the manifestations of `named entities' and lists them in the order they appear in the text, including repetitions of the same `named entity'.
Evaluation
To compare the results of PAMPO with the other NER extractors, we compute the INLINEFORM0 and INLINEFORM1 considering a unique occurrence per entity, instead of all named entities occurrences. Figure FIGREF39 presents the outputs of the four extractors, PAMPO, AlchemyAPI, Rembrandt and Zemanta, for the text in Figure FIGREF37 (a).
To compute the INLINEFORM0 , INLINEFORM1 and INLINEFORM2 measures presented in Table TABREF40 , we used Equations EQREF30 , EQREF31 and EQREF32 with a difference in the weight given to the partial identifications. Based on the example in Figure FIGREF39 , we observed that not all partial correspondences to the named entity on the text have necessarily the same value, i.e., `Atlanta', `Atlanta 1996', `Jogos Olímpicos' or `Jogos Olímpicos de Atlanta' as partial identifications of `Jogos Olímpicos de Atlanta 1996' do not have the same information. Hence we adopted as weight criterion for the partial identifications, the fraction of the named entity that is identified. This means that the previous partial identifications have weights of INLINEFORM3 , INLINEFORM4 , INLINEFORM5 and INLINEFORM6 , respectively. As a result, two extractors will have the same performance even if one identifies the complete named entity `Jogos Olímpicos de Atlanta 1996' and the other splits it into two named entities, `Atlanta 1996' and `Jogos Olímpicos'.
Analysing the mean values of INLINEFORM0 , INLINEFORM1 and INLINEFORM2 (standard deviation between parentheses) given in Table TABREF40 , it is easy to conclude that they are higher in the `Sports news' for all the extractors. Moreover, that difference is less noted in the PAMPO algorithm, which presents better results and a much higher mean INLINEFORM3 , and consequently higher mean INLINEFORM4 , than the other three extractors. The four extractors have similar mean INLINEFORM5 but none has better mean INLINEFORM6 than the PAMPO extractor. The mean INLINEFORM7 , mean INLINEFORM8 and mean INLINEFORM9 for the PAMPO algorithm are consistent with a good performance of the extractor. To further assess the quality of the extractors, the probability density function of the three measures for the two corpora, estimated using a kernel density estimation with 100 equally spaced points (MATLAB 7.10.0 (R2010a)), are plotted in Figure FIGREF41 . As expected, the probability density is higher around the value 1 for all the measures of PAMPO extractor on the two corpora.
Figure FIGREF42 presents scatter plots of INLINEFORM0 vs INLINEFORM1 for the four extractors, PAMPO, AlchemyAPI, Rembrandt and Zemanta for the `Sports news' and `News' corpora, first four panels and four bottom panels, respectively. It is noteworthy that almost all the 881 points of the `Sports news' for PAMPO extractor are in the upper right corner of the scatter plot, as well as almost all the 227 points of the `News'. The other tools present a more dispersed solution quality.
Evaluation by type of entity
To determine if the entity type contributes to output variability in the INLINEFORM0 , an analysis was conducted on the named entities for the classification types: `persons' (PER), `locations' (LOC), `organizations' (ORG) and `miscellaneous' (MISC).
The results (Figure FIGREF44 ) indicate that the INLINEFORM0 varies with the type of entity for the AlchemyAPI, Rembrandt and Zemanta but not for the PAMPO. The INLINEFORM1 of PAMPO extractor is the highest for all types of entities.
In summary, it is apparent from the analysis that PAMPO extracts a set of `named entities' that resembles the actual list of named entities on texts.
To complete the evaluation we also computed INLINEFORM0 , INLINEFORM1 and INLINEFORM2 of PAMPO extraction on the texts in Coleção Dourada-HAREM . This corpus has 129 documents. Using the evaluation criterion defined by curators of HAREM, we obtain a INLINEFORM3 of INLINEFORM4 , a INLINEFORM5 of INLINEFORM6 and a INLINEFORM7 of INLINEFORM8 considering all the categories. Considering that the PAMPO extractor was not designed to extract quantities or time expressions we computed the same measures excluding these two types of entities. While INLINEFORM9 practically keeps the same value ( INLINEFORM10 ), INLINEFORM11 and INLINEFORM12 increase to INLINEFORM13 and INLINEFORM14 , respectively.
PAMPO versus three other extractors
Now, we analyse the differences between measures obtained with PAMPO and with the three other extractors, for each one of the news on the two corpora. To perform a more informative comparison between PAMPO and the other extractors, we count the number of news items that had a positive, a null and a negative difference with respect to each measure and each concurrent extractor. These are summarized in Table TABREF47 for both corpora.
The mean and the standard deviation (between parentheses) for each extractor and each corpus are presented in Table TABREF48 . They will be used to test statistical hypotheses about the mean difference value of INLINEFORM0 , INLINEFORM1 and INLINEFORM2 between PAMPO and the other three extractors.
Based on all the values of the differences between PAMPO and the other extractors, represented in Tables TABREF47 and TABREF48 , we may say that:
the INLINEFORM0 of the PAMPO extractor is the highest in almost all the news;
INLINEFORM0 does not differ much between PAMPO and the other extractors;
as a consequence the INLINEFORM0 of PAMPO is also the highest in almost all the news;
the mean difference of INLINEFORM0 between PAMPO and AlchemyAPI seams to be at least greater than 0.25;
the mean difference of INLINEFORM0 between PAMPO and Rembrandt seams to be at least greater than 0.35;
the mean difference of INLINEFORM0 between PAMPO and Zemanta seams to be at least greater than 0.40;
the mean difference of INLINEFORM0 is positive but near zero for all the three extractors;
the mean difference of INLINEFORM0 between PAMPO and AlchemyAPI seams to be at least greater than 0.15;
the mean difference of INLINEFORM0 between PAMPO and Rembrandt seams to be at least greater than 0.25;
the mean difference of INLINEFORM0 between PAMPO and Zemanta seams to be at least greater than 0.30.
To test the null hypothesis that the mean INLINEFORM0 differences between PAMPO and the other extractors are equal to 0.25, 0.35 and 0.40, for AlchemyAPI, Rembrandt and Zemanta, respectively, ztest was performed considering as alternative the mean INLINEFORM1 differences greater than those values. Based on the results of these two corpora the p-values are smaller than 9.5E-05. Hence, the results obtained so far provide statistical evidence that PAMPO increases NER INLINEFORM2 by at least 0.25.
Remarks and Conclusions
In this work we propose a novel effective method to extract named entities from unstructured text. The proposed PAMPO method is implemented using free software, namely R and available packages. Two manually annotated Portuguese news corpora were used to empirically evaluate the algorithm using the measures of INLINEFORM0 , INLINEFORM1 and INLINEFORM2 . These corpora did not influence the definition of the algorithm or the construction of its pattern bases. We have compared PAMPO with three other NER extractors: AlchemyAPI, Rembrandt and Zemanta. Experimental results clearly show that PAMPO obtains significantly higher INLINEFORM3 and INLINEFORM4 than existing tools. The values of INLINEFORM5 are identical. We may say also that PAMPO's performance in the HAREM corpus was at least as good as the best one of the systems reported over there when we consider all categories of entities. However, when we exclude dates and numeric expressions, it presents better results than the ones reported for other tools.
Despite its simplicity, PAMPO has a very good performance and is highly configurable. The PAMPO algorithm is potentially adaptable to be used for other languages by properly defining the pattern bases. Furthermore, it allows for straightforward improvement of the results by adding terms to the lists.
The results take us one step closer to the creation of a text intelligence system to be used in several applications, namely, in the study of the social context of possible economic and financial offenses. As future work the authors are planning to improve the text mining procedure, by including a classification and a disambiguation step, as well as by automatically characterizing the relations between entities.
Acknowledgements
The authors would like to thank SAPO Labs (http://labs.sapo.pt) for providing the data set of news from Lusa agency. The authors would also like to thank grant #2014/08996-0 and grant #2013/14757-6, São Paulo Research Foundation (FAPESP). This work is partially funded by FCT/MEC through PIDDAC and ERDF/ON2 within project NORTE-07-0124-FEDER-000059 and through the COMPETE Programme (operational programme for competitiveness) and by National Funds through the FCT - Fundação para a Ciência e a Tecnologia (Portuguese Foundation for Science and Technology) within project FCOMP-01-0124-FEDER-037281.
Appendix
list1 - {'grão INLINEFORM0 mestre', 'papa', 'duque', 'duquesa', 'conde', 'condessa', 'visconde', 'viscondessa', 'rei', 'raínha', 'príncipe', 'princesa', 'marquês', 'marquesa', 'barão', 'baronesa', 'bispo', 'presidente', 'secretário', 'secretária', 'ministro', 'ministra', 'primeiro', 'primeira', 'deputado', 'deputada', 'general', 'tenente', 'capitão', 'capitã', 'sargento', 'governador', 'governadora', 'diretor', 'director', 'diretora', 'directora', 'ex', 'filho', 'filha', irmão', 'irmã', 'pai', 'mãe', 'tio', 'tia', 'padrinho', 'madrinha', 'sobrinho', 'sobrinha', 'afilhado', 'afilhada', 'avó', 'avô', 'neto', 'neta', 'enteado', 'enteada', 'padrasto', 'madrasta'}
list2 - {'pron-det', 'adv adv ', 'adv prop', 'adv adj ', 'adv v-fi'}
list3 - {'Aproveitamento', 'Cuidado', 'Decerto', 'Desta', 'Desenvolvimento', 'Lançamento', 'Levantamento', 'Muitos', 'Muitas', 'Nessa', 'Nesse', 'Nessas', 'Nesses', 'Nestes', 'Neste', 'Nesta', 'Nestas', 'Noutro', 'Outros', 'Outro', 'Outra', 'Outras', 'Onde', 'Poucos', 'Poucas', 'Perante', 'Pela', 'Recém', 'Tal', 'Vários', 'Várias', 'Vós', 'Aceite', 'Comprometo', 'Cabe', 'Coloca', 'Conhecemos', 'Casado', 'Considerava', 'Desejo', 'Devíamos', 'Escolhiam, 'Executa', 'Faça', 'Fica', 'Interrompidas', 'Indicar', 'Incluído', 'Leva', 'Morrer', 'Ouvistes', 'Prestaste', 'Praticou', 'Pressiona', 'Pensa', 'Poder', 'Podes', 'Revolta', 'Sabe', 'Ser', 'Ter', 'Toque', 'Toma', 'Trata', 'Vens', 'Verificou', 'Viver', 'Vivemos', 'Venho', 'Reação', 'Sessão', 'Testamento', 'Tolerância', 'Término', 'Vitória', 'Visita', 'Harmonia', 'Iniciado', 'Instalação', 'Ibidem', 'Inventariação', 'Irregularidades', 'Internet', 'Lda', 'Manutenção', 'Nomeado', 'Obediência', 'Petição', 'Passaporte', 'Proposta', 'Programa', 'Proibição', 'Paz', 'Publicação', 'Questionário', 'Quadro', 'Relatório', 'Redução', 'Reorganização','Revolução', 'República', 'Reequilíbrio', 'Anexo', 'Abertura', 'Atestado', 'Ata', 'Adoção', 'Atualização', 'Às', 'Á', 'Capa', 'Convite', 'Compromisso', 'Condecoração', 'Convocatória', 'Cartão', 'Causa', 'Comunicação', 'Corrupção', 'Convergência', 'Decreto', 'Ditadura', 'Democracia', 'Democrata', 'Estrutura', 'Ficha', 'Fax', 'Fixação', 'Futuro', 'Gabinete', 'Glória', 'Janeiro', 'Fevereiro', 'Março', 'Abril', 'Maio', 'Junho', 'Julho', 'Agosto', 'Setembro', 'Outubro', 'Novembro', 'Dezembro', Diário', 'Semanal', 'Mensal', 'Minutos', 'Meses', 'Ano', 'Anos', 'Hoje'} INLINEFORM0 {Portuguese stopwords on R}

Question: what dataset was used?
Answer: CoNLL_X bosque data, News data by Lusa agency, Sports news data

News, Sports news

News, Sports news

Question: by how much did their model improve over current alternatives?
Answer: On average, it had better Recall by 0.481 in case of news dataset and by 0.372 in case of sports news dataset. 
On average, it had better Precision by 0.086 in case of news dataset and by 0.37 in case of sports news dataset. 
On average, it had better F1 by 0.381 in case of news dataset and by 0.616 in case of sports news dataset. 

Pampo had F1 score of 0.932 and 0.971 compared to best alternative result of 0.608 and 0.794 on News and Sport news dataset respectively.

Question: did they experiment with other languages besides portuguese?
Answer: No

No

No

Question: how many rules did they use?
Answer: Unanswerable