Deep Neural Networks for Named Entity Recognition in Italian
Authors: Daniele Bonadiman, Aliaksei Severyn, Alessandro Moschitti (Università di Trento, Google Inc., Qatar Computing Research Institute, 2015)
Topic: Language Models

Abstract
In this paper, we introduce a Deep Neural Network (DNN) for engineering Named Entity Recognizers (NERs) in Italian. Our network uses a sliding window of word contexts to predict tags. It relies on a simple word-level log-likelihood as a cost function and uses a new recurrent feedback mechanism to ensure that the dependencies between the output tags are properly modeled. %These choices make our network simple and computationally efficient. The evaluation on the Evalita 2009 benchmark shows that our DNN performs on par with the best NERs, outperforming the state of the art when gazetteer features are used.

Introduction
Named Entity (NE) recognition is the task of detectings phrases in text, e.g., proper names, which directly refer to real world entities along with their type, e.g., people, organizations, locations, etc.~see, e.g., AUTHOR.%The NER task has been first proposed by the Message Understanding Conference (MUC 6, MUC 7) Conference on Natural Language Learning (CoNLL 2002, 2003) and Automatic Content Extraction program (ACE 2002, 2005). Recently the NER task has been extended to other languages, e.g., Italian (Evalita 2007, 2009). Most NE recognizers (NERs) rely on machine learning models, which require to define a large set of manually engineered features. % For example, the state-of-the-art (SOTA) system for English AUTHOR uses a simple averaged perceptron and a large set of local and non-local features. Similarly, the best performing system for Italian AUTHOR combines two learning systems that heavily rely on both local and global manually engineered features. Some of the latter are generated using basic hand-crafted rules (i.e., suffix, prefix) but most of them require huge dictionaries (gazetteers) and external parsers (POS taggers and chunkers). While designing good features for NERs requires a great deal of expertise and can be labour intensive, it also makes the taggers harder to adapt to new domains and languages since resources and syntactic parsers used to generate the features may not be readily available. Recently, DNNs have been shown to be very effective for automatic feature engineering, demonstrating SOTA results in many sequence labelling tasks, e.g., AUTHOR, also for Italian language AUTHOR. In this paper, we target NERs for Italian and propose a novel deep learning model that can match the accuracy of the previous best NERs without using manual feature engineering and only requiring a minimal effort for language adaptation. In particular, our model is inspired by the successful neural network architecture presented by AUTHOR to which we propose several innovative and valuable enhancements: (i) a simple recurrent feedback mechanism to model the dependencies between the output tags and (ii) a pre-training process based on two-steps: (a) training the network on a weekly labeled dataset and then (b) refining the weights on the supervised training set. Our final model obtains 82.81 in F1 on the Evalita 2009 Italian dataset AUTHOR, which is an improvement of + 0.81 over the AUTHOR system that won the competition. Our model only uses the words in the sentence, four morphological features and a gazetteer. Interestingly, if the gazetteer is removed from our network, it achieves an F1 of 81.42, which is still on par with the previous best systems yet it is simple and easy to adapt to new domains and languages.

Conclusion
In this paper, we propose a new DNN for designing NERs in Italian. Its main characteristics are: (i) the RCWN feedback method, which can model dependencies of the output label sequence and (ii) a pre-training technique involving a weakly supervised dataset. Our system is rather simple and efficient as it involves only one model at test time. Additionally, it does not require time-consuming feature engineering or extensive data processing for their extraction. In the future, we would like to apply rerankers to our methods and explore combinations of DNNs with structural kernels.