LIMA : A Multilingual Framework for Linguistic Analysis and Linguistic Resources Development and Evaluation

The increasing amount of available textual information makes necessary the use of Natural Language Processing (NLP) tools. These tools have to be used on large collections of documents in different languages. But NLP is a complex task that relies on many processes and resources. As a consequence, NLP tools must be both configurable and efficient: specific software architectures must be designed for this purpose. We present in this paper the LIMA multilingual analysis platform, developed at CEA LIST. This configurable platform has been designed to develop NLP based industrial applications while keeping enough flexibility to integrate various processes and resources. This design makes LIMA a linguistic analyzer that can handle languages as different as French, English, German, Arabic or Chinese. Beyond its architecture principles and its capabilities as a linguistic analyzer, LIMA also offers a set of tools dedicated to the test and the evaluation of linguistic modules and to the production and the management of new linguistic resources. 1 Context and objectives In this article, we present the LIMA (CEA List Multilingual Analyzer) platform which is, as GATE (Cunningham et al., 2002), together an architecture, a set of tools and resources and an environment for developing applications based on Natural Language Processing (NLP). This platform was developed by the LVIC laboratory of CEA LIST with the following requirements: • multilingualism, with the objective of dealing with a broad spectrum of languages; • a large diversity of applications. LIMA aims at being used as a basic component for various applications that can be text-based applications such as automatic summarization or question-answering but can also be applications dealing with multimedia documents; • extensibility, that is to say the ability to support the addition of new functionalities. As illustrated by Section 2.2, the current version of LIMA mainly performs analyses up to syntactic analysis but we also aim at extending it to semantic and discourse analyses; • the need for efficiency. A platform such as LIMA must be able to process very large corpora both because the processing of such corpora is more and more required by work in Computational Linguistics (see (Pantel et al., 2009) for instance) and because it also has to be used in an industrial context. The first three requirements make necessary to design an architecture based on modularity and flexibility at a high degree. All languages are not characterized by the same set of linguistic phenomena and as a consequence, their processing doesn’t rely on the combination of the same elementary analyses. Moreover, even if a linguistic analysis module can be used for two different languages, the linguistic resources it relies on are generally specific to each language. The same need for modularity and flexibility comes from the diversity of applications LIMA has to deal with: using the same system for lemmatizing a set of keywords from a base of images, a newspaper article or the transcription of a phone conversation is not the best means to have good results in each of these three contexts. Finally, the main difficulty LIMA had to face was to fulfill these requirements without sacrificing efficiency. Several kinds of architectures were already proposed to address these different issues. Process-oriented architectures focus on the combination and the control of a set of modules together with the communication between them. They generally implement a loosely integration by the means of a “glue” that can be a multi-agent system as in TalLab (Wolinski et al., 1998) or a client-server architecture as in FreeLing (Carreras et al., 2004). Data-oriented architectures also represent a weak type of integration by concentrating on the normalization of data between modules, as in the MULTEXT project (Ide and Véronis, 1994) or the LT XML Library (Brew et al., 1999). The TIPSTER-like architectures (Grishman, 1997) go a step further by imposing both a shared representation of data, often by annotation graphs (Bird and Liberman, 1999), and a uniform interface for modules. The GATE platform (Cunningham et al., 2002) is the typical representative of this kind of architectures but TEXTRACT (Neff et al., 2004) and its most recent descendant, UIMA (Ferrucci and Lally, 2004), also belong to this category. Finally, the highest degree of integration is reached by formalism-oriented architectures in which both data and processes are represented through a declarative formalism associated to a kind of inference engine. This approach was initially dedicated to the development of grammars as in ALVEY (Grover et al., 1993) but was also applied more widely through platforms such as ALEP (Simkins, 1994), NooJ (Koeva et al., 2007) or Outilex (Blanc et al., 2006). As it will be illustrated in the following sections, we chose a TIPSTER-like architecture for LIMA as it represents the best trade-off between modularity and efficiency, which are