Multilingual Content Processing

This contribution describes the consequences of a multilingual set-up, as used in internet information gathering, search and content processing, for the architecture and the different components of such a system. First the scenario is briefly outlined; then the existing technology components are reviewed, with the focus of the effects of multilingual content to such components. Finally, the linguistic resources are discussed which form the backbone of such a multifunctional and multilingual system. It is shown that adding multilinguality to an information system has massive consequences for the design of all system components and resources. 1 The Crosslingual Scenario Information Acquisition has become a major challenge in the internet age. The amount of information to be monitored has grown significantly, but time and resources to perform this task are still constrained. Tools like personalised electronic news clipping, automatic knowledge mining, or internet monitoring try to match the new requirements. As a result, a scenario needs to be designed which requires massive natural language support to meet its goals: • automatic classification of incoming texts is used as a first-level filter, to get rid of irrelevant material • key word and index term extraction allows for a finer-graded determination of the topic of a document. Links to some classification or ontology scheme take the analysis beyond pure text search into the field of knowledge mining and the semantic web. • Named entity recognition and fact extraction take care of the fact that people do not search strings but semantically meaningful objects, i.e. content. Profiles of interest look for content, not just string comparison. Such information systems (developed e.g. in the ECSENSUS project and described in (Thurmair, 2004)) face an additional challenge now, as multilinguality is a major obstacle for efficient information access. In globalised contexts, relevant information is available in a variety of languages, many of them not spoken by the information recipient. Queries, interest profile definitions etc. are stated in the interface language (i.e. the users’ native language) but the documents and the information sources to be looked up are in different languages, the document languages. One could claim now that the extension consists only of adding two additional translation steps, one on the query side, and one on the answering side (to re-translate hits e.g. in Korean into the interface languages, or somehow make the content available for further use), and perhaps a language identification tool to determine the document language. However, this is not quite the case; indeed there are massive consequences on the single components and technologies used, if multilingual aspects should be considered. This refers to the design of such components themselves (section 2 below) as well as the design of the linguistic resources required (section 3). It is known that some technologies are restricted to, or give better results in, specialised domains. This fact corresponds to a difference in the targeted markets, i.e. there is a difference between the corporate market (large internationalised companies), and the general internet market. While the former can often be constrained to specific applications (like: automotive, banking), and therefore produce higher quality in linguistic support tools, the internet applications must cope with any request. Experience in Machine Translation but also Information Extraction shows that results can be better, and systems are more likely to be accepted by users, if they can be tailored to a given application or context; general purpose tools are much more difficult to design. 2 Challenges in crosslingual Scenarios A closer look into the scenario just described shows that the requirement of multilinguality influences the architecture and the design of all components of an information system; this section discusses some of them. 2.1 Techniques for Query Analysis and Expansion This section discusses the query part of a crosslingual scenario, and design adaptations on the search components. Definition of Search terms The standard practice of analysing a search request is to remove the stop words, stem the content words, create some link between them (e.g. Boolean operators) and send the query to the search backend. This is not simply feasible in a crosslingual environment. • conventional techniques to increase recall (like truncation or stemming) are not applicable because the resulting strings often cannot be translated: If integration is stemmed to integrthis is a nontranslatable string. Therefore they cannot be searched in the document languages, and so they decrease the overall. system quality. • It is known that most terms in special languages are multiword terms. Translating the single words very often leads to useless document language queries, and to completely irrelevant search results: Searching for en nuclear power plants (de Atomkraftwerk) fails if nuclear (->Kern-), power (->Macht) and plant (-> Pflanze) are translated in isolation.