Multilingual Systems : Grammar Acquisition by Machine Learning ©

Multilingual systems design is discussed from the point of view of machine learning methods applicability for grammar acquisition. The key idea is developing a synergistic approach combining semantic grammar rules with the machine learning mechanisms of grammar rules extraction from text corpora. This work is the further development of the methods and the rule system presented in [16], which were employed in machine translation system “Cognitive Translator”. 1 Synthetic Approach to Multilingual Systems Design This paper highlights some pressing issues of natural language syntax and semantics which are of prime importance to the tasks of multilingual systems design and implementation. The approach proposed consists in synergistic employment of linguistic rulebased methods presenting the core grammar for the tasks of machine translation and knowledge management, and machine learning techniques for new grammar rules acquisition and disambiguation of structures. The following fundamental features of the given work differentiate it from other projects: the emphasis on the semantics of grammar, i.e. the study of language configurations basing on the Functional Transfer Fields (FTF) [16] and their projections into particular language structures; this will result in developing a computational variant of a multilingual semantic grammar (MSG), and the semantic grammar applicable for language learning by students; MSG is a projection of the Multivariant Cognitive Transfer Grammar (MCTG) designed in our project for the English-Russian correspondencies onto a wider range of languages including German, French and Italian; MSG development is dominant to the lexical semantic studies which are conducted on the basis of the existing English-Russian computational vocabulary obtained from parallel text corpora; The semantic grammar is used for the establishment of regular correlates between structures (configurations) and lexical units, i.e. cross-level correspondences; the construction of the systemic crosslingual presentation of phrase structures conveying the similar meanings in the languages under study and employing it as the core of the rule system for the algorithms of syntactic-semantic transfer in machine translation, multilingual knowledge management and language learning systems; the synthetic approach enables the system to generalize the rules and avoid the overgeneration of rules thus resulting in translation accuracy improvement. The texts of business and scientific discourse have been studied, and specific structures are extracted and serve the source for further rule set development. Experiments will be made with the texts of corporative documents and patents. 2 Machine Learning Techniques in Natural Language Processing The research in the NLP area is substantially stimulated by the fact that the market for MT grows mature in 2002 2004, and more corporations realize that the implementation of a customized MT solution can give a great advantage to the company and make them lead in the competition for today's multilingual customers. The aim of machine learning is to infer automatically a model for some domain on the basis of the given data from the domain, thus a system learning syntactic rules would be supplied with a set of phrase structure rules to be used for training. Recently more attention has been paid to the construction of N-grams capturing sophisticated presentations of syntactic and semantic structures: using long-distance triggers instead of local N-grams [23,20], applying variable-length Ngrams [19], including semantic information to the Ngrams, e.g. semantic word associations based on the latent semantic indexing [14]. The learning algorithms can be of two types: unsupervised and supervised. An unsupervised algorithm has to induce a model capable of generalizing to new data it hasn’t been given before, and does this purely from the data. A supervised algorithm is trained on a set of correct answers to the learning data, so that the induced model would result in more accurate decisions. At present we employ the latter approach in our work using Bayesian methods operating on the phrase structure weights obtained from the text corpora. This model will be developed into the probabilistic functional tree substitution grammar (PFTSG) mechanism operating with the system of multivariant cognitive transfer rules. Machine learning is largely founded on the stochastic research paradigm rooted in the development of speech recognition algorithms, character recognition, spelling correction. An essential probabilistic framework employs the Bayes Rule and the noisy channel model which play an important role in many problems, in particular, part-of-speech tagging and probabilistic parsing. The algorithm which is essentially employed within this architecture is known as dynamic programming algorithm. The recent decade has witnessed a considerable progress of natural language processing (NLP) techniques based on machine learning. The appearance of large parallel texts corpora promoted the statistical methods of NLP which now augment the scheme of the principal existing approaches to machine translation (MT) design – direct translation, transfer and interlingua-based methods. A statistical machine translation was first introduced by [5,6]. The starting point for any natural language processing system is tagging module design. Different stochastic taggers appeared in the 1980s. The idea shared by all stochastic taggers consists in choosing the most likely tag for a given word. One of the most popular probabilistic taggers is the Hidden Markov Model (or HMM tagger) for a given sentence or word sequence, HMM taggers pick the tag sequence that maximizes the following formula: P(word| tag) * P(tag|previous n tags). An approach to machine learning based on rules and stochastic tagging is known as the TransformationBased Learning (TBL). TBL is a supervised learning technique, and employs a pre-tagged training corpus. For probabilistic parsing stochastic grammars are applied. A Probabilistic Context-Free Grammar (PCFG) is a 5-tuple G = (N,T,P,S,D), where N is a set of nonterminal symbols, T is a set of terminal symbols, P is a set of productions of the form A -> b, where A is a nonterminal symbol, b is a string of symbols, S is a designated start symbol, D is a function assigning probabilities to each rule in P. Probabilistic tree substitution gramar (PTSG) : the definition is the same as PCFG but rather than a set of rules, we have a set of tree fragments of arbitrary depth whose top and interior nodes are nonterminals and whose leaf nodes are terminals or nonterminals, and the probability function assigns probabilities to these fragments. PTSGs are thus a generalization of PCFGs, and are stochastically more powerful , because one can give particular probabilities to fragments or even whole parses which cannot be generated as a multiplication of rule probabilities in a PCFG. We considered both TBL and PTSG when working out the way how to develop the stochastic learning apparatus for our formalism of Multivariant Cognitive Transfer Grammar (MCTG). The accepted approach in the initial variant was the use of Bayesian methods operating on the phrase structure weights. The development of speech-to-speech translation systems [15,10] substantially stimulated the research in the field of MT. The existing computational resources provided for today’s MT systems allow to accumulate and recall previously corrected translations (Translation Memory and Example-based Machine Translation) [27,7]. A model for machine translation based on an aligned text corpus is example-based machine translation, which means that the example-based translation employs the closest match in aligned corpora as a template for translation. The descriptions of example-based MT systems are given in [18,24]. The evidence of the latest research and development projects shows that machine learning methods alone are unlikely to yield the finely tuned language processing decisions. However the use of combined techniques brings the increase of “meaningful” performance of language processing systems in different aspects, thus it was exhibited in [25] that introduction of linguistic (morphological) parse rules into the search engine considerably enforces the precision of search results. New solutions and combinations of methods are being set forward. It is possible to say that at the present moment the systems solely based on translation memory are giving way to the systems which comprise several complementary techniques: though the systems based on the principles of Translation memory produce understandable translations, they still lack grammatical accuracy. 3 The State of the Art in Present Day Machine Translation When considering the performance of the systems we relied rather on subjective evaluation of translations correctness than any tools giving numeric expression of translation quality. The task to develop such tools is of great complicity, though at present some interesting methods have been set forward [13]. The projects were analyzed taking into account underlying models employed, the degree of particularity in semantic presentations, methods used for language parse and generation, performance and functionality aspired for, and the degree of human participation in detailed description of language phenomena. Basing on these criteria the projects fall into the four groups. The first group is constituted by the projects which lay emphasis on reducing the human intervention into the process of language acquisition and disambiguation. Statistical MT systems based on training data sets on parallel corpora, where rules are extracted automatically from texts, are developing methods that automatically identify the most relevant contextual features for determining the sense of any ambiguous word or word combination. The advantages of this approach consist in fully or to a