The Processing & Recognition of Symbol Sequences

The Processing & Recognition of Symbol Sequences Mark W. Andrews (mwa1@cornell.edu) Department of Psychology; Uris Hall Ithaca, NY 14853 USA Abstract It is proposed that learning a language (or more gener- ally, a sequence of symbols) is formally equivalent to reconstructing the state-space of a non-linear dynamical system. Given this, a set of results from the study of nonlinear dynamical systems may be especially relevant for an understanding of the mechanisms underlying lan- guage processing. These results demonstrate that a dy- namical system can be reconstructed on the basis of the data that it emits. They imply that with minimal assump- tions the structure of an arbitrary language can be inferred entirely from a corpus of data. State-Space reconstruc- tion can implemented in a straightforward manner in a model neural system. Simulations of a recurrent neural network, trained on a large corpus of natural language, are described. Results imply that the network sucessfully recognizes temporal patterns in this corpus. Introduction Complex pattern recognition is often characterized by means of a simple geometric analogy. Any object or pattern may be described as a single point in a high- dimensional space. For example, a square grayscale im- age that is 256 pixels in length, may be described as a point in the 256 2 dimensional space of all possible im- ages. A collection of such images is a set of points in this space. If these patterns are not entirely random, this set will reside in a subspace of lower dimensionality. To learn the structure of these images, an organism or ma- chine must discover a compact parametric representation of this subspace. This might take the form of, for exam- ple, finding a reasonably small set of basis vectors that will span the subspace and projecting each image onto these vectors. Having done this, each image can be clas- sified in terms of a new and more meaningful coordinate system. You effectively describe ’what is there’ in terms of ’what is known’. This geometric approach is routinely employed in the study of visual object recognition, but may easily be extended to a wide range of categorization and classi- fication tasks. In almost all cases, however, the pat- terns under study have been multi-dimensional static pat- terns. In contrast, the study of temporal pattern recogn- tion using this or related approaches has not been well- developed. For example, one of the most widely em- ployed techniques for temporal pattern recognition, Hid- den Markov Models are limited in their generality due to their fundamental inability to handle patterns above a certain complexity. This absence of general models for temporal pattern recognition is evident in the study of human language processing, which traditionally has es- chewed serious consideration of statistical learning and pattern recognition. This paper aims to introduce a general framework for the study of temporal pattern recognition. This is devel- oped in the context to language processing, but it could be extended in a straightforward manner to most other cases of temporal patterns. First, a general characteriza- tion of the problem of language learning and language processingis proposed. Then, some recent results in the study of nonlinear dynamical systems are described. These are seen as being especially relevant for an under- standing of the mechanisms underlying temporal pattern recognition, especially with regard to language process- ing. Finally, simulations with a recurrent neural network are described, which suggest successful pattern recogni- tion of English sentences. The processing of symbol sequences A paradigm for the study temporal pattern processing, especially language processing, has developed as a re- sult of the deep relationship between formal languages and abstract automata (Chomsky 1963) 1 . Any language (or more generally, any sequence of symbols), can be de- scribed as the product of a particular automaton. By this account, learning a language is equivalent to identifying a particular automaton on the basis of a sample of the language that it generates. More formally, an automa- ton A is specified by the quadruple X Y F G . X and Y are sets known as the state and the output spaces, re- spectively. The functions F : X X and G : X Y 1 The correspondence between formal languages and ab- stract automata can be summarized by the so-called Chomsky hierarchy: Classes of automata that are increasingly restrictive versions of the Turing machine produce classes of languages described by increasingly restrictive generative grammars. The regular languages R are produced by strictly finite automata, the context-free languages CF are produced by pushdown stack automata, the context-sensitive languages CS are produced by linear bounded automata and the recursively enumerable lan- guages RN are produced by unrestricted Turing machines. R CF CS RN, and likewise for their corresponding automata.