Unsupervised Efficient Learning and Representation of Language Structure

We describe a linguistic pattern acquisition algorithm that learns, in an unsupervised fashion, a streamlined representation of corpus data. This is achieved by compactly coding recursively structured constituent patterns, and by placing strings that have an identical backbone and similar context structure into the same equivalence class. The resulting representations constitute an efficient encoding of linguistic knowledge and support systematic generalization to unseen sentences. Motivation Considerations of representational parsimony dictate that the explanation for the pattern of acceptable sentences in a language be as concise as possible. A reduced representation of linguistic knowledge need not, however, take the form of a meta-language such as a prescriptive ruleset or grammar [1]. Instead, syntax may constitute an abstraction, emerging from a corpus of language [2], yet coexisting within the same representational mechanism that embodies the data. The process of abstraction can be guided by principles such as complementarity of distributions: tokens that function similarly in some sense (phonological, morphological, syntactic or semantic) but represent systematic rather than free variation will form complementary distributions or classes (e.g., [3, 4]). In thinking about emergent regularities [2], or syntactic-semantic constructions [5], we adopt Langacker’s vision: “. particular statements (specific forms) coexist with general statements (rules accounting for those forms) in a speaker’s representation of linguistic convention, which incorporates a huge inventory of specific forms learned as units (conventional expressions). Out of this sea of particularity speakers extract whatever generalizations they can. Most of these are of limited scope, and some forms cannot be assimilated to any general patterns at all. Fully general rules are not the expected case in this perspective, but rather a special, limiting case along a continuum that also embraces totally idiosyncratic forms and patterns of all intermediate degrees of generality. ” [6], p.43. Langacker’s conception of grammar as an inventory of linguistic units, which is structured in the sense that some units function as components of others, is well-suited to serve as a basis for a psychologically motivated theory of language learning, due to its clear parallels with the notion of unitization that arises in cognitive psychology [7]. Recent developments in probability and information theory and in computational learning have rendered distributional [1, 8] methods of linguistic unitization both more tractable and more readily relatable to grammarbased formalisms [9]. The present paper describes a project that aims to transform the idea of the emergence of distributional syntactic and semantic knowledge into a concrete computational model, constrained by psycholinguistic data and striving for biological plausibility. The ADIOS model The ADIOS (Automatic DIstillation Of Structure) model constructs syntactic representations of a sample of language from raw, unlabeled corpus data. The model has two components: (1) a Representational Data Structure (RDS) graph, and (2) a Pattern Acquisition (PA) algorithm that learns the RDS in an unsupervised fashion. The PA algorithm aims to detect patterns — repetitive sequences of “significant” paths (strings) of primitives occurring in the corpus. In that, it is related to prior work on alignment-based learning [10] and regular expression (“local grammar”) extraction [11] from corpora. We stress, however, that our algorithm requires no prejudging either of the scope of the primitives or of their classification, say, into syntactic categories: all the information needed for its operation is extracted from the corpus in an unsupervised fashion. In the initial phase of the algorithm, the text is segmented into the smallest possible morphological constituents (e.g., ed is split off both walked and bed; the algorithm later discovers that bed should be left whole, on statistical grounds). 1 This initial set of unique constituents is the vertex set of the newly formed RDS (multi-)graph. A directed edge is inserted between two vertices whenever the corresponding transition exists in the corpus (Figure 1(a)); the edge is labeled by the sentence number and by its within-sentence index. Thus, corpus sentences and sub-sentences initially correspond We remark that the algorithm can work in any language, with any set of tokens, including individual characters – or phonemes, if applied to speech.