Extracting an expectation-based lexicon for UD treebanks
Authors: Matteo Gay, Cristiano Chesi (IUSS, 2023)
Topic: Language Models

Abstract
In this paper, we present a refinement of the deterministic lexicon extraction algorithm discussed in [1]. This new version is able to infer categorial expectations from any University Dependency (UD) treebank [2] and recast non-trivial backward dependencies in terms of movement operations [3,4]. The improvements with respect to a preliminary version of the algorithm are related to a more granular definition of the functional categories and a significant reduction in lexical ambiguity. These results might constitute a substantial baseline for (very) large language models assessment in terms of descriptive adequacy [5].

Introduction
The advent of large language models (LLMs), especially those based on pre-trained transformers (GPT-like, [6]) demonstrated the impressive capability of next-token prediction task in training. This prediction is largely based on expectations of various kinds (morphosyntactic, semantic, and inferential) that, in virtue of a well-tuned attention mask mechanism can generalize categorial constraints across word sequences, apparently, without significant distance or directionality limitations among tokens, though the number of tokens accepted in input at each iteration remains fixed. This represents a clear difference with respect to recurrent networks training (e.g., LSTM, [7]) which benefits from a more cognitively plausible working memory mechanism that tries to preserve a sound incremental processing also modeling “backward dependencies” (i.e., when β depends on α, but β linearly precedes α). It has been demonstrated however that these new generation attention-based transformers are computationally very intensive [8] and their performance on various linguistic datasets demonstrates the implausibility of their linguistic generalizations [9]. In a recent study, Wilcox, Futrell & Levy [10] demonstrate the impressive ability of certain recurrent networks (JRNN and GRNN) to model islands constraints in a way that is comparable (or even better, sometimes) to that of larger LMs such as GTP-3. These results, apparently, not only challenge the poverty of stimulus hypothesis [5], but also suggest that, as far as linguistic competence is concerned, models with a smaller number of parameters (and a different architecture) might be more efficient in learning a human-like linguistic competence. What is then the minimum model size we might afford to maintain this high level of meta-linguistic competence? In this paper, to address the “minimum model size problem,” we stress the relevance of a cognitively plausible expectation mechanism (model architecture) that benefits from explicit modeling of (at least) certain categorial constraints (minimum number of parameters needed). Expectation-based Minimalist Grammars (e-MGs, [11]), for instance, simply rely on the notion of satisfied local selection (through categorial features) to identify a successful local dependency or the requirement to be satisfied through a non-local one. The advantages are (i) the complete transparency of this model compared to a cognitively plausible grammatical theory [12] and (ii) the computational complexity cost in recognition which grows polynomially to the length of the sentence (unless specific parameterization to extend empirical coverage is adopted, [1]). A possible disadvantage of this approach is related to the fact that a fully explicit lexicon is required to run the recognition or the generation algorithm. Based on [13], this work extends the proposed algorithm to retrieve a large-scale lexicon from annotated datasets such as UD treebanks. In Section 2, we will introduce the critical aspects of this procedure, essentially focusing on the directionality of each dependency and on the inherent lexical ambiguity obtained by applying a naïf lexicon extraction procedure. In Section 3, we propose a refinement of the extraction approach, and we will calculate the efficiency of this new method in terms of reduction of lexical and syntactic ambiguity and, overall, lexicon size. Section 4 will conclude this paper by suggesting further improvements to be evaluated in the future.

Conclusion
These preliminary results are encouraging and demonstrate, on the one hand, that structural ambiguity can be reasonably reduced by assuming properly crafted linguistic generalizations, while, on the other hand, valency-based theories [29] are less accurate and induce an unwanted level of ambiguity that can be readily avoided. More work needs to be done at least in the following direction: lexical items are considered as atomic entities. As demonstrated by the efficacy of word embeddings based on sub-word units (notably the byte-pair encoding, [30,31]), this step is more than necessary (and doable in the current e-MG framework, see [32]). Returning to the original consideration of how these results can serve as a baseline for LLM assessment, there is at least one possible approach we consider: if explicit categorial expectations are implicitly represented in LLMs, then the number of features needed in the lexicon should be considered as the minimal dimensionality required for word embedding. As far as structural constraints are concerned, this should represent the lowest number of parameters (i.e., levels of abstraction/representation) for the lexical item. Although no lexical semantic considerations are addressed here (even though the categorial approach is, in principle, also tenable from the semantic compositional perspective), specific generalizations should be obtainable both with e-MGs and with LLMs of comparable size in terms of parameters (i.e., roughly corresponding to the number of categories expected or required by the e-MGs lexicon). The phenomena to be tested should include island violations (e.g., “*what did John read the book that was talking about?”), thematic selection constraints (e.g., “*John put,” “*John eats a sandwich to Mary”), etc., along the lines, for instance, of [33] or [9]. If more parameters (where “more” corresponds to at least one order of magnitude) are needed to perform similarly to e-MGs on these constraints, we might conclude that LLMs still need some “optimization” since they do not yet qualify as efficient and descriptively adequate.