SlowGuess/ACL_2020_OCR huggingface dataset:
Raphaël Bailly
SAMM, EA 4543, FP2M 2036 CNRS
Université Paris 1 Pantheon-Sorbonne
raphael.bailly@univ-paris1.fr
Kata Gabor
ERTIM, EA 2520
INALCO
kata.gabor@inalco.fr
This paper is a theoretical contribution to the debate on the learnability of syntax from a corpus without explicit syntax-specific guidance. Our approach originates in the observable structure of a corpus, which we use to define and isolate grammaticality (syntactic information) and meaning/pragmatics information. We describe the formal characteristics of an autonomous syntax and show that it becomes possible to search for syntax-based lexical categories with a simple optimization process, without any prior hypothesis on the form of the model.
Syntax is the essence of human linguistic capacity that makes it possible to produce and understand a potentially infinite number of unheard sentences. The principle of compositionality (Frege, 1892) states that the meaning of a complex expression is fully determined by the meanings of its constituents and its structure; hence, our understanding of sentences we have never heard before comes from the ability to construct the sense of a sentence out of its parts. The number of constituents and assigned meanings is necessarily finite. Syntax is responsible for creatively combining them, and it is commonly assumed that syntax operates by means of algebraic compositional rules (Chomsky, 1957) and a finite number of syntactic categories.
One would also expect a computational model of language to have - or be able to acquire - this compositional capacity. The recent success of neural network based language models on several NLP tasks, together with their "black box" nature, attracted attention to at least two questions. First, when recurrent neural language models generalize to unseen data, does it imply that they acquire syntactic knowledge, and if so, does it translate into human-like compositional capacities (Baroni,
2019; Lake and Baroni, 2017; Linzen et al., 2016; Gulordava et al., 2018)? Second, whether research into neural networks and linguistics can benefit each other (Pater, 2019; Berent and Marcus, 2019); by providing evidence that syntax can be learnt in an unsupervised fashion (Blevins et al., 2018), or the opposite, humans and machines alike need innate constraints on the hypothesis space (a universal grammar) (Adhiguna et al., 2018; van Schijndel et al., 2019)?
A closely related question is whether it is possible to learn a language's syntax exclusively from a corpus. The poverty of stimulus argument (Chomsky, 1980) suggests that humans cannot acquire their target language from only positive evidence unless some of their linguistic knowledge is innate. The machine learning equivalent of this categorical "no" is a formulation known as Gold's theorem (Gold, 1967), which suggests that the complete unsupervised learning of a language (correct grammaticality judgments for every sequence), is intractable from only positive data. Clark and Lappin (2010) argue that Gold's paradigm does not resemble a child's learning situation and there exist algorithms that can learn unconstrained classes of infinite languages (Clark and Eyraud, 2006). This ongoing debate on syntax learnability and the poverty of the stimulus can benefit from empirical and theoretical machine learning contributions (Lappin and Shieber, 2007; McCoy et al., 2018; Linzen, 2019).
In this paper, we argue that syntax can be inferred from a sample of natural language with very minimal supervision. We introduce an information theoretical definition of what constitutes syntactic information. The linguistic basis of our approach is the autonomy of syntax, which we redefine in terms of (statistical) independence. We demonstrate that it is possible to establish a syntax-based lexical classification of words from a corpus without a prior hypothesis on the form of a syntactic
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 477-487 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Our work is loosely related to previous attempts at optimizing language models for syntactic performance (Dyer et al., 2016; Adhiguna et al., 2018) and more particularly to Li and Eisner (2019) because of their use of mutual information and the information bottleneck principle (Tishby et al., 1999). However, our goal is different in that we demonstrate that very minimal supervision is sufficient in order to guide a symbolic or statistical learner towards grammatical competence.
2 Language models and syntax
As recurrent neural network based language models started to achieve good performance on different tasks (Mikolov et al., 2010), this success sparked attention on whether such models implicitly learn syntactic information. Language models are typically evaluated using perplexity on test data that is similar to the training examples. However, lower perplexity does not necessarily imply better syntactic generalization. Therefore, new tests have been put forward to evaluate the linguistically meaningful knowledge acquired by LMs.
A number of tests based on artificial data have been used to detect compositionality or systematicity in deep neural networks. Lake and Baroni (2017) created a task set that requires executing commands expressed in a compositional language. Bowman et al. (2015) design a task of logical entailment relations to be solved by discovering a recursive compositional structure. Saxton et al. (2019) propose a semi-artificial probing task of mathematics problems.
Linzen et al. (2016) initiated a different line of linguistically motivated evaluation of RNNs. Their data set consists in minimal pairs that differ in grammaticality and instantiate sentences with long distance dependencies (e.g. number agreement). The model is supposed to give a higher probability to the grammatical sentence. The test aims to detect whether the model can solve the task even when this requires knowledge of a hierarchical structure. Subsequently, several alternative tasks were created along the same concept to overcome specific shortcomings (Bernardy and Lappin, 2017; Gulordava et al., 2018), or to extend the scope to different languages or phenomena (Ravfogel et al., 2018, 2019).
It was also suggested that the information content of a network can be tested using "probing
tasks” or “diagnostic classifiers” (Giulianelli et al., 2018; Hupkes et al., 2018). This approach consists in extracting a representation from a NN and using it as input for a supervised classifier to solve a different linguistic task. Accordingly, probes were conceived to test if the model learned parts of speech (Saphra and Lopez, 2018), morphology (Belinkov et al., 2017; Peters et al., 2018a), or syntactic information. Tenney et al. (2019) evaluate contextualized word representations on syntactic and semantic sequence labeling tasks. Syntactic knowledge can be tested by extracting constituency trees from a network's hidden states (Peters et al., 2018b) or from its word representations (Hewitt and Manning, 2019). Other syntactic probe sets include the work of Conneau et al. (2018) and Marvin and Linzen (2018).
Despite the vivid interest for the topic, no consensus seems to unfold from the experimental results. Two competing opinions emerge:
- Deep neural language models generalize by learning human-like syntax: given sufficient amount of training data, RNN models approximate human compositional skills and implicitly encode hierarchical structure at some level of the network. This conjecture coincides with the findings of, among others Bowman et al. (2015); Linzen et al. (2016); Giulianelli et al. (2018); Gulordava et al. (2018); Adhiguna et al. (2018).
- The language model training objective does not allow to learn compositional syntax from a corpus alone, no matter what amount of training data the model was exposed to. Syntax learning can only be achieved with task-specific guidance, either as explicit supervision, or by restricting the hypothesis space to hierarchically structured models (Dyer et al., 2016; Marvin and Linzen, 2018; Chowdhury and Zamparelli, 2018; van Schijndel et al., 2019; Lake and Baroni, 2017).
Moreover, some shortcomings of the above probing methods make it more difficult to come to a conclusion. Namely, it is not trivial to come up with minimal pairs of naturally occurring sentences that are equally likely. Furthermore, assigning a (slightly) higher probability to one sentence does not reflect the nature of knowledge behind a grammaticality judgment. Diagnostic classifiers may do well on a linguistic task because they learn to
solve it, not because their input contains a hierarchical structure (Hewitt and Liang, 2019). In what follows, we present our assessment on how the difficulty of creating a linguistic probing data set is interconnected with the theoretical problem of learning a model of syntactic competence.
2.1 Competence or performance, or why syntax drowns in the corpus
If syntax is an autonomous module of linguistic capacity, the rules and principles that govern it are formulated independently of meaning. However, a corpus is a product of language use or performance. Syntax constitutes only a subset of the rules that generate such a product; the others include communicative needs and pragmatics. Just as meaning is uncorrelated with grammaticality, corpus frequency is only remotely correlated with human grammaticality judgment (Newmeyer, 2003).
Language models learn a probability distribution over sequences of words. The training objective is not designed to distinguish grammatical from agrammatical, but to predict language use. While Linzen et al. (2016) found a correlation between the perplexity of RNN language models and their syntactic knowledge, subsequent studies (Bernardy and Lappin, 2017; Gulordava et al., 2018) recognized that this result could have been achieved by encoding lexical semantic information, such as argument typicality. E.g. "in dogs (...) bark", an RNN might get the right agreement by encoding information about what typically barks" (Gulordava et al., 2018).
Several papers revealed the tendency of deep neural networks to fixate on surface cues and heuristics instead of "deep" generalization in solving NLP tasks (Levy et al., 2015; Niven and Kao, 2019). In particular, McCoy et al. (2019) identify three types of syntactic heuristics that get in the way of meaningful generalization in language models.
Finally, it is difficult to build a natural language data set without semantic cues. Results from the syntax-semantics interface research show that lexical semantic properties account for part of syntactic realization (Levin and Rappaport Hovav, 2005).
3 What is syntax a generalization of?
We have seen in section 2 that previous works on the linguistic capacity of neural language models concentrate on compositionality, the key to creative use of language. However, this creativity is not
present in language models: they are bound by the type of the data they are exposed to in learning.
We suggest that it is still possible to learn syntactic generalization from a corpus, but not with likelihood maximization. We propose to isolate the syntactic information from shallow performance-related information. In order to identify such information without explicitly injecting it as direct supervision or model-dependent linguistic presuppositions, we propose to examine inherent structural properties of corpora. As an illustration, consider the following natural language sample:
cats eat rats
rats fear cats
mathematicians prove theorems
doctors heal wounds
According to the Chomskyan principle of the autonomy of syntax (Chomsky, 1957), the syntactic rules that define well-formedness can be formulated without reference to meaning and pragmatics. For instance, the sentence Colorless green ideas sleep furiously is grammatical for humans, despite being meaningless and unlikely to occur. We study whether it is possible to deduce, from the structural properties of our sample above, human-like grammaticality judgments that predict sequences like cats rats fear as agrammatical, and accept e.g. wounds eat theorems as grammatical.
We distinguish two levels of observable structure in a corpus:
1. the proximity; the tendency of words to occur in the context of each other (in the same document/same sentence, etc.)
2. the order in which the words appear.
Definition 1. Let  $L$  be a language over vocabulary  $V$ . The language that contains every possible sequence obtained by shuffling the elements in a sequence of  $L$  will be denoted  $\overline{L}$ .
If  $V^{*}$  is the set of every possible sequence over vocabulary  $V$  and  $L$  is the language instantiated by our corpus,  $L$  is generated by a mixture of contextual and syntactic constraints over  $V^{*}$ . We are looking to separate the syntactic specificities from the grammatically irrelevant, contextual cues. The processes that transform  $V^{*}$  into  $\overline{L}$ , and  $\overline{L}$  into  $L$
$$
V ^ {*} \xrightarrow {\text {p r o x i m i t y}} \overline {{L}} \xrightarrow {\text {o r d e r}} L
$$
are entirely dependent on words: it should be possible to encode the information used by these processes into word categories.
In what follows, we will provide tools to isolate the information involved in proximity from the information involved in order. We also relate these categories to linguistically relevant concepts.
3.1 Isolating syntactic information
For a given word, we want to identify the information involved in each type of structure of the corpus, and represent it as partitions of the vocabulary into lexical categories:
1. Contextual information is any information unrelated to sentence structure, and hence, grammaticality: this encompasses meaning, topic, pragmatics, corpus artefacts etc. The surface realization of sentence structure is a language-specific combination of word order and morphological markers.
2. Syntactic information is the information related to sentence structure and - as for the autonomy requirement - nothing else: it is independent of all contextual information.
In the rest of the paper we will concentrate on English as an example, a language in which syntactic information is primarily encoded in order. In section 5 we present our ideas on how to deal with morphologically richer languages.
Definition 2. Let  $L$  be a language over vocabulary  $V = \{v_{1},\ldots \}$ , and  $P = (V,C,\pi :V\mapsto C)$  a partition of  $V$  into categories  $C$ . Let  $\pi (L)$  denote the language that is created by replacing a sequence of elements in  $V$  by the sequence of their categories.
One defines the partition  $P_{tot} = \{\{v\}, v \in V\}$  (one category per word) and the partition  $P_{nul} = \{V\}$  (every word in the same category).
$P_{tot}$  is such that  $\pi_{tot}(L)\sim L$ . The minimal partition  $P_{nul}$  does not contain any information.
A partition  $P = (V, C, \pi)$  is contextual if it is impossible to determine word order in language  $L$  from sequences of its categories:
Definition 3. Let  $L$  be a language over vocabulary  $V$ , and let  $P = (V, C, \pi)$  be a partition over  $V$ . The partition  $P$  is said to be contextual if
$$
\pi (L) = \pi (\overline {{L}})
$$
The trivial partition  $P_{nul}$  is always contextual.
Example. Consider the natural language sample. We refer to the words by their initial letters:  $r(ats), e(at) \ldots$ , thus we have  $V = \{c, e, r, f, m, p, t, d, h, w\}$ . and  $L = \{cer, rfc, mpt, dhw\}$ .
One can check that the partition  $P_{1}$  ..
$$
\begin{array}{l} c _ {1} = \{c, r, e, f \} \\ c _ {2} = \{m, p, t \} \\ c _ {3} = \{d, h, w \} \\ \end{array}
$$
is contextual: the well-formed sequences over this partition are  $c_{1}c_{1}c_{1}$ ,  $c_{2}c_{2}c_{2}$  and  $c_{3}c_{3}c_{3}$ . These patterns convey the information that words like 'mathematicians' and 'theorems' occur together, but do not provide information on order. Therefore  $\pi_1(L) = \{c_1c_1c_1,c_2c_2c_2,c_3c_3c_3\} = \pi_1(\overline{L})$ .  $P_{1}$  is also a maximal partition for that property: any further splitting leads to order-specific patterns. Intuitively, this partition corresponds to the semantic categories Animals = {r, c, e, f}, Science = {m, p, t}, and Medicine = {d, h, w}.
A syntactic partition has two characteristics: its patterns encode the structure (in our case, order), and it is completely autonomous with respect to contextual information. Let us now express this autonomy formally.
Two partitions of the same vocabulary are said to be independent if they do not share any information with respect to language  $L$ . In other words, if we translate a sequence of symbols from  $L$  into their categories from one partition, this sequence of categories will not provide any information on how the sequence translates into categories from the other partition:
Definition 4. Let  $L$  be a language over vocabulary  $V$ , and let  $P = (V, C, \pi)$  and  $P' = (V, C', \pi')$  be two partitions of  $V$ .  $P$  and  $P'$  are considered as independent with respect to  $L$  if
$$
\begin{array}{l} \forall c _ {i _ {1}} \dots c _ {i _ {n}} \in \pi (L), \forall c _ {j _ {1}} ^ {\prime} \dots c _ {j _ {n}} ^ {\prime} \in \pi^ {\prime} (L) \\ \pi^ {- 1} (c _ {i _ {1}} \dots c _ {i _ {n}}) \cap \pi^ {\prime - 1} (c _ {j _ {1}} ^ {\prime} \dots c _ {j _ {n}} ^ {\prime}) \neq \emptyset \\ \end{array}
$$
Definition 5. Let  $L$  be a language over  $V$ , and let  $P = (V, C, \pi)$  be a partition.  $P$  is said to be syntactic if it is independent of any contextual partition of  $V$ .
A syntactic partition is hence a partition that does not share any information with contextual partitions; or, in linguistic terms, a syntactic pattern is equally applicable to any contextual category.
Example. We can see that the partition  $P_{2}$ :
$$
\begin{array}{l} c _ {4} = \{c, r, m, t, d, w \} \\ c _ {5} = \{e, f, p, h \} \\ \end{array}
$$
is independent of the partition  $P_{1}$ : one has  $\pi_2(L) = \{c_4c_5c_4\}$ . Knowing the sequence  $c_{4}c_{5}c_{4}$  does not provide any information on which  $P_{1}$  categories the words belong to.  $P_{2}$  is therefore a syntactic partition.
Looking at the corpus, one might be tempted to consider a partition  $P_{3}$  that sub-divides  $c_{4}$  into subject nouns, object nouns, and - if one word can be mapped to only one category - "ambiguous" nouns:
$$
c _ {6} = \{m, d \}
$$
$$
c _ {7} = \{t, w \}
$$
$$
c _ {8} = \{c, r \}
$$
$$
c _ {9} = \{e, f, p, h \}
$$
The patterns corresponding to this partition would be  $\pi_3(L) = \{c_6c_9c_7,c_8c_9c_8\}$ . These patterns will not predict that sentence (2) is grammatical, because the word wounds was only seen as an object. If we want to learn the correct generalization we need to reject this partition in favour of  $P_{2}$ .
This is indeed what happens by virtue of definition 5. We notice that the patterns over  $P_{3}$  categories are not independent of the contextual partition  $P_{1}$ : one can deduce from the rule  $c_{8}c_{9}c_{8}$  that the corresponding sentence cannot be e.g. category  $c_{2}$ :
$$
\pi_ {3} ^ {- 1} (c _ {8} c _ {9} c _ {8}) \cap \pi_ {1} ^ {- 1} (c _ {2} c _ {2} c _ {2}) = \emptyset
$$
$P_{3}$  is hence rejected as a syntactic partition.
$P_{2}$  is the maximal syntactic partition: any further distinction that does not conflate  $P_{1}$  categories would lead to an inclusion of contextual information. We can indeed see that category  $c_{4}$  corresponds to Noun and  $c_{5}$  corresponds to Verb. The syntactic rule for the sample is Noun Verb Noun. It becomes possible to distinguish between syntactic and contextual acceptability: cats rats fear is acceptable as a contextual pattern  $c_{1}c_{1}c_{1}$  under 'Animals', but not a valid syntactic pattern. The sequence wounds eat theorems is syntactically well-formed by  $c_{5}c_{6}c_{5}$ , but does not correspond to a valid contextual pattern.
In this section we provided the formal definitions of syntactic information and the broader contextual information. By an illustrative example we gave an intuition of how we apply the autonomy of syntax principle in a non probabilistic grammar. We now turn to the probabilistic scenario and the inference from a corpus.
4 Syntactic and contextual categories in a corpus
As we have seen in section 2, probabilistic language modeling with a likelihood maximization objective does not have incentive to concentrate on syntactic generalizations. In what follows, we demonstrate that using the autonomy of syntax principle it is possible to infer syntactic categories for a probabilistic language.
A stochastic language  $L$  is a language which assigns a probability to each sequence. As an illustration of such a language, we consider the empirical distribution induced from the sample in section 3.
$$
L = \{c e r (\frac {1}{4}), r f c (\frac {1}{4}), m p t (\frac {1}{4}), d h w (\frac {1}{4}) \}
$$
We will denote by  $p_L(v_{i_1}\dots v_{i_n})$  the probability distribution associated to  $L$ .
Definition 6. Let  $V$  be a vocabulary. A (probabilistic) partition of  $V$  is defined by  $P = (V, C, \pi : V \mapsto \mathbb{P}(C))$  where  $\mathbb{P}(C)$  is the set of probability distributions over  $C$ .
Example. The following probabilistic partitions correspond to the non-probabilistic partitions (contextual and syntactic, respectively) defined in section 3. We will now consider these partitions in the context of the probabilistic language  $L$ .
$$
\pi_ {1} = \begin{array}{c} c \\ r \\ e \\ f \\ m \\ p \\ t \\ d \\ h \\ w \end{array} \left( \begin{array}{c c c} 1 & 0 & 0 \\ 1 & 0 & 0 \\ 1 & 0 & 0 \\ 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 1 & 0 \\ 0 & 1 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ 0 & 0 & 1 \end{array} \right), \pi_ {2} = \begin{array}{c} c \\ r \\ e \\ f \\ m \\ p \\ t \\ d \\ h \\ w \end{array} \left( \begin{array}{c} 1 & 0 \\ 1 & 0 \\ 0 & 1 \\ 0 & 1 \\ 1 & 0 \\ 0 & 1 \\ 1 & 0 \\ 1 & 0 \\ 0 & 1 \\ 1 & 0 \end{array} \right)
$$
From a probabilistic partition  $P = (V, C, \pi)$  as defined above, one can map a stochastic language  $L$  to a stochastic language  $\pi(L)$  over the sequences of categories:
$$
p _ {\pi} \left(c _ {i _ {1}} \dots c _ {i _ {n}}\right) =
$$
$$
\sum_ {u _ {j _ {1}} \ldots u _ {j _ {n}}} (\prod_ {k} \pi (c _ {i _ {k}} | u _ {j _ {k}})) p _ {L} (u _ {j _ {1}} \ldots u _ {j _ {n}})
$$
As in the non-probabilistic case, the language  $\overline{L}$  will be defined as the language obtained by shuffling the sequences in  $L$ .
Definition 7. Let  $L$  be a stochastic language over vocabulary  $V$ . We will denote by  $\overline{L}$  the language obtained by shuffling the elements in the sequences
of  $L$  in the following way: for a sequence  $v_{1}\ldots v_{n}$  one has
$$
p _ {\overline {{L}}} \left(v _ {1} \dots v _ {n}\right) = \frac {1}{n !} \sum_ {\left(i _ {1} \dots i _ {n}\right) \in \sigma (n)} p _ {L} \left(v _ {i _ {1}} \dots v _ {i _ {n}}\right)
$$
One can easily check that  $\pi (\overline{L}) = \overline{\pi(L)}$
Example. The stochastic patterns of  $L$  over the two partitions are, respectively:
$$
\begin{array}{l} \pi_ {1} (L) = \left\{c _ {1} c _ {1} c _ {1} (\frac {1}{2}), c _ {2} c _ {2} c _ {2} (\frac {1}{4}), c _ {3} c _ {3} c _ {3} (\frac {1}{4}) \right\} \\ \pi_ {2} (L) = \{c _ {4} c _ {5} c _ {4} (1) \} \\ \end{array}
$$
We can now define a probabilistic contextual partition:
Definition 8. Let  $L$  be a stochastic language over vocabulary  $V$ , and let  $P = (V, C, \pi)$  be a probabilistic partition.  $P$  will be considered as contextual if
We now want to express the independence of syntactic partitions from contextual partitions. The independence of two probabilistic partitions can be construed as an independence between two random variables:
Definition 9. Consider two probabilistic partitions  $P = (V, C, \pi)$  and  $P' = (V, C', \pi')$ . We will use the notation
$$
\left(\pi \cdot \pi^ {\prime}\right) _ {v} \left(c _ {i}, c _ {j} ^ {\prime}\right) = \pi_ {v} \left(c _ {i}\right) \pi_ {v} ^ {\prime} \left(c _ {j} ^ {\prime}\right)
$$
and the notation
$$
P \cdot P ^ {\prime} = (V, C \times C ^ {\prime}, \pi \cdot \pi^ {\prime})
$$
$P$  and  $P^{\prime}$  are said to be independent (with respect to  $L$ ) if the distributions inferred over sequences of their categories are independent: