SlowGuess/ACL_2020_OCR huggingface dataset:
$$
\forall w \in \pi (L), \forall w ^ {\prime} \in \pi^ {\prime} (L),
$$
$$
p _ {\pi \cdot \pi^ {\prime}} (w, w ^ {\prime}) = p _ {\pi} (w) p _ {\pi^ {\prime}} (w ^ {\prime})
$$
A syntactic partition will be defined by its independence from contextual information:
Definition 10. Let  $P$  be a probabilistic partition, and  $L$  a stochastic language. The partition  $P$  is said to be syntactic if it is independent (with respect to  $L$ ) of any possible probabilistic contextual partition in  $L$ .
Example. The partition  $P_{1}$  is contextual, as  $\pi_{1}(L) = \overline{\pi_{1}(L)}$ . The partition  $P_{2}$  is clearly independent of  $P_{1}$  w.r.t.  $L$ .
4.1 Information-theoretic formulation
The definitions above may need to be relaxed if we want to infer syntax from natural language corpora, where strict independence cannot be expected. We propose to reformulate the definitions of contextual and syntactic information in the information theory framework.
We present a relaxation of our definition based on Shannon's information theory (Shannon, 1948). We seek to quantify the amount of information in a partition  $P = (V, C, \pi)$  with respect to a language  $L$ . Shannon's entropy provides an appropriate measure. Applied to  $\pi(L)$ , it gives
$$
H (\pi (L)) = - \sum_ {w \in \pi (L)} p _ {\pi} (w) (\log (p _ {\pi} (w)))
$$
For a simpler illustration, from now on we will consider only languages composed of fixed-length sequences  $s$ , i.e.  $|s| = n$  for a given  $n$ . If  $L$  is such a language, we will consider the language  $\overline{\overline{L}}$  as the language of sequences of size  $n$  defined by
$$
p _ {\overline {{L}}} (v _ {i _ {1}} \dots v _ {i _ {n}}) = \prod_ {j} p _ {L} (v _ {i _ {j}})
$$
where  $p_L(v)$  is the frequency of  $v$  in language  $L$ .
Proposition 1. Let  $L$  be a stochastic language,  $P = (V, C, \pi)$  a partition. One has:
$$
H (\pi (\overline {{\overline {{L}}})) \geq H (\pi (\overline {{L}})) \geq H (\pi (L))
$$
with equality iff the stochastic languages are equal.
Let  $C$  be a set of categories. For a given distribution over the categories  $p(c_{i})$ , the partition defined by  $\pi (c_i|v) = p(c_i)$  (constant distribution w.r.t. the vocabulary) contains no information on the language. One has  $p_{\pi}(c_{i_1}\ldots c_{i_k}) = p(c_{i_1})\ldots p(c_{i_k})$ , which is the unigram distribution, in other words  $\pi (L) = \pi (\overline{\overline{L}})$ . As the amount of syntactic or contextual information contained in  $\overline{\overline{L}}$  can be considered as zero, a consistent definition of the information would be:
Definition 11. Let  $P = (V, C, \pi)$  be a partition, and  $L$  a language. The information contained in  $P$  with respect to  $L$  is defined as
$$
I _ {L} (P) = H (\pi (\overline {{\overline {{L}}}})) - H (\pi (L))
$$
Lemma 1. Information  $I_L(P)$  defined as above is always positive. One has  $I_{\overline{L}}(P) \leq I_L(P)$ , with equality iff  $\pi(\overline{L}) = \pi(L)$ .
After having defined how to measure the amount of information in a partition with respect to a language, we now translate the independence between two partitions into the terms of mutual information:
Definition 12. We follow notations from Definition 9. We define the mutual information of two partitions  $P = (V, C, \pi)$  et  $P' = (V, C', \pi')$  with respect to  $L$  as
$$
I _ {L} (P; P ^ {\prime}) = H (P) + H (P ^ {\prime}) - H (P \cdot P ^ {\prime})
$$
This directly implies that
Lemma 2.  $P = (V,C,\pi)$  and  $P^{\prime} = (V,C^{\prime},\pi^{\prime})$  are independent w.r.t.  $L$
$$
\Leftrightarrow I _ {L} (P; P ^ {\prime}) = 0
$$
Proof. This comes from the fact that, by construction, the marginal distributions of  $\pi \cdot \pi^{\prime}$  are the distributions  $\pi$  and  $\pi^\prime$ .
With these two definitions, we can now propose an information-theoretic reformulation of what constitutes a contextual and a syntactic partition:
Proposition 2. Let  $L$  be a stochastic language over vocabulary  $V$ , and let  $P = (V, C, \pi)$  be a probabilistic partition.
-  $P$  is contextual iff
$$
I _ {L} (P) = I _ {\overline {{L}}} (P)
$$
-  $P$  is syntactic iff for any contextual partition  $P_{*}$
$$
I _ {L} (P; P _ {*}) = 0
$$
4.2 Relaxed formulation
If we deal with non artificial samples of natural language data, we need to prepare for sampling issues and word (form) ambiguities that make the above formulation of independence too strict. Consider for instance adding the following sentence to the previous sample:
doctors heal fear
The distinction between syntactic and contextual categories is not as clear as before. We need a relaxed formulation for real corpora: we introduce  $\gamma$ -contextual and  $\mu, \gamma$ -syntactic partitions.
Definition 13. Let  $L$  be a stochastic language.
- A partition  $P$  is considered as  $\gamma$ -contextual if it minimizes
$$
I _ {L} (P) (1 - \gamma) - I _ {\bar {L}} (P) \tag {1}
$$
- A partition  $P$  is considered  $\mu, \gamma$ -syntactic if it minimizes
$$
\max  _ {P ^ {*}} I _ {L} (P; P _ {*}) - \mu I _ {L} (P) \tag {2}
$$
for any  $\gamma$ -contextual partition  $P^{*}$ .
Let  $P$  and  $P^{\prime}$  be two partitions for  $L$ , such that
$$
\Delta_ {I} (L) = I _ {P ^ {\prime}} (L) - I _ {P} (L) \geq 0
$$
then the  $\gamma$ -contextual program (1) would choose  $P'$  over  $P$  iff
$$
\frac {\Delta_ {I} (L) - \Delta_ {I} (\overline {{L}})}{\Delta_ {I} (L)} \leq \gamma
$$
Let  $P^*$  be a  $\gamma$ -contextual partition. Let
$$
\Delta_ {M I} (L, P ^ {*}) = I _ {L} \left(P ^ {\prime}; P ^ {*}\right) - I _ {L} (P; P ^ {*})
$$
then the  $\mu, \gamma$ -syntactic program (2) would choose  $P'$  over  $P$  iff
$$
\frac {\Delta_ {M I} (L , P ^ {*})}{\Delta_ {I} (L)} \leq \mu
$$
Example. Let us consider the following partitions: -  $P_{1}$  and  $P_{2}$  refer to the previous partitions above:  $\{Animals, Science, Medicine\}$  and  $\{Noun, Verb\}$  -  $P_{A}$  is adapted from  $P_{1}$  so that 'fear' belongs to Animals and Medicine
$$
\{c, e, r, f (\frac {1}{2}) \}, \{m, p, t \}, \{d, h, w, f (\frac {1}{2}) \}
$$
-  $P_B$  merges Animals and Medicine from  $P_1$
$$
\{c, e, r, f, d, h, w \}, \{m, p, t \}
$$
-  $P_{sent}$  describes the probability for a word to belong to a given sentence (5 categories)
-  $P_{C}$  is adapted from  $P_{2}$  so that 'fear' belongs to Verb and Noun
$$
\{c, r, m, t, d, w, f (\frac {1}{2}) \}, \{e, p, h, f (\frac {1}{2}) \}
$$
Figure 1:  $I_L(P) - I_{\overline{L}}(P)$  represented w.r.t.  $I_L(P)$  for different partitions: acceptable solutions of program (1) lie on the convex hull boundary of the set of all partitions. Solution for  $\gamma$  is given by the tangent of slope  $\gamma$ . Non trivial solutions are  $P_B$  and  $P_1$ .
-  $P_{D}$  is adapted from  $P_{2}$  and creates a special category for 'fear'
$$
\{c, r, m, t, d, w \}, \{e, p, h \}, \{f \}
$$
-  $P_{\text{posi}}$  describes the probability for a word to appear in a given position (3 categories)
Figure 2:  $I_L(P;P_B)$  represented w.r.t.  $I_L(P)$  for different partitions: acceptable solutions of program (2) lies on the convex hull boundary of the set of all partitions. Solution for  $\mu$  is given by the tangent of slope  $\mu$ . Non-trivial solution is  $P_2$ .
Acceptable solutions of (1) and (2) are, respectively, on the convex hull boundary in Fig.1 and Fig.2. While the lowest parameter (non trivial) solutions are  $P_B$  for context and  $P_2$  for syntax, one can check that partitions  $P_1$ ,  $P_A$  and  $P_{sent}$  are all close to the boundary in Fig.1, and that partitions  $P_C$ ,  $P_D$  and  $P_{posi}$  are all close to the boundary in Fig.2, as expected considering their information content.
In this section we illustrate the emergence of syntactic information via the application of objectives (1) and (2) to a natural language corpus. We show that the information we acquire indeed translates into known syntactic and contextual categories.
For this experiment we created a corpus from the Simple English Wikipedia dataset (Kauchak, 2013), selected along three main topics: Numbers, Democracy, and Hurricane, with about 430 sentences for each topic and a vocabulary of 2963 unique words. The stochastic language is the set  $L^3$  of 3-gram frequencies from the dataset. In order to avoid biases with respect to the final punctuation, we considered overlapping 3-grams over sentences. For the sake of evaluation, we construct one contextual and one syntactic embedding for each word. These are the probabilistic partitions over gold standard contextual and syntactic categories. The contextual embedding  $P_{con}$  is defined
by relative frequency in the three topics. The results for this partition are  $I_{L^3}(P_{con}) = 0.06111$  and  $I_{\overline{L^3}}(P_{con}) = 0.06108$ , corresponding to a  $\gamma$  threshold of  $6.22 \cdot 10^{-4}$  in (1), and thus distribution over topics can be considered as an almost purely contextual partition. The syntactic partition  $P_{syn}$  is the distribution over POS categories (tagged with the Stanford tagger, Toutanova et al. (2003)).
Using the gold categories, we can manipulate the information in the partitions by merging and splitting across contextual or syntactic categories. We study how the information calculated by (1) and (2) evolve; we validate our claims if we can deduce the nature of information from these statistics.
Figure 3: Increase of information  $\Delta_I$  in three scenarios: syntactic split, topic split and random split.
We start from the syntactic embeddings and we split and merge over the following POS categories: Nouns (NN), Adjectives (JJ), Verbs (V), Adverbs(ADV) and Wh-words (WH). For a pair of categories (say  $NN + V$ ), we create:
-  $P_{\text{merge}}$  merges the two categories  $(NN + V)$
-  $P_{\text{syntax}}$  splits the merged category into  $NN$  and  $V$  (syntactic split)
-  $P_{\text {topic }}$  splits the merged category into  $(NN + V)_{t_1}, (NN + V)_{t_2}$  and  $(NN + V)_{t_3}$  along the three topics (topic split)
-  $P_{random}$  which splits the merged category into  $(NN + V)_1$  and  $(NN + V)_2$  randomly (random split)
It is clear that each split will increase the information compared to  $P_{merge}$ . We display the simple information gains  $\Delta_I$  in Fig.3. The question is whether we can identify if the added information is syntactic or contextual in nature, i.e. if we can find a  $\mu$  for which the  $\mu$ ,  $\gamma$ -syntactic program (2)
selects every syntactic splitting and rejects every contextual or random one.
Figure 4: Ratio  $\Delta_{MI} / \Delta_I$  in three scenarios: syntactic split, topic split and random split. Considering objective (2) with parameter  $\mu = 0.5$  leads to discrimination between contextual and syntactic information.
Fig.4 represents the ratio between the increase of mutual information (relatively to  $P_{con}$ )  $\Delta_{MI}$  and the increase of information  $\Delta_I$ , corresponding to the threshold  $\mu$  in (2). It shows that indeed for a  $\mu = 0.5$  syntactic information (meaningful refinement according to POS) will be systematically selected, while random or topic splittings will not. We conclude that even for a small natural language sample, syntactic categories can be identified based on statistical considerations, where a language model learning algorithm would need further information or hypotheses.
4.4 Integration with Models
We have shown that our framework allows to search for syntactic categories without prior hypothesis of a particular model. Yet if we do have a hypothesis, we can indeed search for the syntactic categories that fit the particular class of models  $\mathcal{M}$ . In order to find the categories which correspond to the syntax rules that can be formulated in a given class of models, we can integrate the model class in the training objective by replacing entropy by the negative log-likelihood of the training sample.
Let  $M \in \mathcal{M}$  be a model, which takes a probabilistic partition  $P = (V, C, \pi)$  as input, and let  $LL(M, P, L_S)$  be the log-likelihood obtained for sample  $S$ . We will denote
$$
\begin{array}{l} \tilde {H} (L _ {S}, P) = - \sup  _ {M \in \mathcal {M}} L L (M, P, L _ {S}) \\ \tilde {I} _ {L _ {S}} (P) = \tilde {H} (\overline {{\overline {{L _ {S}}}}}, P) - \tilde {H} (L _ {S}, P) \\ \end{array}
$$
Following Definition 12, we define
$$
\begin{array}{l} \tilde {I} _ {L _ {S}} (P; P ^ {\prime}) = \\ \tilde {H} (L _ {S}, P) + \tilde {H} (L _ {S}, P ^ {\prime}) - \tilde {H} (L _ {S}, P \cdot P ^ {\prime}) \\ \end{array}
$$
We may consider the following program:
- A partition  $P$  is said to be  $\gamma$ -contextual if it minimizes
$$
\tilde {I} _ {L _ {S}} (P) (1 - \gamma) - \tilde {I} _ {\overline {{L _ {S}}}} (P)
$$
- Let  $P_*$  be a  $\gamma$ -contextual partition for  $L$ ,  $\mu \in \mathbb{R}^+$ ,  $k \in \mathbb{N}$ . The partition  $P$  is considered  $\mu, \gamma$ -syntactic if it minimizes
$$
\max _ {P ^ {*}} \tilde {I} _ {L _ {S}} (P; P ^ {*}) - \mu \tilde {I} _ {L _ {S}} (P)
$$
In this paper, we proposed a theoretical reformulation for the problem of learning syntactic information from a corpus. Current language models have difficulty acquiring syntactically relevant generalizations for diverse reasons. On the one hand, we observe a natural tendency to lean towards shallow contextual generalizations, likely due to the maximum likelihood training objective. On the other hand, a corpus is not representative of human linguistic competence but of performance. It is however possible for linguistic competence - syntax - to emerge from data if we prompt models to establish a distinction between syntactic and contextual (semantic/pragmatic) information.
Two orientations can be identified for future work. The immediate one is experimentation. The current formulation of our syntax learning scheme needs adjustments in order to be applicable to real natural language corpora. At present, we are working on an incremental construction of the space of categories.
The second direction is towards extending the approach to morphologically rich languages. In that case, two types of surface realization need to be considered: word order and morphological markers. An agglutinating morphology probably allows a more straightforward application of the method, by treating affixes as individual elements of the vocabulary. The adaptation to other types of morphological markers will necessitate more elaborate linguistic reflection.
Kuncoro Adhiguna, Dyer Chris, Hale John, Yogatama Dani, Clark Stephen, and Blunsom Phil. 2018. LSTMs can learn syntax-sensitive dependencies well, but modeling structure makes them better. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.
Marco Baroni. 2019. Linguistic generalization and compositionality in modern artificial neural networks. CoRR, abs/1904.00157.
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. What do neural machine translation models learn about morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics.
Iris Berent and Gary Marcus. 2019. No integration without structured representations: Response to Pa-ter. Language, 95:1:e75-e86.
Jean-Philippe Bernardy and Shalom Lappin. 2017. Using deep neural networks on learn syntactic agreement. Linguistic Issues in Language Technology, 15(2):1-15.
Terra Blevins, Omer Levy, and Luke Zettlemoyer. 2018. Deep RNNs encode soft hierarchical syntax. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.
Samuel R. Bowman, Christopher D. Manning, and Christopher Potts. 2015. Tree-structured composition in neural networks without tree-structured architectures. In NIPS Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches.
Noam Chomsky. 1957. Syntactic Structures. Mouton, Berlin, Germany.
Noam Chomsky. 1980. Rules and representations. *Behavioral and Brain Sciences*, 3(1):1-15.
Shammur Absar Chowdhury and Roberto Zamparelli. 2018. RNN simulations of grammaticality judgments on long-distance dependencies. In Proceedings of the 27th International Conference on Computational Linguistics.
Alexander Clark and Rémi Eyraud. 2006. Learning auxiliary fronting with grammatical inference. In Conference on Computational Language Learning.
Alexander Clark and Shalom Lappin. 2010. Unsupervised learning and grammar induction. In Handbook of Computational Linguistics and Natural Language Processing. Wiley-Blackwell, Oxford.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. 2018. What you can cram into a single  $\$ \& !\#$ * vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.
Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Gottlob Frege. 1892. Über Sinn und Bedeutung. Zeitschrift für Philosophie und philosophische Kritik, 100:25-50.
Mario Giulianielli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. 2018. Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information. In EMNLP Workshop Blackbox NLP: Analyzing and Interpreting Neural Networks for NLP.
E. Mark Gold. 1967. Language identification in the limit. Information and control, 10:5:447-474.
Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing.
John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the North American Chapter of the Association for Computational Linguistics.
Dieuwke Hupkes, Sara Veldhoen, and Willem H. Zuidema. 2018. Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907-926.
David Kauchak. 2013. Improving text simplification language modeling using unsimplified text data. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics.
Brenden M. Lake and Marco Baroni. 2017. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In 34th International Conference on Machine Learning.
Shalom Lappin and Stuart Shieber. 2007. Machine learning theory and practice as a source of insight into universal grammar. Journal of Linguistics, 43:393-427.
Beth Levin and Malka Rappaport Hovav. 2005. Argument Realization. Cambridge University Press, Cambridge.
Omer Levy, Steffen Remus, Chris Biemann, and Ido Dagan. 2015. Do supervised distributional methods really learn lexical inference relations? In Proceedings of the North American Chapter of the Association for Computational Linguistics - Human Language Technologies.
Xiang Lisa Li and Jason Eisner. 2019. Specializing word embeddings (for parsing) by information bottleneck. In 2019 Conference on Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing.
Tal Linzen. 2019. What can linguistics and deep learning contribute to each other? Response to Pater. Language, 95(1):e98-e108.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4.
Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Richard McCoy, Robert Frank, and Tal Linzen. 2018. Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks. *ArXiv*, abs/1802.09091.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Tomas Mikolov, Martin Karafiát, Lukás Burget, Jan ernocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In INTER-SPEECH.
Frederick J. Newmeyer. 2003. Grammar is grammar and usage is usage. Language, 79:4:682-707.
Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Joe Pater. 2019. Generative linguistics and neural networks at 60: Foundation, friction, and fusion. Language, 95:1:41-74.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Matthew E. Peters, Mark Neumann, Luke Zettlemoyer, and Wentau Yih. 2018b. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Shauli Ravfogel, Yoav Goldberg, and Tal Linzen. 2019. Studying the inductive biases of RNNs with synthetic variations of natural languages. CoRR, abs/1903.06400.
Shauli Ravfogel, Yoav Goldberg, and Francis Tyers. 2018. Can LSTM learn to capture agreement? the case of basque. In EMNLP Workshop Blackbox NLP: Analyzing and Interpreting Neural Networks for NLP.
Naomi Saphra and Adam Lopez. 2018. Language models learn POS first. In EMNLP Workshop Blackbox NLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019. Analysing mathematical reasoning abilities of neural models. In Proceedings of the 7th International Conference on Learning Representations.
Marten van Schijndel, Aaron Mueller, and Tal Linzen. 2019. Quantity doesn't buy quality syntax with neural language models. In Proceedings of Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing.
Claude E. Shannon. 1948. A mathematical theory of communication. Bell System Technical Journal, 27:379-423 and 623-656.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? Probing for sentence structure in contextualized word representations. In International Conference on Learning Representations.
Naftali Tishby, Fernando Pereira, and William Bialek. 1999. The information bottleneck method. In Annual Allerton Conference on Communication, Control and Computing, pages 368-377.
Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the North American Chapter of the Association for Computational Linguistics.