SlowGuess/ACL_2020_OCR huggingface dataset:
Valentin Hofmann\*, Hinrich Schütze‡, Janet B. Pierrehumbert†*
* Faculty of Linguistics, University of Oxford
$^{\dagger}$ Department of Engineering Science, University of Oxford
$^{\ddagger}$ Center for Information and Language Processing, LMU Munich
valentin.hofmann@ling-phil.ox.ac.uk
There has been little work on modeling the morphological well-formedness (MWF) of derivatives, a problem judged to be complex and difficult in linguistics (Bauer, 2019). We present a graph auto-encoder that learns embeddings capturing information about the compatibility of affixes and stems in derivation. The auto-encoder models MWF in English surprisingly well by combining syntactic and semantic information with associative information from the mental lexicon.
A central goal of morphology is, as famously put by Aronoff (1976), "to tell us what sort of new words a speaker can form." This definition is tightly intertwined with the notion of morphological well-formedness (MWF). While non-existing morphologically well-formed words such as pro\(computer\)ism conform to the morphological patterns of a language and could be formed, non-existing morphologically ill-formed words such as pro\)and\)ism violate the patterns and are deemed impossible (Allen, 1979).
More recent research has shown that MWF is a gradient rather than binary property: non-existing words that conform to the morphological patterns of a language differ in how likely they are to be actually created by speakers (Pierrehumbert, 2012). This is particularly true in the case of derivational morphology, which is not obligatory and often serves communicative needs (Bauer, 2019). As a result, the degree of MWF of a non-existing derivative is influenced by a multitude of factors and judged to be hard to predict (Bauer, 2001).
In NLP, the lack of reliable ways to estimate the MWF of derivatives poses a bottleneck for generative models, particularly in languages exhibiting a rich derivational morphology; e.g., while inflected
(a) Mental lexicon  $(\mathcal{L})$
(b) Two-mode projection  $(\mathcal{B})$
Figure 1: Derivatives in the mental lexicon  $\mathcal{L}$  (a) and the derivational graph (DG), their derivational projection  $\mathcal{B}$  (b). Predicting whether a word is part of a derivational abstraction corresponds to predicting a single edge in the DG (dotted lines).
forms can be translated by generating morphologically corresponding forms in the target language (Minkov et al., 2007), generating derivatives is still a major challenge for machine translation systems (Sreelekha and Bhattacharyya, 2018). Similar problems exist in the area of automatic language generation (Gatt and Krahmer, 2018).
This study takes a first step towards computationally modeling the MWF of English derivatives. We present a derivational graph auto-encoder (DGA) that combines semantic and syntactic information with associative information from the mental lexicon, achieving very good results on MWF prediction and performing on par with a character-based LSTM at a fraction of the number of trainable parameters. The model produces embeddings that capture information about the compatibility of affixes and stems in derivation and can be used as pretrained input to other NLP applications.
2 Derivational Morphology
2.1 Inflection and Derivation
Linguistics divides morphology into inflection and derivation. While inflection refers to the different
<sup>1</sup>We make all our code and data publicly available at https://github.com/valentinhofmann/dga.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1127-1138 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
word forms of a lexeme, e.g., listen, listens, and listened, derivation refers to the different lexemes of a word family, e.g., listen, listener, and listenable. There are several differences between inflection and derivation, some of which are highly relevant for NLP.
Firstly, while inflection is obligatory and determined by syntactic needs, the existence of derivatives is mainly driven by communicative goals, allowing to express a varied spectrum of meanings (Acquaviva, 2016). Secondly, derivation can produce a larger number of new words than inflection since it is iterable (Haspelmath and Sims, 2010); derivational affixes can be combined, in some cases even recursively (e.g., post\(post\)modern\)ism). However, morphotactic constraints restrict the ways in which affixes can be attached to stems and other affixes (Hay and Plag, 2004); e.g., the suffix \)less$ can be combined with $ness (atom$less$ness) but not with $ity (atom$less$ity).
The semantic and formal complexity of derivation makes predicting the MWF of derivatives more challenging than the MWF of inflectional forms (Anshen and Aronoff, 1999; Bauer, 2019). Here, we model the MWF of derivatives as the likelihood of their existence in the mental lexicon.
2.2 Derivatives in the Mental Lexicon
How likely a derivative is to exist is influenced by various factors (Bauer, 2001; Pierrehumbert and Granell, 2018). In this study, we concentrate on the role of the structure of the mental lexicon.
The mental lexicon can be thought of as a set of associations between meaning  $m$  and form  $f$ , i.e., words, organized in a network, where links correspond to shared semantic and phonological properties (see Pierrehumbert (2012) for a review). Since we base our study on textual data, we will treat the form of words orthographically rather than phonologically. We will refer to the type of information conveyed by the cognitive structure of the mental lexicon as associative information.
Sets of words with similar semantic and formal properties form clusters in the mental lexicon (Alegre and Gordon, 1999). The semantic and formal properties reinforced by such clusters create abstractions that can be extended to new words (Bybee, 1995). If the abstraction hinges upon a shared derivational pattern, the effect of such an extension is a new derivative. The extent to which a word
conforms to the properties of the cluster influences how likely the abstraction (in our case a derivational pattern) is to be extended to that word. This is what is captured by the notion of MWF.
2.3 Derivational Graphs
The main goal of this paper is to predict the MWF of morphological derivatives (i.e., how likely is a word to be formed as an extension of a lexical cluster) by directly leveraging associative information. Since links in the mental lexicon reflect semantic and formal similarities of various sorts, many of which are not morphological (Tamariz, 2008), we want to create a distilled model of the mental lexicon that only contains derivational information. One way to achieve this is by means of a derivational projection of the mental lexicon, a network that we call the Derivational Graph (DG).
Let  $\mathcal{L} = (\mathcal{W},\mathcal{Q})$  be a graph of the mental lexicon consisting of a set of words  $\mathcal{W}$  and a set of links between the words  $\mathcal{Q}$ . Let  $\mathcal{W}_a\subset \mathcal{W}$  be a set of words forming a fully interconnected cluster in  $\mathcal{L}$  due to a shared derivational pattern  $a$ . We define  $\mathcal{S}_a$  as the set of stems resulting from stripping off  $a$  from the words in  $\mathcal{W}_a$  and  $\mathcal{R}_a = \{(s,a)\}_{s\in \mathcal{S}_a}$  as the corresponding set of edges between the stems and the shared derivational pattern. We then define the two-mode derivational projection  $\mathcal{B}$  of  $\mathcal{L}$  as the Derivational Graph (DG) where  $\mathcal{B} = (\mathcal{V},\mathcal{E})$ ,  $\mathcal{V} = \bigcup_{a}(S_{a}\cup \{a\})$  and  $\mathcal{E} = \bigcup_{a}\mathcal{R}_{a}$ . Figure 1 gives an example of  $\mathcal{L}$  and DG  $(= \mathcal{B})$ .
The DG is a bipartite graph whose nodes consist of stems  $s \in S$  with  $\mathcal{S} = \bigcup_{a} \mathcal{S}_{a}$  and derivational patterns  $a \in \mathcal{A}$  with  $\mathcal{A} = \bigcup_{a} \{a\}$ . The derivational patterns are sequences of affixes such as re $\Phi$ size $\Phi$ ate $\Phi$ ion in the case of revitalization. The cognitive plausibility of this setup is supported by findings that affix groups can trigger derivational generalizations in the same way as individual affixes (Stump, 2017, 2019).
We define  $\mathbf{B} \in \mathbb{R}^{|\mathcal{V}| \times |\mathcal{V}|}$  to be the adjacency matrix of  $\mathcal{B}$ . The degree of an individual node  $n$  is  $d(n)$ . We further define  $\Gamma^1(n)$  as the set of one-hop neighbors and  $\Gamma^2(n)$  as the set of two-hop neighbors of  $n$ . Notice that  $\Gamma^1(s) \subseteq \mathcal{A}$ ,  $\Gamma^1(a) \subseteq \mathcal{S}$ ,  $\Gamma^2(s) \subseteq \mathcal{S}$ , and  $\Gamma^2(a) \subseteq \mathcal{A}$  for any  $s$  and  $a$  since the DG is bipartite.
The advantage of this setup of DGs is that it abstracts away information not relevant to derivational morphology while still allowing to interpret results in the light of the mental lexicon. The cre
Figure 2: Experimental setup. We extract DGs from Reddit and train link prediction models on them. In the shown toy example, the derivatives super $\$ 8$ minecraft and affirm $\$ 9$ ation are held out for the test set.
ation of a derivative corresponds to a new link between a stem and a derivational pattern in the DG, which in turn reflects the inclusion of a new word into a lexical cluster with a shared derivational pattern in the mental lexicon.
3 Experimental Data
3.1 Corpus
We base our study on data from the social media platform Reddit.2 Reddit is divided into so-called subreddits (SRs), smaller communities centered around shared interests. SRs have been shown to exhibit community-specific linguistic properties (del Tredici and Fernandez, 2018).
We draw upon the Baumgartner Reddit Corpus, a collection of publicly available comments posted on Reddit since 2005. The preprocessing of the data is described in Appendix A.1. We examine data in the SRs r/cfb (cfb - college football), r/gaming (gam), r/leagueof legends (lol), r/movies (mov), r/nba (nba), r/nfl (nfl), r/politics (pol), r/science (sci), and r/technology (tec) between 2007 and 2018. These SRs were chosen because they are of comparable size and are among the largest SRs (see Table 1). They reflect three distinct areas of interest, i.e., sports (cfb, nba, nfl), entertainment (gam, lol, mov), and knowledge (pol, sci, tec), thus allowing for a multifaceted view on how topical factors impact MWF: seeing MWF as an emergent property of the mental lexicon entails that communities with different lexica should differ in what derivatives are most likely to be created.
3.2 Morphological Segmentation
Many morphologically complex words are not decomposed into their morphemes during cognitive processing (Sonnenstuhl and Huth, 2002). Based on experimental findings in Hay (2001), we segment a morphologically complex word only if the stem has a higher token frequency than the deriva
SR	nw	nt	|S|	|A|	|E|
cfb	475,870,562	522,675	10,934	2,261	46,110
nba	898,483,442	801,260	13,576	3,023	64,274
nfl	911,001,246	791,352	13,982	3,016	64,821
gam	1,119,096,999	1,428,149	19,306	4,519	107,126
lol	1,538,655,464	1,444,976	18,375	4,515	104,731
mov	738,365,964	860,263	15,740	3,614	77,925
pol	2,970,509,554	1,576,998	24,175	6,188	143,880
sci	277,568,720	528,223	11,267	3,323	58,290
tec	505,966,695	632,940	11,986	3,280	63,839
Table 1: SR statistics.  $n_w$ : number of tokens;  $n_t$ : number of types;  $|S|$ : number of stem nodes;  $|\mathcal{A}|$ : number of affix group nodes;  $|\mathcal{E}|$ : number of edges.
tive (in a given SR). Segmentation is performed by means of an iterative affix-stripping algorithm introduced in Hofmann et al. (2020) that is based on a representative list of productive prefixes and suffixes in English (Crystal, 1997). The algorithm is sensitive to most morpho-orthographic rules of English (Plag, 2003): when  $\$ \text{ness}$  is removed from happi $\$ \text{ness}$ , e.g., the result is happy, not happi. See Appendix A.2. for details.
The segmented texts are then used to create DGs as described in Section 2.3. All processing is done separately for each SR, i.e., we create a total of nine different DGs. Figure 2 illustrates the general experimental setup of our study.
4 Models
Let  $W$  be a Bernoulli random variable denoting the property of being morphologically well-formed. We want to model  $P(W|d, C_r) = P(W|s, a, C_r)$ , i.e., the probability that a derivative  $d$  consisting of stem  $s$  and affix group  $a$  is morphologically well-formed according to SR corpus  $C_r$ .
Given the established properties of derivational morphology (see Section 2), a good model of  $P(W|d, C_r)$  should include both semantics and formal structure,
$$
P (W | d, C _ {r}) = P (W | m _ {s}, f _ {s}, m _ {a}, f _ {a}, C _ {r}), \tag {1}
$$
where  $m_s, f_s, m_a, f_a$ , are meaning and form (here
2reddit.com
3files.pushshift.io/redit/comments
Figure 3: DGA model architecture. The DGA takes as input an adjacency matrix  $\mathbf{B}$  and additional feature vectors  $\mathbf{x}_s$  and  $\mathbf{x}_a$  and learns embeddings  $\mathbf{z}_s$  and  $\mathbf{z}_a$ .
modeled orthographically, see Section 2.2) of the involved stem and affix group, respectively. The models we examine in this study vary in which of these features are used, and how they are used.
4.1 Derivational Graph Auto-encoder
We model  $P(W|d, C_r)$  by training a graph autoencoder (Kipf and Welling, 2016, 2017) on the DG  $\mathcal{B}$  of each SR. The graph auto-encoder attempts to reconstruct the adjacency matrix  $\mathbf{B}$  (Section 2.3) of the DG by means of an encoder function  $g_{\theta}$  and a decoder function  $h_{\theta}$ , i.e., its basic structure is
$$
\tilde {\mathbf {B}} = h _ {\theta} \left(g _ {\theta} (\mathbf {B})\right), \tag {2}
$$
where  $\tilde{\mathbf{B}}$  is the reconstructed version of  $\mathbf{B}$ . The specific architecture we use (see Figure 3), which we call a Derivational Graph Auto-encoder (DGA), is a variation of the bipartite graph auto-encoder (van den Berg et al., 2018).
Encoder. The encoder  $g_{\theta}$  takes as one of its inputs the adjacency matrix  $\mathbf{B}$  of the DG  $\mathcal{B}$ . This means we model  $f_{s}$  and  $f_{a}$ , the stem and affix group forms, by means of the associative relationships they create in the mental lexicon. Since a DG has no information about semantic relationships between nodes within  $S$  and  $\mathcal{A}$ , we reintroduce meaning as additional feature vectors  $\mathbf{x}_s, \mathbf{x}_a \in \mathbb{R}^n$  for  $m_{s}$  and  $m_{a}$ , stem and affix group embeddings that are trained separately on the SR texts. The input to  $g_{\theta}$  is thus designed to provide complementary information: associative information (B) and semantic information  $(\mathbf{x}_s$  and  $\mathbf{x}_a)$ .
For the encoder to be able to combine the two types of input in a meaningful way, the choice of  $g_{\theta}$  is crucial. We model  $g_{\theta}$  as a graph convolutional network (Kipf and Welling, 2016, 2017), providing an intuitive way to combine information from the DG with additional information. The graph convolutional network consists of  $L$  convolutional layers. Each layer (except for the last one) performs two steps: message passing and activation.
During the message passing step (Dai et al., 2016; Gilmer et al., 2017), transformed versions of
the embeddings  $\mathbf{x}_s$  and  $\mathbf{x}_a$  are sent along the edges of the DG, weighted, and accumulated. We define  $\Gamma_{+}^{1}(s) = \Gamma^{1}(s)\cup \{s\}$  as the set of nodes whose transformed embeddings are weighted and accumulated for a particular stem  $s$ .  $\Gamma_{+}^{1}(s)$  is extracted from the adjacency matrix  $\mathbf{B}$  and consists of the one-hop neighbors of  $s$  and  $s$  itself. The message passing propagation rule (Kipf and Welling, 2016, 2017) can then be written as
$$
\mathbf {m} _ {s} ^ {(l)} = \sum_ {n \in \Gamma_ {+} ^ {1} (s)} \frac {\mathbf {x} _ {n} ^ {(l - 1)} \mathbf {W} ^ {(l)}}{\sqrt {\left| \Gamma_ {+} ^ {1} (s) \right| \left| \Gamma_ {+} ^ {1} (n) \right|}}, \tag {3}
$$
where  $\mathbf{W}^{(l)}$  is the trainable weight matrix of layer  $l$ ,  $\mathbf{x}_n^{(l-1)}$  is the embedding of node  $n$  from layer  $l-1$  with  $\mathbf{x}_n^{(0)} = \mathbf{x}_n$ , and  $\sqrt{|\Gamma_+^1(s)| |\Gamma_+^1(n)|}$  is the weighting factor. The message passing step is performed analogously for affix groups. The matrix form of Equation 3 is given in Appendix A.3.
Intuitively, a message passing step takes embeddings of all neighbors of a node and the embedding of the node itself, transforms them, and accumulates them by a normalized sum. Given that the DG  $\mathcal{B}$  is bipartite, this means for a stem  $s$  that the normalized sum contains  $d(s)$  affix group embeddings and one stem embedding (and analogously for affix groups). The total number of convolutional layers  $L$  determines how far the influence of a node can reach. While one convolution allows nodes to receive information from their one-hop neighbors (stems from affix groups they co-occur with and vice versa), two convolutions add information from the two-hop neighbors (stems from stems co-occurring with the same affix group and vice versa), etc. (see Figure 4).
During the activation step, the output of the convolutional layer  $l$  for a particular stem  $s$  is
$$
\mathbf {x} _ {s} ^ {(l)} = \operatorname {R e L U} \left(\mathbf {m} _ {s} ^ {(l)}\right), \tag {4}
$$
where  $\mathrm{ReLU}(\cdot) = \max (0,\cdot)$  is a rectified linear unit (Nair and Hinton, 2010). The final output of the encoder is
$$
\mathbf {z} _ {s} = \mathbf {m} _ {s} ^ {(L)}, \tag {5}
$$
i.e., there is no activation in the last layer. The activation step is again performed analogously for affix groups.  $\mathbf{z}_s$  and  $\mathbf{z}_a$  are representations of  $s$  and  $a$  enriched with information about the semantics of nodes in their DG neighborhood.
(a)  $L = 1$
(b)  $L = 2$
(c)  $L = 3$
Figure 4: Influence of  $L$ , the number of convolutional layers, on message passing. The blue nodes illustrate neighbors whose messages can be received by the orange node under varying  $L$ .
Decoder. We model the decoder as a simple bilinear function,
$$
h _ {\theta} \left(\mathbf {z} _ {s}, \mathbf {z} _ {a}\right) = \sigma \left(\mathbf {z} _ {s} ^ {\top} \mathbf {z} _ {a}\right), \tag {6}
$$
where  $\sigma$  is the sigmoid and  $\mathbf{z}_s$  and  $\mathbf{z}_a$  are the outputs of the encoder. We set  $P(W|d, C_r) = h_\theta(\mathbf{z}_s, \mathbf{z}_a)$  and interpret this as the probability that the corresponding edge in a DG constructed from a corpus drawn from the underlying distribution exists. The resulting matrix  $\tilde{\mathbf{B}}$  in Equation 2 is then the reconstructed adjacency matrix of DG.
Notice that the only trainable parameters of the DGA are the weight matrices  $\mathbf{W}^{(l)}$ . To put the performance of the DGA into perspective, we compare against four baselines, which we present in decreasing order of sophistication.
4.2 Baseline 1: Character-based Model (CM)
We model  $P(W|d, C_r)$  as  $P(W|f_s, f_a, C_r)$  using a character-based model (CM), i.e., as opposed to the DGA,  $f_s$  and  $f_a$  are modeled directly by means of their orthographic form. This provides the CM with phonological information, a central predictor of MWF (see Section 2.2). CM might also learn semantic information during training, but it is not directly provided with it. Character-based models show competitive results on derivational tasks (Cotterell et al., 2017; Vylomova et al., 2017; Deutsch et al., 2018), a good reason to test their performance on MWF prediction.
We use two one-layer bidirectional LSTMs to encode the stem and affix group into a vector  $\mathbf{o}$  by concatenating the last hidden states from both LSTM directions  $\vec{\mathbf{h}}_s, \vec{\mathbf{h}}_s, \vec{\mathbf{h}}_a$ , and  $\vec{\mathbf{h}}_a$ ,
$$
\mathbf {o} = \left[ \vec {\mathbf {h}} _ {s} \oplus \vec {\mathbf {h}} _ {s} \oplus \vec {\mathbf {h}} _ {a} \oplus \vec {\mathbf {h}} _ {a} \right], \tag {7}
$$
where  $\oplus$  denotes concatenation.  $\mathbf{o}$  is then fed into a two layer feed-forward neural network with a ReLU non-linearity after the first layer. The activation function after the second layer is  $\sigma$ .
4.3 Baseline 2: Neural Classifier (NC)
We model  $P(W|d, C_r)$  as  $P(W|m_s, m_a, C_r)$  using a neural classifier (NC) whose architecture is similar to the auto-encoder setup of the DGA.
Similarly to the DGA,  $m_{s}$  and  $m_{a}$  are modeled by means of stem and affix group embeddings trained separately on the SRs. The first encoder-like part of the NC is a two-layer feed-forward neural network with a ReLU non-linearity after the first layer. The second decoder-like part of the NC is an inner-product layer as in the DGA. Thus, the NC is identical to the DGA except that it does not use associative information from the DG via a graph convolutional network; it only has information about the stem and affix group meanings.
4.4 Baseline 3: Jaccard Similarity (JS)
We model  $P(W|d, C_r)$  as  $P(W|f_s, f_a, C_r)$ . Like in the DGA, we model the stem and affix group forms by means of the associative relationships they create in the mental lexicon. Specifically, we predict links without semantic information.
In feature-based machine learning, link prediction is performed by defining similarity measures on a graph and ranking node pairs according to these features (Liben-Nowell and Kleinberg, 2003). We apply four common measures, most of which have to be modified to accommodate the properties of bipartite DGs. Here, we only cover the best performing measure, Jaccard similarity (JS). JS is one of the simplest graph-based similarity measures, so it is a natural baseline for answering the question: how far does simple graph-based similarity get you at predicting MWF? See Appendix A.4 for the other three measures.
The JS score of an edge  $(s, a)$  is traditionally defined as
$$
\zeta_ {J S} (s, a) = \frac {\left| \Gamma^ {1} (s) \cap \Gamma^ {1} (a) \right|}{\left| \Gamma^ {1} (s) \cup \Gamma^ {1} (a) \right|}. \tag {8}
$$
However, since  $\Gamma^1 (s)\cap \Gamma^1 (a) = \emptyset$  for any  $s$  and  $a$  (the DG is bipartite), we redefine the set of common neighbors of two nodes  $n$  and  $m$ ,  $\Gamma_{\cap}(n,m)$ , as  $\Gamma^2 (n)\cap \Gamma^1 (m)$ , i.e., the intersection of the two-hop neighbors of  $n$  and the one-hop neighbors of
Besides the simple dot-product decoder, we also implemented a bilinear decoder with  $h(\mathbf{z}_s,\mathbf{z}_a) = \sigma (\mathbf{z}_s^\top \mathbf{Q}\mathbf{z}_a)$ , where  $\mathbf{Q}$  is a trainable weight matrix. However, the model performed significantly worse.
We also experimented with only one layer, but it performed considerably worse.
$m$ , and analogously  $\Gamma_{\cup}(n,m)$  as  $\Gamma^2 (n)\cup \Gamma^1 (m)$ . Since these are asymmetric definitions, we define
$$
\zeta_ {J S} (s, a) = \frac {\left| \Gamma_ {\cap} (s , a) \right|}{\left| \Gamma_ {\cup} (s , a) \right|} + \frac {\left| \Gamma_ {\cap} (a , s) \right|}{\left| \Gamma_ {\cup} (a , s) \right|} \tag {9}
$$
JS assumes that a stem that is already similar to a lexical cluster in its derivational patterns is more likely to become even more similar to the cluster than a less similar stem.
4.5 Baseline 4: Bigram Model (BM)
We again model  $P(W|d, C_r)$  as  $P(W|f_s, f_a, C_r)$ , leaving aside semantic information. However, in contrast to JS, this model implements the classic approach of Fabb (1988), according to which pairwise constraints on affix combinations, or combinations of a stem and an affix, determine the allowable sequences. Taking into account more recent results on morphological gradience, we do not model these selection restrictions with binary rules. Instead, we use transition probabilities, beginning with the POS of the stem  $s$  and working outwards to each following suffix  $a^{(s)}$  or preceding prefix  $a^{(p)}$ . Using a simple bigram model (BM), we can thus calculate the MWF of a derivative as
$$
P (W | d, C _ {r}) = P \left(a ^ {(s)} | s\right) \cdot P \left(a ^ {(p)} | s\right), \tag {10}
$$