Categorial Grammar Induction from Raw Data
Year: 2023

Abstract:
Grammar induction, the task of learning a set of grammatical rules from raw or minimally labeled text data, can provide clues about what kinds of syntactic structures are learnable without prior knowledge. Recent work (e.g., Kim et al., 2019; Zhu et al., 2020; Jin et al., 2021a) has achieved advances in unsupervised induction of probabilistic context-free grammars (PCFGs). However, categorial grammar induction has received less recent attention, despite allowing inducers to support a larger set of syntactic categories—due to restrictions on how categories can combine—and providing a transparent interface with compositional semantics, opening up possibilities for models that jointly learn form and meaning. Motivated by this, we propose a new model for inducing a basic (Ajdukiewicz, 1935; Bar-Hillel, 1953) categorial grammar. In contrast to earlier categorial grammar induction systems (e.g., Bisk and Hockenmaier, 2012), our model learns from raw data without any part-of-speech information. Experiments on child-directed speech show that our model attains a recall-homogeneity of 0.33 on average, which dramatically increases to 0.59 when a bias toward forward function application is added to the model.

heading: 1 Introduction
text: One of the core motivating questions of modern linguistics relates to language acquisition: How can a child pick up complex linguistic rules from limited exposure to language? Chomsky (e.g., 1965) introduced the well-known argument from the poverty of the stimulus, which claims that the linguistic input received by children is insufficiently rich to account for the knowledge they acquire—and therefore that humans must be born with prior knowledge about language. In contrast, empiricist accounts of language acquisition argue that statistical cues (Saffran et al., 1996) or other factors such as social interaction (Tomasello, 2005) may pro-
vide enough information on their own to support language acquisition.
Computational modeling provides one useful tool for judging between these competing accounts. Questions about the learnability of linguistic structures can be tested empirically by seeing if a model with minimal prior knowledge can learn these structures from corpus data (Pullum and Scholz, 2002).
Along these lines, a range of studies over several decades have tested whether induction models can acquire probabilistic context-free grammars (PCFGs) from text data (Lari and Young, 1990; Klein and Manning, 2002). Although PCFG induction is considered a difficult problem (Carroll and Charniak, 1992), recent systems have achieved performance improvements thanks to new types of Bayesian and neural network models. Recent systems have been able to induce grammars with accuracy levels (measured by recall-homogeneity) approaching fifty percent on corpora of child-directed speech (Jin et al., 2018, 2021a,b).
Although PCFGs are a convenient formalism for computational modeling, they are not the only viable option. A second line of research—albeit one less currently active than PCFG modeling—has examined the learnability of categorial grammar formalisms (Bisk and Hockenmaier, 2012; Bisk et al., 2015), particularly Combinatory Categorial Grammar (CCG; Steedman, 2000). A notable advantage of categorial grammars over PCFGs is their clean mapping between syntactic and semantic composition, which allows them to be used as a tool for predicting lambda calculus encodings of meaning (Zettlemoyer and Collins, 2005). Categorial grammars also impose constraints regarding which syntactic categories can combine, providing a practical advantage for designing induction systems that support a large set of categories.
Motivated by these advantages, this work presents a neural network–based system that adapts a state-of-the-art PCFG induction model (Jin et al.,
2368
2021a) to instead learn a basic categorial grammar.1 Unlike the previously mentioned categorial grammar induction systems, our model learns from entirely unlabeled data.
An initial version of the model attains an average recall-homogeneity (RH) score of 0.33 on an English corpus of child-directed speech (Experiment 1). A high variance across randomly initialized runs is observed, with a cluster of runs achieving RH on par with state-of-the-art PCFG inducers and another cluster achieving poor RH. In Experiment 2, we test a modified version of the model with a bias term encouraging forward function application, which appeared more often in the better-performing runs in Experiment 1. The modified model reaches an average RH of 0.59, surpassing results reported from Jin et al. (2021a) and other PCFG inducers.

heading: 2 Related Work
text: PCFG induction is a longstanding area of interest in computational linguistics (Lari and Young, 1990; Carroll and Charniak, 1992; Klein and Manning, 2002). As neural modeling has made unsupervised induction more feasible, recent work has experimented with learning compound PCFGs (Kim et al., 2019), simultaneously inducing phrase structure grammars and lexical dependencies (Zhu et al., 2020), and boosting model performance by grounding on multimodal data (Zhao and Titov, 2020; Zhang et al., 2021, 2022), among other innovations.
A somewhat earlier line of research established the potential for learning an alternative type of grammar, a CCG, from data with a small set of broadly defined part-of-speech categories (noun, verb, etc.) (Bisk and Hockenmaier, 2012, 2013; Bisk et al., 2015). Bisk et al. (2015) showed that only a small number of labeled data points with POS tags are needed to induce a CCG. However, induction of CCGs (or other categorial grammars) has received less recent attention, with CCG research more focused on tasks such as supertagging (Bhargava and Penn, 2020; Prange et al., 2021) or incremental parsing (Stanojević et al., 2021).
A third relevant area of research is work focused on mapping a sentence to its logical form via CCG parsing (Zettlemoyer and Collins, 2005;
1The code for the categorial grammar inducer is available at https://github.com/christian-clark/ cgInduction.
Kwiatkowski et al., 2010; Kwiatkowski et al., 2013). These studies reveal that categorial grammar induction may be useful not only as a method of testing the learnability of syntactic rules, but also as a tool for semantic parsing.

heading: 3 Background

heading: 3.1 Basic Categorial Grammar
text: The induction models in this paper learn a basic categorial grammar, also known as an AB grammar (Ajdukiewicz, 1935; Bar-Hillel, 1953). This type of grammar was chosen for its simplicity and its suitability for extension through additional composition operations. A basic categorial grammar uses a set of primitive categories (e.g., S or N for sentences or nouns) as well as the type-combining operators \ and /, which indicate compatibility with an argument preceding or following the category, respectively. These type-combining operators can be used to define complex categories (e.g., N\N or (S/N)/N).
The models use two composition operations: backward function application and forward function application. Backward function application occurs when a phrase of category X\Y combines with a phrase of category Y on the left to yield a larger phrase of category X. Forward function application occurs when a phrase of category X/Y combines with a phrase of category Y on the right to yield a larger phrase of category X. In such cases, X\Y and X/Y are called the functor categories, Y is called the argument category, and X is called the result category. See Figure 1 for an example parse using a basic categorial grammar.

heading: 3.2 The Jin et al. (2021a) PCFG induction model
text: Our induction model uses a formulation for sentence probabilities based on the word-level PCFG model from Jin et al. (2021a), which we summarize in this section.
In unsupervised training, the objective function that the model maximizes is the marginal probability of the sentences in the dataset. For a single sentence σ, each possible parse tree (assumed to be in Chomsky Normal Form) can be divided into a set of of nodes τ undergoing nonterminal expansions cη → cη1 cη2 and a set of nodes τ′ undergoing terminal expansions cη → wη. Here, η ∈ {1, 2}∗ is a Gorn address specifying a path of left and right branches from the root node of the parse tree, cη is the nonterminal category at node η, and wη is the word located at node η. C is the set of nonterminal categories. The marginal probability of σ is calculated by summing over all possible parse trees:
P(σ) = ∑
τ,τ′
∏ η∈τ P(cη → cη1 cη2) · ∏ η∈τ′ P(cη → wη)
(1) A set of Bernoulli distributions are defined to separate the nonterminal and terminal expansion rules:
P(Term | cη) = softmax{0,1} (NTerm(E δcη)) (2)
Here, cη is a nonterminal category, δcη is a vector representing a Kronecker delta function with 1 at index cη and 0 elsewhere, and E ∈ Rd×|C| is a matrix of nonterminal category embeddings of size d. NTerm is a residual network with 2 identical blocks. Given the input xb−1,cη , each residual block computes its output as follows:
xb,cη = ReLU(W ′ b ReLU(Wb xb−1,cη + bb) + b′b) + xb−1,cη (3)
Fully connected layers are used before and after the residual blocks:
x0,cη = ReLU(W0 E δcη + b0), (4) scη = ReLU(Wsoft xB,cη + bsoft) (5)
All W’s and b’s are weight and bias parameters respectively.
Binary-branching nonterminal expansion probabilities are computed as follows:
P(cη → cη1 cη2) = P(Term=0 | cη) · P(cη → cη1 cη2 | cη,Term=0), (6)
which in turn uses the following distribution over expansion rules:
P(cη → cη1 cη2 | cη,Term=0) = softmax
cη1,cη2 (Wnont E δcη + bnont), (7)
where Wnont and bnont are additional model parameters. The categorial grammar induction model presented in this work modifies Equation (7); see Section 4.2.
Finally, lexical unary-expansion rule probabilities are computed as follows:
P(cη → wη) = P(Term=1 | cη) · P(cη → wη | cη,Term=1) (8)
A softmax is taken over words in the vocabulary:
P(cη → wη | cη,Term=1) = softmax wη (N′(E δcη)),
(9) where N′ is another residual network, similar to NTerm except that the output layer’s dimension is the size of the vocabulary.
Jin et al. (2021a) also introduce a character-level expansion model as an alternative to Equation (9). However, they report that the word-level model performs slightly better on English data from the CHILDES corpus. Because the current study works with the same English data, we only test the wordlevel model.

heading: 4 Induction Model
text: The model introduced in this paper extends the Jin et al. (2021a) model from Section 3.2 to induce a categorial grammar. This section details how categories and expansion rule probabilities are defined in the new model.

heading: 4.1 Categories
text: We define the set of categories C according to a number of primitives P and a maximum category depth D. Primitives in the induction model are labeled as integers 0, 1, 2, . . . . A category’s depth is defined according to its tree representation (see Figure 3(a) of Prange et al. (2021) for an example).2 For instance, the primitive category 1 has depth 0, and the category 2/(1\0) has depth 2.
The number of possible categories |CP,D| with P primitives and maximum depth D can be computed
2The tree representation of a syntactic category should not be confused with the parse tree for an entire sentence.
with the following recurrence relation:
|CP,0| = P (10) |CP,i| = 2|CP,i−1|2 + P (11)
Our experiments below use the category set C = C3,2, the entire set of 885 possible categories with P = 3 and D = 2. This is nearly 10 times the number of categories (90) used by Jin et al. (2021a).

heading: 4.2 Binary expansion rule probabilities
text: The categorial grammar induction model modifies Equation (7) from Jin et al. (2021a) to take advantage of constraints imposed by categorial grammar categories. In a PCFG, a parent category cη may expand into any two child categories cη1 and cη2 However, in a basic CG, this expansion is only possible if one child (the functor) can combine with the other child (the argument) to produce the parent (the result). There are two possibilities:
• The argument is the left child cη1. Then the right child must be the functor, and its category must be cη2 = cη\cη1. • The argument is the right child cη2. Then the
left child must be the functor, and its category must be cη1 = cη/cη2.
If cη2 , cη\cη1 and cη1 , cη/cη2, then it is impossible for cη to expand to cη1 and cη2, and so P(cη → cη1 cη2 | cη,Term=0) = 0. For all other cases, where the binary expansion is possible, the probabilities are calculated as follows:
P(cη → cη1 cη2 | cη,Term=0) =
softmax (c′,o)∈Carg×{L,R} ( [WL WR ] δcη + [ bL bR ] ) (12)
The model parameters WL,WR ∈ R|Carg |×|Cres | are weights associating each parent category with each possible left-child and right-child argument category; bL,bR ∈ R|Carg | are the corresponding bias vectors. Carg,Cres ⊂ C are the sets of possible argument and result categories respectively, both of which comprise all categories of depth up to D − 1:
Carg = Cres = {c ∈ C | depth(c) ≤ D − 1} (13) Argument and result categories cannot have depth D because this would require functor categories to have depth greater than D.
The variable o ∈ {L,R} expresses the location of the argument child relative to the functor child.
If o = L, then the argument is to the left of the functor, and so cη1 = c′ and cη2 = cη\c′. If o = R, then the argument is to the right of the functor, and so cη1 = cη/c′ and cη2 = c′.
Equation (12) results in a considerable space complexity improvement compared to (7). For an induction model using category set C, Equation (7) requires taking a softmax over |C|2 possible pairs of children. Equation (12) only requires a softmax over 2|Carg| = O(
√|C|) categories. The experiment presented in Section 6 uses a modified bias term in Equation (12) in order to encourage the model to prefer forward function application over backward function application. The bias term for left-child arguments bL is replaced with a new bias b′L = bL − k, where k ∈ R|Carg | has the same constant value in each dimension. As will be explained below, a bias toward forward function application results in a preference for rightbranching structures, which improves the performance of the induction model.

heading: 5 Experiment 1: Basic Induction Model

heading: 5.1 Corpora
text: The induction model was evaluated on childdirected speech in English from CHILDES (MacWhinney, 2000), specifically the Adam and Eve sections of the Brown corpus (Brown, 1973). The Adam section, which was used for hyperparameter optimization, contains interactions between a child and his caretakers, with the child’s age ranging from 2 years and 3 months to 5 years and 2 months and a total of 28,780 sentences. The Eve section was used for held-out testing; it contains similar interactions from a child whose age ranges from 1 year and 6 months to 2 years and 3 months, with a total of 14,251 sentences. Syntactic annotations for the Adam and Eve sections came from Pearl and Sprouse (2013).

heading: 5.2 Procedures
text: The induction model used the Adam optimizer with a learning rate of 0.0001, a category embedding size of 64, and a hidden layer size of 64. Hyperparameters were selected based on a grid search on the Adam corpus. For evaluation on Eve, ten randomly initialized models were run for 20 epochs each with a batch size of 2 sentences.
The evaluation metrics we considered were unlabeled F1 score and recall-homogeneity (RH; Jin et al., 2021b). Recall measures what proportion
of (unlabeled) constituents in the annotated trees are present in the predicted trees. Homogeneity—a commonly used metric in part-of-speech tagging evaluations—measures to what degree a single induced category maps to a single category in the annotations. Specifically, it measures the relative increase in the log of the expected probability of a gold category, given the predicted category that covers the same span. RH is simply the product of unlabeled recall and homogeneity. The RH metric is motivated by assumptions that (a) induced grammars should not be penalized for predicting extra constituents, since flatter trees in the annotations may have been chosen for convenience rather than any theoretical motivation; and (b) induced grammars should not be penalized for making finergrained distinctions between categories (e.g., noun cases) than are present in the annotations, since less granular categories similarly may have been chosen for convenience.
In keeping with Seginer (2007) and Jin et al. (2021a), punctuation was retained in the input data during training but removed during evaluation. Unary chains were removed from parse trees, with only the top category used for evaluation.

heading: 5.3 Results
text: Figure 2 presents the main results. The mean RH and F1 score across the ten runs were 0.33 and 0.52 respectively. The mean RH value is well below the average of 0.49 reported for the word-level PCFG inducer from Jin et al. (2021a). However, the means alone do not best describe Figure 2, as RH and F1 both seem to show bimodal distributions. Six runs produced poor RH and F1 (averaging 0.22 and 0.37 respectively), while the other four runs produced much better values (averaging 0.50 and 0.74 respectively).3
Figure 3 offers another vantage point into the pattern of results by separating recall and homogeneity, the two metrics combined in RH. (Recall also influences F1.) Again, a sharp division is observed between runs with low versus high recall. However, homogeneity appears to vary somewhat independently from recall.
One possible explanation for this trend would be that the poorly performing runs get caught in local maxima of the objective function. If this were the case, we would expect to see higher log likelihood
3We verified that these were the same six and four runs, i.e., no run produced good RH and poor F1 or vice versa.
in the well-performing runs, which should reach a better maximum. However, Figure 4 shows that this does not occur: The well-performing and poorly performing runs are associated with similar ranges of log likelihood values.
If log likelihood fails to distinguish the clusters of good and poor runs, what else might? One pattern immediately stood out during qualitative inspection of the models’ predicted trees: Models with high RH and F1 scores tend to predict trees with frequent forward function application and right branching, while poorly performing models predict trees with backward function application and left branching. This pattern was quantitatively confirmed by counting the proportion of right- and left-branching nodes in the induced trees. The six runs with worse performance use left branching 82% of the time, while the four runs with better performance use right branching 65% of the time. These values were computed by counting the proportion of branching nodes who appear as left versus right children.
As an illustration, Figure 5 compares predictions from two models on the same sentence from the Eve corpus. Figure 5a shows the prediction from a model that performed poorly overall, containing a left-branching pattern combined with the use of backward function application. Figure 5b shows the prediction from a model that performed well, which has opposite patterns. In both cases, the dispreferred type-combining operator (e.g., / in 5a) appears in complex categories but is rarely applied, so that a category such as 1/2 in 5a is treated like a primitive.
Figure 6 contains confusion matrices relating the induced categories with the human-annotated
categories for the run that produced the highest RH and F1 score in Experiment 1. The recall table (a) shows that most of the annotated categories are only represented by one or two different induced categories, and the precision table (b) shows that induced categories are seldom crossing brackets.

heading: 6 Experiment 2: Induction Model with Forward Function Application Bias
text: To try to produce more consistent induction results with English-like branching behavior, our second experiment biased the induction model toward using forward function applications (i.e., the / operator). While in principle it is possible for rightbranching structures to use backward function ap-
plication and left-branching structures to use forward function application, this did not often occur in the results from Experiment 1 and seems less likely in general given the available categories in C.

heading: 6.1 Procedures
text: This experiment used the same corpora and procedures as Experiment 1, with the exception of the modified bias term mentioned at the end of Sec-
tion 4.2. Each dimension in the vector k was set to 100; this value was large enough to ensure that forward function application and right-branching tree structures were exclusively used. Smaller values for k were also tested on the Adam corpus but achieved slightly worse log likelihoods.

heading: 6.2 Results
text: Figure 7 shows the RH across the 10 runs in this experiment. Because the models invariably predicted right-branching structures, all had the same F1 score of 0.76. Compared to Experiment 1, RH scores showed much more consistency, with a relatively uniform spread of values within the narrow range of 0.57 to 0.62. (Since recall did not vary, all variation in RH was due to differences in homogeneity between models.) Despite the models’ inflexibility in assigning tree structures, these RH scores surpassed those reported by Jin et al. (2021a,b).

heading: 7 Discussion and Conclusion
text: We introduce an induction model that learns a basic categorial grammar from unlabeled data. The original version of the model, tested in Experiment 1, shows promising results in several runs, but inconsistent performance in general. A modified version of the model that consistently uses forward func-
tion application far outperforms the original model. In general, the experimental results appear to support the empiricist claim that syntactic structure is learnable with relatively simple prior knowledge.
While results from the biased model achieve an impressive RH compared to Jin et al. (2021a,b), they leave open several questions. One obvious question is whether it is possible to consistently achieve comparable results to the biased runs without removing the model’s ability to do backward function application, since this operation is a core ingredient of basic categorial grammars and is regularly used in hand-labeled parses of English sentences, e.g., to combine the NP and N\NP in Figure 1. Although the log likelihood objective on its own appears to be insufficient to ensure stable behavior (similar to behavior reported in earlier PCFG studies such as Johnson et al. 2007), it may be possible to find a middle ground between Experiments 1 and 2 with a modified objective function or a weaker form of bias.
Another question is whether the induction model can support more complex operations, such as the forward and backward composition operations defined by CCG. This seems possible in principle; additional weight matrices could be added to Equation (12), so that probabilities for additional operations could be learned. We are excited to explore this possibility in future work.
Limitations
More work is needed to uncover the causes of the inconsistent performance across randomly initialized models in Experiment 1. Although the bias toward forward function application implemented in Experiment 2 was effective in our experiments, it is unlikely to work as a general-purpose method, since languages vary in their branching characteristics and in the contexts in which they apply forward and backward function application.
Ethics Statement
We foresee no ethical issues arising from this research. The reader may refer to Brown (1973) for information on how the Adam and Eve childdirected speech corpora were collected.

heading: Acknowledgements
text: We thank the anonymous reviewers for their helpful comments. This work was supported by the National Science Foundation grant #1816891. All
views expressed are those of the authors and do not necessarily reflect the views of the National Science Foundation.

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: In and following Section 7.
7 A2. Did you discuss any potential risks of your work? We see no apparent risks.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 3.2
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We are not currently sharing any new artifacts for this project. Should we share them in the future, we will discuss the terms of use.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? It seems very clear that our extension of the existing Jin et al. 2021 model is being used for the intended purpose, to model grammar induction.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 5.1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments? Sections 5 and 6
7 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? These are relatively small-scale models, so there are few concerns about others having the resources to run them
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 5.2

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Sections 5.3 and 6.2
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Should we share them in the future, we will discuss the terms of use. 7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? It seems very clear that our extension of the existing Jin et al. 2021 model is being used for the intended purpose, to model grammar induction. B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank