CoELM: Construction-Enhanced Language Modeling (2024)

Abstract: Recent studies have shown that integrating constructional information can improve the performance of pre-trained language models (PLMs) in natural language understanding. However, exploration into leveraging constructional information to enhance generative language models for natural language generation has been limited. Additionally, probing studies indicate that PLMs primarily grasp the syntactic structure of constructions but struggle to capture their semantics. In this work, we encode constructions as inductive biases to explicitly embed constructional semantics and guide the generation process. We begin by presenting a construction grammar induction framework designed to automatically identify constructions from corpora. Subsequently, we propose the ConstructionEnhanced Language Model (CoELM). It introduces a construction-guided language modeling approach that employs a dynamic sequence reassembly strategy during pre-training. Extensive experiments have demonstrated the superiority of CoELM across various benchmarks.

heading: 1 Introduction
text: The constructions are linguistic structures that are viewed as a set of form-meaning pairs in Construction Grammar (CxG; Goldberg, 2003; Goldberg et al., 2005). Specifically, a construction is represented as a sequence of slot-constraints from a usage-based perspective (Dunn, 2017, 2019), where the slots encompass different levels of abstraction (e.g., lexical, syntactic). Based on these slots, constructions can be classified into three main types (Ungerer and Hartmann, 2023): (i) Schematic constructions contain open slots solely and convey a general meaning. For example, “SubjectVerb–Object1–Object2” is a ditransitive construction (Goldberg, 1995) that denotes the abstract
† Equal contribution and shared co-first authorship. ∗ Corresponding author.
meaning of transfer. (ii) Semi-idiomatic constructions are partially filled with lexical elements, for instance, “give-Pronoun-a-break”. (iii) Idiomatic constructions are sequences composed of successive fixed words, such as “like a bat out of hell”.
Recent studies (Tayyar Madabushi et al., 2020; Xu et al., 2023) have leveraged constructional information on encoder-only pre-trained language models (PLMs), such as BERT and RoBERTa (Devlin et al., 2019; Liu et al., 2019). These studies demonstrate that incorporating constructional information can enhance the performance of language models in various Natural Language Understanding (NLU) tasks. However, prior research has not explored the utilization of constructional information to augment generative language models for Natural Language Generation (NLG). In particular, the prevalent approach in generative models features a decoder-only architecture (OpenAI, 2023; Touvron et al., 2023), which contrasts with the encoder-only architecture in pre-training tasks. There is a significant gap that our work aims to address.
While PLMs can access constructional information without explicit training (Madabushi et al., 2023), probing studies (Weissweiler et al., 2023a,c; Tseng et al., 2022) suggest that they primarily grasp the syntactic structures but fail to fully capture semantic aspects. In Appendix B, we investigate the perception of language models for constructional semantics via probing experiments. Larger models exhibit better awareness of constructional semantics for both semi-idiomatic and schematic constructions compared to smaller ones, which aligns with the findings by Murty et al. (2023) on PLMs’ capacity to generalize hierarchical structures, albeit slowly. These observations indicate that constructions consist of hierarchically abstract slots, posing intrinsic challenges for PLMs as they involve noncompositional meanings not directly linked to individual words. Consequently, this insight motivates us to adapt pre-training tasks to facilitate the swift
10061
acquisition of constructional information. In summary, our research focuses on developing effective pre-training strategies to efficiently learn constructional information, thereby enhancing language modeling capabilities essential for NLG. To this end, we propose a two-step approach. Initially, we employ automated methods to identify constructions from corpora, creating a comprehensive inventory for pre-training. Subsequently, these identified constructions are encoded as inductive biases during the pre-training phase, which allows model to explicitly capture constructional information and direct the generation process.
In the first step, recognizing the labor-intensive nature of manual construction inventory creation, we prioritize automated methods for identifying constructions. These methods are advantageous as they can uncover a variety of patterns that linguists might overlook (Madabushi et al., 2023). In usage-based CxG, a construction is depicted as a sequence of slot-constraints. To quantify the relationship between individual words and generate these constraints, either frequency-based or associationbased models (Dunn, 2017) are applied. As suggested by Dunn (2019), association-based models provide better generalizations for slot-constraints when compared to frequency-based counterparts.
However, manual inspection of the inventory of constructions indicates that some constructions are relatively short or tend to contain generic labels (e.g., “Preposition + his”; Tsao and Wible, 2013; Tayyar Madabushi et al., 2020), lacking a clear mapping of form to a specific function or meaning. Through empirical study, we explore the possible reasons. Existing methods employ threshold-based strategies, such as frequency and association metrics (Dunn, 2017), to assess the eligibility of adding new slots to a construction. A construction is truncated if the association strength between adjacent slots drops below the specified threshold.
To tackle the issue, we introduce a construction grammar induction framework called CxGLearner. Within this framework, the association strength among slots is assessed through a PLM-based Association Strength Estimator (ASE), which can consider more extended distances when assessing slot constraints. Additionally, inspired by nucleus sampling (Holtzman et al., 2020), we establish a nucleus set that utilizes the ASE’s output distribution. This set guides the decision on whether to append slots to the sequence of candidate constructions, aiming to create complete constructions without
resorting to rigid threshold-based truncation. For the second step, we propose a constructionguided language modeling approach with dynamic sequence reassembly strategy for pre-training. This strategy involves initially identifying all constructs within the input sequences that instantiate the constructions from the inventory. Then we insert constructions before their corresponding constructs, thereby explicitly embedding constructional semantics to guide the generation process. And the curriculum learning (Bengio et al., 2009) is employed to regulate the reassembly of both the types and quantities of constructions in sequences. Meanwhile, this approach ensures compatibility with existing generative language models, eliminating the need for additional decoding procedures.
Subsequently, our Construction-Enhanced Language Model (CoELM)1 is pre-trained from scratch. Extensive experiments have validated the effectiveness of integrating constructional information into language modeling. Consequently, encoding constructions as inductive biases not only expedites the acquisition of constructional semantics for language model but also significantly enhances its performance across a range of benchmark tasks.

heading: 2 Induction Framework for CxG
text: In this section, we propose an unsupervised computational framework, which automatically inducts generic usage-based constructions from corpora.

heading: 2.1 Computational Language Framework
text: Previous works have attempted to automatically extract constructions from corpora (Dunn, 2017; Feng et al., 2022; Lyngfelt et al., 2018), with Dunn (2019) proposing a relatively sensible pipeline. However, as we mentioned in the Introduction, the search strategy in these works determines slot access constructions solely based on neighboring slots and a hard threshold, leading to truncation. To address this issue and ensure compatibility with language modeling, we propose a construction grammar induction framework (CxGLearner) to acquire an inventory of constructions. Since our framework comprises abundant implementation details, we briefly describe the entire pipeline in this section, leaving the detailed information to Appendix D.
As shown in Figure 1, our framework initially encodes the corpus into abstract representations
1Our code is publicly available at https://github.com/ xlxwalex/CoELM
at different levels (e.g., lexical, syntactic). We then iterate through each position and level of slots among the encoded corpus, recursively searching to extract potential construction candidates. To determine whether slots can continue to expand potential candidate sequences, we propose the Association Strength Estimator (ASE), which is pretrained to simulate the association strength between sequences and slots. Subsequently, these constructions are pruned and optimized (Appendix D) to acquire the final inventory of generic constructions.

heading: 2.2 Association Strength Estimator (ASE)
text: Since usage-based constructions are represented as sequences of slot constraints (Dunn, 2017), the key issue is how to measure the constraint relations between slots. Instead of frequency, association strength (Dunn, 2019; Gries, 2013) is applied to quantify the co-occurrence among multi-unit within a language. Prior research (Dunn, 2019) assesses if the association strength2 between adjacent slots are sufficient for combination, which can only focus on localized features. As shown in Figure 1, a potential solution is to determine the association strength between the formed sequence of slots and the slot under consideration. However, this would introduce significant computational complexity for direct association strength assessment. Further details on complexity are discussed in Appendix A.
Recently, studies on the interpretability of language model backbones, specifically transformer, have shown that the attention mechanism in transformer exhibits a dynamic tendency to favor unique key tokens that often co-occur with query tokens (Tian et al., 2023a,b). This evidence aligns with the quantitative goal of measuring association strength, which motivates us to employ ASE, a pre-trained GPT-based language model, for estimating the association strength between sequences and slots.
We first split the background corpus for ASE pre-training. Then the pre-training task of ASE is adapted as next slot prediction, which randomly selects different abstraction levels of the slots in the background corpus. This facilitates the association between slots of varying levels. Then the probability distribution of the output can be exploited to estimate the association strength. The feasibility of ASE is also illustrated in Appendix H.
2Given that our candidate sequences initiate backward searches from arbitrary positions, and considering computational complexity, we opt to estimate unidirectional instead of bidirectional association strength of Dunn (2019).

heading: Encoder & Extractor

heading: 2.3 Candidate Extraction with Nucleus Set
text: To extract construction candidates from the candidate corpus, we initiate a potential sequence with each slot in each sentence. We then conduct a recursive search for the next potential slot across different abstraction levels. A slot is incorporated into the current sequence if it is included in the nucleus set according to the output distribution of ASE. The inclusion process ceases when a slot no longer meets this criterion.
As shown in Figure 1, we regard the candidate sequence as the input for ASE. The ASE then generates a probability distribution for the next slot, denoted as P (si). Drawing inspiration from nucleus sampling (Holtzman et al., 2020) and leveraging the unique aspects of ASE, we utilize the nucleus set instead of a hard threshold to prevent truncation.
We first rank the output probabilities from ASE in descending order. Subsequently, the top-k slots with the highest probability P (si) are retained. Then the probabilities for these slots are normalized to acquire P (si) ′ . Finally, we select the smallest set of top slots, known as the nucleus set N(p), whose cumulative probability surpasses the threshold p, that is, ∑ s∈N(p) P (si)
′ ≥ p. The slot is deemed to fulfill the association condition if it is included in the nucleus set and can be expanded to the candidate sequence, otherwise, it is discarded. In Figure 1, a schematic example begins with the syntactic slot “NOUN” and forms a potential sequence of “NOUN-can-be-ADV”. To determine whether the next possible slots, i.e. “ADJ” and “suitable”, can be added to the sequence, we obtain the output distribution by feeding sequence into ASE. Then “ADJ” is included in the nucleus set
with a cumulative probability of 0.8 , while “suitable” does not meet the condition. Thus, “ADJ” can be expanded to the potential sequence.
This approach offers more flexibility in capturing the dynamic association strength between sequences and slots than the previous method.

heading: 3 Construction-Enhanced LM
text: In this section, we explore how the acquired construction inventory can be utilized to enhance the constructional awareness of language modeling.

heading: 3.1 Constructions as Inductive Bias
text: To employ constructions as inductive biases for steering the generative language model towards a more effective understanding of constructional semantics, we will address two main sub-issues:
(i) How to incorporate the meaning of construction to guide the generation? We propose a dynamic sequence reassembly strategy to tackle this issue. The input sequences are first matched for all the constructs and then distinguished by inserting corresponding constructions before the constructs into sequences. Thus, the constructional semantics can be explicitly fused to language model.
(ii) How to be compatible with existing language models, avoiding external decoding procedure, and adapting to various types of constructions at different granularities? For this issue, we introduce Curriculum Learning (Bengio et al., 2009) to dynamically regulate the selection of constructions.

heading: 3.2 Dynamic Sequence Reassembly Strategy
text: In order to leverage constructions, it is crucial to associate these patterns with their respective constructional semantics (i.e., meaning). We first integrate the construction inventory with the token vocabulary, embedding both into vector space by looking up the embedding matrix E ∈ R|V|+|G|, where |V| and |G| refer to the size of vocabulary and construction inventory, respectively. Then we enhance constructional perception in language models through dynamic sequence reassembly. As shown in Figure 2, we insert constructions before their corresponding constructs during pre-training, thus guiding the generation process. However, the complexity of matching multiple constructions from the encoded multi-level sequences arises due to the overlapping nature of usage-based CxG (Xu et al., 2023; Dunn, 2019). Thus, we refine the Aho-Corasick algorithm (Aho and Corasick, 1975) for construction matching, denoted as AC(·). It matches all
constructions and generates a dictionary M. This dictionary maps each position index of sequence to a set m: the lengths mL, indexes mI , types mT (true for schematic constructions, otherwise false) and probabilities mP of all constructions beginning at that position. In the schematic sample of Figure 2, four constructions are matched at positions starting at 0 and 8. Then each position is traversed and the selection probability ps is exploited to determine the insertion of constructions, while the abstract probability pa is employed for dictating the likelihood of choosing schematic constructions. The CHOICE(·) function is designed to randomly select indexes for overlapping constructions according to a predefined probability distribution. In instances where no constructions are selected, the function yields a value of −1.

heading: 3.3 Architecture of Language Model
text: In this work, we pre-train a language model (CoELM) from scratch and our model architecture largely follows Llama 2 (Touvron et al., 2023), which aligns with the latest best practices in language modeling (Zhang et al., 2024): (1) We use Rotary Positional Embedding (RoPE; Su et al., 2024) as our preferred positional embedding. (2) To attain a more stable training, we normalize the input prior to each transformer layer with RMSNorm (Zhang and Sennrich, 2019). (3) Instead of the standard ReLU non-linearity, we adopt the approach of Llama 2 by combining Swish with the Gated Linear Unit, a method we refer to as SwiGLU (Shazeer, 2020). Additionally, Flash Attention (Dao, 2023) is employed to boost device throughput during the pre-training phase.

heading: 3.4 Training Method
text: In pre-training, our objective is to learn the relationship between constructions and the tokens that compose them. We approach the training process
Algorithm 1: Dynamic reassembly process Input: Construction list G and encoded sequence S.
Probabilities of ps and pa. Output: Reassembled sequence S ′ .
1 The length n and levels l of S: n, l← S.SHAPE 2 The matching results m ∈M← AC(S, G, l, n) 3 Initialize S ′ ← [ ] 4 for i = 0 to n− 1 do 5 token← S[i, 0] // equivalent to lexical slot 6 if i inM.KEYS() then 7 m
′ P ← [p× pa × ps if mT else p× (1− pa)× ps for p in mP ]
8 ind← CHOICE(mI , weight=m ′ P )
9 if ind ≥ 0 then 10 token← S[i : i+mL[ind] , 0] 11 S ′ ← [S ′ , mI [ind] ] 12 i← i+mL[ind] 13 end 14 end 15 S ′ ← [S ′ , token] 16 end 17 return S ′
from two perspectives: (1) Schematic constructions have more abstract slots than semi-idiomatic and idiomatic constructions, resulting in relatively coarser constructional semantics. Thus, we anticipate the model to distinguish schematic constructions during the early stage, and focus more on semi-idiomatic and idiomatic constructions later. (2) Since the language model can learn hierarchical structures, albeit slowly, we only guide the generation process at an early stage. This strategy not only expedites learning but also eliminates the need for specialized handling of the construction units in the generated sequence during inference.
Drawing on the approach of Bengio et al. (2009), we adopt curriculum learning to achieve our objectives. They are both employed in the dynamic reassembling process (ps and pa) in Algorithm 1. Curriculum learning defines the pacing function that regulates the sequence and timing of introducing various difficulty levels of samples during training. As depicted in Figure 3, we establish two distinct pacing functions for different aspects of our goal. The model initiates with a predetermined number of warm-up steps tw to establish token relationship. This is succeeded by a stage where the construction guidance is progressively diminished until stop steps ts. For the selection and abstraction probabilities, we employ linear-decay and cosine-decay pacing functions, respectively.

heading: 4 Experiments

heading: 4.1 Experiments Setup
text: Pre-training corpus. The entire pre-training corpus is sampled from multiple subsets of RedPajama (Soboleva et al., 2023) and Pile (Gao et al., 2020), which aligns with the settings in previous pre-trained language models. Our corpus comprises a total of 188B tokens after pre-processing (Chen et al., 2023), such as deduplication. The detailed statistics and pre-processing procedure of the pre-training corpus are provided in Appendix C.
Evaluation tasks. For evaluating the performance of model (CoELM), we utilize the Language Model Evaluation Harness (Gao et al., 2023) framework to conduct evaluation on various language modeling and commonsense reasoning tasks.
Implementation. In CxGLearner framework, we set the parameters of k and p to be 20 and 0.6, respectively. In total, we induct 13,262 generic constructions for the inventory. As for our CoELM, we adopt the similar hyper-parameters in Pythia410M (Biderman et al., 2023) for comparison. This includes 24 layers with 16 heads, with a hidden size of 1024 and an intermediate size set to 4096. We employ the SentencePiece (Kudo and Richardson, 2018) as the tokenizer with the vocab size to be 50257. During pre-training, we utilize the AdamW optimizer (Loshchilov and Hutter, 2018) with a cosine learning rate scheduler. We exploit 2,000 warmup steps and the batch size is set to 1 million tokens. We pre-train 250,000 steps for running 1.4 epoch on entire corpus. And the tw and ts for curriculum learning are set to 2,000 and 80,000 steps, respectively. More detailed information for settings is demonstrated in Appendix F.
Comparable LMs. We primarily focus on language models which are close to the number of parameters in our implementation. Specifically, we compare our CoELM with GPT-2 Medium (345M; Radford et al., 2019), OPT-350M (Zhang et al.,
2022), Pythia-410M (Biderman et al., 2023) and BLOOM-560M (Workshop et al., 2022).

heading: 4.2 Experiment Results
text: To evaluate the ability of CoELM for reasoning and language modeling, we conduct experimental comparisons on several common benchmarks (Statistics for these tasks are shown in Appendix E). Our main results on the comprehension and reasoning tasks are shown in Table 1. We conduct both zero-shot and few-shot experiments for comparison, from which several observations can be obtained.
First, Pythia outperforms other language models under zero-shot setting, while OPT and BLOOM exhibit similar performance, despite OPT having fewer parameters. GPT-2 achieves higher performance than OPT and BLOOM, but lower than Pythia. Remarkably, our CoELM significantly outperforms other models on most of tasks, especially with huge gaps in the Arc-E, HellaSwag and WSC tasks. This suggests that our CoELM, with constructional information incorporated, possesses superior comprehension and reasoning capabilities.
Second, few-shot learning is designed to answer the questions generalized from a small number of examples. Thus, five exemplars are provided for the prompts. We observe that the performance of language models can be enhanced in the majority of tasks through a few-sample setting, although on a few tasks, the performance decreases instead. For instance, in HellaSwag, a natural language reasoning task with complex contextual premises, a deeper understanding of linguistic knowledge inherent in the language models is required. Conse-
quently, irrelevant knowledge from other exemplars may interfere with answer generation.
Third, the OPT model has the largest improvement in few-shot learning compared to the zeroshot setting. Additionally, all models show significant gains on the OBQA task and our CoELM also has a large improvement on the Arc-E task. Although the average performance of our CoELM is reduced compared to the zero-shot setting due to the effect of the WSC task, CoELM achieves higher average performances across all the benchmarks than other language models.
Moreover, we evaluate the performance on generative task for word prediction. The main results of LAMBADA task are demonstrated in Table 2. In the zero-shot setting, it can be observed that Pythia is significantly better than other language models. And our CoELM outperforms Pythia, demonstrating its language modeling capabilities. However, the performance of all the language models is significantly lower in the few-shot setting compared to zero-shot, which can be due to the effect of ir-
relevant context on the accuracy of the language modeling as well as the ability of the language models to generate long texts. In few-shot setting, Pythia is the least affected and achieves the best performance. The results also suggest the necessity for our CoELM to explore the pre-training process in the long-text scenarios in the future.
These observations confirm the validity of CoELM, with its construction-enhanced generation process. Furthermore, constructions can enhance language modeling across these benchmarks.

heading: 4.3 Comparative Analysis
text: Ablation Study. Since the language models are pre-trained on different sizes and types of corpora as well as varying hyper-parameter settings, we conduct ablation study experiments to further investigate the effectiveness of our CoELM. Therefore, we pre-train a language model from scratch based on Llama architecture, same as our CoELM. We ensure that all hyper-parameters are consistent with our CoELM, except that the model will not be applied to dynamic sequence reassembly (DSR). This model is referred to as w/o DSR. Our model is pre-trained on corpora with 188B tokens, which is smaller than other pre-trained models. For example, Pythia is pre-trained on 207B corpora. As the Pythia suite provides the intermediate results during pre-training, we employ the results of Pythia with 93k steps, which is pre-trained an entire epoch on the corpora. Similarly, we use the intermediate results of the Llama baseline model and CoELM pre-trained on a single epoch for comparison. The results on four benchmarks are shown in Table 3.
It can be observed that the performance of Pythia is slightly lower than that demonstrated in Table 1, which suggests that more pre-training steps could potentially enhance the performance of PLMs. Meanwhile, Pythia and w/o DSR perform very close to each other except for WSC task, indicating that the differences in corpora have less impact on language modeling. Our CoELM achieves better performances on most tasks, further supporting the validity of constructional incorporation for enhanc-
ing language modeling capabilities.
Language Modeling Acceleration. During pretraining, the curves of loss and perplexity of language models first plummet in a power law, and then seemingly enter a near-linear decline. As the constructions are employed as the inductive bias for our CoELM, they guide the generation process. We observe that they can also accelerate the convergence of the model during the pre-training process, as shown in Figure 4. The CoELM w/o DSR enters a linear descent phase at around 40k steps, while CoELM requires only 15k steps and exhibits lower perplexity. Thus, at the end of pre-training, our CoELM also has a lower perplexity on the validation corpus with respect to w/o DSR.
Furthermore, we compare the intermediate performances on five benchmarks between CoELM with 20k steps and w/o DSR with 40k steps and the results are illustrated in Figure 4. In all five tasks, CoELM outperforms w/o DSR. Particularly, there is a big gap in the HellaSwag task. This demonstrates that with the assistance of constructional information, the language model can establish relationships among tokens more efficiently, thus accelerating the language modeling procedure. This characteristic may be extendable to larger scale language models, left for future exploration.
Wug Test. Prior work (Weissweiler et al., 2023b) has applied the wug test, a probing task used to investigate whether the pre-trained language models possess human-like linguistic knowledge. The wug test focuses on morphology, creating non-existent words according to systematic patterns of covariation in form and meaning, such as the past tense variation. This task contains 50 irregular verbs and perturbs these verbs by one or two letters to produce new words. Then these verbs are annotated for past tense of the nonce word, e.g., , wug → wugged. Meanwhile, annotators are requested to

heading: N-shot Model PPL. Accuracy
text: provide multiple possible past tense forms based on morphology for these verbs.
Therefore, we investigate whether constructions can contribute to addressing unseen words based on the linguistic knowledge for language models. To evaluate the performance of wug test, we follow Weissweiler et al. (2023b), considering an instance correct only if the generated answer falls within the set of possible past tense forms of the instance. We report the lowest perplexity of the possible forms within the set and the accuracy in Table 4. The 410M and 1B size models of Pythia and TinyLlama (Zhang et al., 2024) with 1.1B parameters are employed for the comparison. For TinyLlama, we utilize the intermediate model that is pre-trained with 240k steps and fed to 500B tokens.
In zero-shot setting, our CoELM significantly outperforms the other language models, despite both Pythia-1B and TinyLlama having a far greater number of parameters. This indicates that our CoELM has a better comprehension of linguistic knowledge by incorporating constructional information, enabling it to generalize to unseen words. As for the few-shot learning3, Pythia-1B and TinyLlama achieve better performances on wug test. This is due to the better in-context learning capability of the larger scale models (Dong et al., 2022). However, CoELM also improves with few-shot prompting and outperforms Pythia-410M, demonstrating the effectiveness of our CoELM.
Dimensionality Reduction Analysis. To further explore the impact of constructional information for language modeling, we conduct a visual analysis of representation obtained from our CoELM,
3For few-shot setting, we do not employ the prompts from Language Model Evaluation Harness framework in addition to the other Baselines from CoELM since the prompts constructed from the wug dataset introduce more unseen words, see Appendix J for more detailed discussion.
w/o DSR and Pythia. First, we randomly select five semi-idiomatic constructions from the learned inventory. Then we search the WikiText corpus (Merity et al., 2017) for 50 constructs of each construction. These constructs are fed into different models and the representations are acquired from the last token of each construct. Subsequently, we apply dimensionality reduction using UMAP (McInnes et al., 2018). As shown in Figure 5, the representations in CoELM form clusters with distinct boundaries, while the representations in w/o DSR and Pythia are diffuse among different clusters. This observation suggests that our CoELM effectively incorporates constructional semantics through the dynamic sequence reassembly strategy.

heading: 5 Related Work
text: Construction Grammar (CxG) has been applied to various natural language processing tasks. Tsao and Wible (2013) utilize constructions as contextual features for word similarity detection. Raghuram et al. (2017) leverage Embodied CxG to address the reference resolution problem, and Nevens et al. (2019) apply Fluid CxG for semantic parsing in visual question answering. Xu et al. (2023) integrate constructional information to improve the performance of PLMs in NLU tasks. However, there has been no effort to ascertain whether constructions can provide benefits for generative language models. Our work aims to bridge this gap.
Recent research investigates whether PLMs can access constructions. Tayyar Madabushi et al. (2020) find that BERT is capable of accessing a considerable amount of constructional information without explicit training. Li et al. (2022) conduct probing studies to show that PLMs can capture argument structure constructions. Tseng et al. (2022) investigate PLMs’ awareness of constructions and find that predicting open slots is more challenging than closed ones. Weissweiler et al. (2022) find that while PLMs can recognize the construction, they struggle with understanding its meaning. Fur-
thermore, Weissweiler et al. (2023c) discover that both autoregressive and non-autoregressive PLMs can distinguish constructional patterns but fail to apply their meanings. These findings underscore the need to adapt pre-training tasks to enable PLMs to rapidly acquire constructional information.
An inventory of constructions is a valuable resource for construction-based research. As manual extraction is often time consuming, automated approaches have been proposed to identify constructions from corpora. These methods are advantageous for uncovering patterns that may have been previously unexplored or highly conventional (Madabushi et al., 2023), potentially overlooked in traditional analyses. Wible and Tsao (2010) present a hybrid n-grams model to discover constructions that meet a frequency threshold. Dunn (2017) proposes an induction algorithm to extract constructions based on association strength. Lyngfelt et al. (2018) develop automatic tools to detect partially schematic constructions using frequency metrics. Dunn (2019) suggests that associationbased models produce better generalizations for slot-constraints compared to frequency-based models. However, manual analyses reveal that some extracted constructions are short or lack informativeness (Lyngfelt et al., 2018; Tayyar Madabushi et al., 2020). Li et al. (2022) argue that there is currently a lack of a high-quality and wide-coverage construction dataset. Therefore, we investigate potential reasons and develop a construction induction framework to address the issue.

heading: 6 Conclusion
text: Our research is dedicated to developing effective pre-training strategies for the rapid acquisition of constructional information to enhance language modeling capabilities. To achieve this goal, we initially introduce a construction induction framework that automatically identifies constructions from corpora. We employ a PLM-based Association Strength Estimator (ASE) and a nucleus set to generate complete constructions, avoiding the rigid threshold-based truncation. Subsequently, we propose a construction-guided language modeling approach utilizing a dynamic sequence reassembly strategy. This method encodes constructions as inductive biases during pre-training, thereby guiding the generation process. We pretrain our Construction-Enhanced Language Model (CoELM) from scratch, and extensive experiments
have confirmed the effectiveness of CoELM.

heading: Limitations
text: In this work, the limitations can be summarized into two main aspects for future discussion:
(1) As we mentioned in Appendix C, we primarily focus on the generation of natural language from a linguistic perspective, without utilizing code data for pre-training procudure. However, constructions also exist in code (Mosses, 2021), which can be viewed as pairing of form and function. Nevertheless, they differ in their presentation compared to constructions in linguistics. Therefore, we will further discuss them in future work.
(2) In our main experiments, we observe that smaller-scale models exhibit diminished performance on instruction following and long text, resulting in insignificant improvements over the zeroshot setting. Therefore, we will further explore the refinement of the pre-training data procedure and the optimization of model architecture.
(3) In order to acquire better interpretable constructions, as we stated in Appendix D.1, we utilize the language-specific hierarchy, i.e., XPOS. Consequently, our pre-training procedure is conducted exclusively with the English corpus (More details are shown in Appendix C). This limitation restricts our CoELM from being multilingual. In subsequent research endeavors, we intend to investigate the significance of generic constructions within the domain of language modeling.
(4) Due to limited computational resources, we are constrained to pre-training a relatively smallscale model. However, the probing experiments we conduct in Appendix B reveal that incorporating constructional information into larger-scale models may also enhance their language modeling capabilities, which we leave for future exploration.

heading: Ethics Statement
text: In this work, we use publicly available corpora and benchmarks under their licenses. These publicly available data are checked to ensure that they do not include any offensive and illegal content.

heading: Acknowledgements
text: We thank all anonymous reviewers for their insightful comments and suggestions. This research is supported by the Science and Technology Project of Zhejiang Province (2022C01044) and the National Natural Science Foundation of China (51775496).

heading: A Background of Construction Grammar
text: Construction Grammar (CxG) is an influential paradigm within cognitive linguistics, positing that grammar encompasses a meaningful continuum of lexicon, morphology, and syntax. This perspective diverges from traditional views that prioritize a stable framework of arbitrary rules for producing well-formed sequences. In CxG, the fundamental units of grammar are constructions, which is defined as symbolic pairings of form and meaning (Langacker, 1987; Goldberg, 1995, 2006). These pairings underscore the central thesis of the theory: grammatical structure is inherently meaningful, with each construction serving as a vital component in the conveyance of specific semantic content. This approach emphasizes their role in linking morphosyntactic patterns directly to their associated meanings. As described in the Introduction, Construction Grammar (CxG) is characterized by its diverse syntactic structures, which span different levels of abstraction. These structures can comprise both partially and fully specified components (Goldberg, 2003), which allows for the substitution of specific words for productivity.
The CxG paradigm has evolved to several distinct implementations, each adopting a unique perspective on language structure and processing. These include the formal approaches such as Fluid Construction Grammar (FCG; Steels and de Beule, 2006), which emphasizes the dynamic nature of linguistic constructions, while Embodied Construction Grammar (ECG; Goldberg et al., 2005) integrates the role of sensory and motor experiences in linguistic interpretation. Sign-Based Construction Grammar (SBCG; Rambelli et al., 2019) combines insights from formal linguistic theory with signbased language analysis. However, these formal approaches all rely heavily on manual definition with prior knowledge, such as FrameNet (Baker et al., 1998). In usage-based CxG, patterns that occur with sufficient frequency can also be considered as constructions (Goldberg, 2006). It enables constructions to be acquired from the corpus.
In previous research, Dunn (2017, 2019) propose a computationally slot-constraints pipeline for construction extraction, which is composed of three stages: (1) In representation stage, the slots are represented to three levels, i.e., lexical, syntactic and joint semantic-syntactic. The syntactic slots are denoted to the universal POS tags, while the semantic-syntactic slots are formed from static em-
bedding of words (e.g., FastText) that are clustered in discrete semantic domains via K-Means clustering. (2) In construction identification stage, Dunn (2017, 2019) initially measures the slot constraints based on the association strength, which is applied to quantify the co-occurrence among multi-units within a language (Gries, 2013). The unidirectional association strength ∆u can be defined as:
∆u = C(x,y)
Cx − Cy − C(x,y) Call − Cx
(1)
where x and y are the adjacent slots. Cx and Cy are the frequency of x and y in the corpus, while C(x,y) is the frequency of co-occurrence of x and y. And Call is the total number of occurrences of all slots. In order to compute the association strength, Dunn (2017, 2019) first counts the corpus and calculates the frequency of 1-gram and 2-gram items. However, this approach can only assess the association strength between two adjacent slots, thus focusing solely on localized features. Following this, candidate sequences are generated through a recursive search conditioned on the association strength between slots using a hard threshold. Subsequently, these candidate sequences are pruned to derive potential constructions. (3) In candidate evaluation stage, the tabu search is applied to determine the optimal set of constructions with Minimum Description Length (MDL; Rissanen, 1978) metric. Then the inventory of constructions is acquired.
In order to assess slot by considering longer slot constraints for construction candidates, one possible solution would be to determine the association strength between the formed sequence of slots and the slot under consideration. However, if we directly compute the association strength between sequences and slots based on the corpus via the approach of Dunn (2019), this would introduce an unacceptable computational complexity. As an example, if we have N possible slots and the currently formed sequence is of length L, then in order to assess the association strength between the sequence and the (L+ 1)-th slot, we have two implementations: (1) we can store all the candidates of length L. (2) The frequency of occurrences of sequences and slots can be re-computed in the corpus each time. For approach (1), it would require storing NL floating point numbers for the frequency of L-gram items in the worst case. If both N and L are large (In practice, N is about 50k, and L is between 3 and 7), it may exceed the server’s capacity. As for approach (2), it is computationally infeasi-
ble to re-compute the frequency of sequences each time from a corpus containing around 3B tokens. Therefore, we propose the Association Strength Estimator (ASE) to approximate the association strength between sequences and slots, which can compute the association strength fast on a limited memory footprint.

heading: B Probing Experiments
text: To investigate the ability of PLMs to capture constructional semantics, we conduct probing experiments on different scales of GPT-2 language models (Radford et al., 2019). We focus primarily on semi-idiomatic and schematic constructions, which exhibit productivity and can be filled with words in their abstraction slots. As shown in Figure 6, we randomly select five semi-idiomatic and schematic constructions, respectively. Then we match 50 constructs (abstract slots filled with actual words) from WikiText corpus (Merity et al., 2017) for each construction. After feeding these constructs into the GPT-2 models to obtain representations (Jiang et al., 2023), we apply dimensionality reduction with UMAP (McInnes et al., 2018) and then cluster the reduced representation via HDBCAN (McInnes et al., 2017). As these constructs are the instantiation of constructions, their representations also inherit the constructional semantics of corresponding constructions (Goldberg, 2003). For constructs instantiated from the same construction, it should be more plausible that they will be clustered together. Thus, we utilize two metrics for evaluating clustering accuracy: Purity and the Adjusted Rand Index (ARI; Hubert and Arabie, 1985). The higher these metrics are, the better the model can distinguish the constructional semantics.
We perform five independent probing experiments, which are shown in Figure 6 (a) and (b). Our main finding is that larger models indeed exhibit
constructional semantic awareness capabilities for both semi-idiomatic and schematic constructions. This aligns with the idea that larger models, trained with larger corpora and more training steps under the Scaling Law (Hoffmann et al., 2022), are more adept at capturing such nuances. Meanwhile, (c)~(f) are the clustering results after dimensionality reduction. It is observed that the representations in larger models form clusters with distinct boundaries, while the representations in smaller models are diffuse among different clusters
To investigate if the observed trend in constructional semantic perception on larger scale models aligns with our findings, we then conduct probing experiments on Llama models (Touvron et al., 2023). As shown in Figure 7, we exploit TinyLlama-1.1B, Llama-7B, and Llama-13B for our experiment. TinyLlama (Zhang et al., 2024) is pre-trained on SlimPajama (Soboleva et al., 2023), which replicates the pre-training data of Llama, and thus can be directly compared with the other two Llama models. In terms of trends, the Llama series with a larger number of parameters exhibits the same behavior as GPT-2 series. In both semiidiomatic and schematic constructions, we observe that the purity and ARI metrics increase with the number of parameters. This also indicates that larger models indeed have more powerful modeling capabilities for constructional semantics. Our findings suggest the potential of regarding constructions as the inductive bias for language models to boost their performance.

heading: C The Details of Our Pre-training Data
text: Our training corpus comprises a variety of sources, which cover a diverse set of domains. They are mainly from RedPajama (Soboleva et al., 2023) and Pile (Gao et al., 2020), which align with previous practices in pre-training language models. As we discuss in Limitations, our work is centered on the generation of natural language, whereas code behaves differently in terms of constructions. Con-

heading: Corpus Tokens Sampling prop. Epochs
text: sequently, our study does not incorporate any coderelated subsets (e.g., GitHub and Stack Exchange).
In the previous work (Lee et al., 2022; Li et al., 2023), the importance of a high-quality pre-training corpus for model generalization performance is illustrated. Thus, we follow the instructions in Chen et al. (2023); Biderman et al. (2023) to deduplicate the corpus. As shown in Table 5, the corpus consists of eight main subsets: (1) CCNews. We preprocess and randomly sample a news subset of CommonCrawl4, which contains news articles all over the world, ranging from 2016 to 2023. (2) C4 (Raffel et al., 2020) is a publicly available corpus which contains massive web texts. We sample and retain the English documents from it. (3) Wikipedia is a widely used corpus for pre-training language models that contains high-quality knowledge. Therefore, we employ all English documents and increase its sampling probability during pretraining. (4) Books is an open-source subset in RedPajama, which we utilize the entire corpus. And follow the advice in Touvron et al. (2023), its sampling probability is also be raised. (5) Law is derived from the FreeLaw project included within the Pile. It is a publicly available resource for academic studies in the legal realm. (6) ArXiv is a collection of research papers across multiple fields (e.g., computer science, math). As ArXiv papers are written in LaTeX, which is a typesetting language, we parse these files and keep the text parts for pre-training. (7) USPTO contains background sections from patents granted by the United States Patent and Trademark Office, which incorporates a large number of technical documents. (8) Other corpus is composed of three subsets in Pile, i.e., Enron Emails, PhilPapers and NIH Grant Abstracts.
4https://data.commoncrawl.org/CC-NEWS/index.html
D Implementation of CxGLearner
In Section 2, our induction framework5 is introduced for automatically searching constructions from the corpora. We will elaborate on the specifics of the framework for better comprehension.

heading: D.1 Overall of Induction Framework
text: As shown in Figure 8, our framework initially preprocesses the corpus across various domains. Then the processed corpus from each domain is segmented into three distinct parts for different stages. During the first stage, the background corpus serves to pre-train a language model, referred to as the Association Strength Estimator (ASE). The ASE assesses the association strength between sequences and slots, and it also helps generate representations for candidates. Subsequently, a recursive search with the ASE is conducted on the candidate corpus to extract potential candidates, which are then pruned in the hypothesis space. Finally, we apply heuristic algorithms to optimize potential candidates, simulating psychological principles, with the aim of deriving the optimal set of constructions for each domain. Ultimately, the generic construction inventory is determined by synthesizing the construction sets across all domains.

heading: D.2 Pre-Processer and Encoder
text: Initially, we need to pre-process the corpus to avoid impacting the encoder. Thus, a Unicode checker is used to fix broken Unicode characters and the white spaces in corpus are also normalized for uniformity. Then we substitute the format-specific information in the text (e.g., email, phone) with special tags, such as <EMAIL> and <PHONE>.
We then partition the corpus into three distinct parts, which are in the ratio 6:3:1 for background, candidate and learner corpus, respectively. After
5Our code of induction framework will be publicly available at https://github.com/xlxwalex/CxGLearner
that, these sub-corpora are encoded into slots at different levels of abstraction. In this work, we encode the corpus with three levels, i.e., lexical, Universal Part-of-Speech tags (UPOS; Petrov et al., 2012) and Language-Specific Part-of-Speech tags (XPOS; Marcus et al., 1993). And the UPOS and XPOS provide coarse-grained as well as fine-grained slots for the syntactic level, respectively. Though the state-of-the-art models have reached the human ceiling on the part-of-speech tagging task and are close to solving the problem completely (Manning, 2011), we employ the voting method of ensemble learning (Zhou, 2012) to synthesize the results of multiple parsers (e.g., spaCy and Stanford Parser; Honnibal et al., 2020; Manning et al., 2014) to obtain results as robust as possible for the syntactic level slots. Since the granularity of slots at the syntactic level is at the word level, whereas the slots at the lexical level have sub-word granularity, we assign the same syntactic level slots to all sub-words of a specific word, and their intervals are tracked with the masks to align them at different levels.

heading: D.3 Pre-training Procedure of ASE
text: Prior works (Dunn, 2017, 2019) have utilized an association-based approach to assess the constraints between adjacent slots, which only focus on localized features. A potential approach for addressing a global perspective is to utilize the entire generated potential construction sequence as context in determining whether the next slot satisfies the association condition instead of just focusing on the adjacent slots. However, this can introduce significant complexity due to a large solution space.
Thus, as described in Section 2.2, we propose a PLM-based Association Strength Estimator (ASE) to assess the slot constraints. The ASE is based on GPT-2 architecture (Radford et al., 2019), which is pre-trained on background corpus for each domain via the Next Slot Prediction (NSLP) task. During pre-training, we select slots randomly from different levels for the encoded pre-trained corpus. In order to accommodate to slots of different granularity, we adapt the Whole Word Masking (WWM; Cui et al., 2021) technique. If a slot at the syntactic level is selected and its corresponding lexical slot is a sub-word token identified by the mask, then the entire word is selected as a whole. The ASE is pre-trained for multiple epochs on the corpus, and different slots are selected dynamically at each epoch so that more combinations of slots can be captured. Then we feed the sequence into the ASE
for inferencing, whose output can be exploited as an estimate of the association strength between the sequence and an arbitrary slot. Furthermore, ASE is also employed to generate representations of constructions for the learner component.

heading: D.4 Candidate Extraction and Pruning
text: The component of extractor is employed to generate potential candidates from the candidate corpus. As introduced in Section 2.3, all the slots in each sentence are utilized as starting positions in the potential sequence, and then a recursive search is utilized to add slots to the sequence that satisfy the association condition (the ASE and the strategy of nucleus set are applied to determine the addition of slots, which is elaborated in Section 2.3). In search procedure, we use a breadth-first traversal to determine whether the constraints hold among the current potential sequence and all the hierarchical slots in the next position. Meanwhile, as long as the condition of adding slots is satisfied, the added sequence will be included to the result set as a potential construction sequence.
Under this setting, it might result in some of the candidates with non-optimal boundaries. Thus, we implement a pruning procedure to refine the hypothesis space of these candidates. The process comprises four stages of pruning: (1) Rule pruning. It filters out constructions that fail to comply with the set rules, such as cross-sentence. (2) Frequency pruning. In usage-based perspective, constructions are the patterns that occur with sufficient frequency. Thus, this pruning stage aims at preserving as many generic candidates as possible. (3) Horizontal pruning. It tends to keep longer potential candidates. More specifically, if a short potential sequence is only contained within a longer candidate sequence, then we just keep the longer one. (4) Vertical pruning. It is employed to eliminate candidates that include superfluous abstraction slots. If a slot in a construction can only be instantiated to one possible word and the instantiated construction is included in the result set, we only retain the instantiated construction.

heading: D.5 Optimization of Construction Inventory
text: After extracting potential construction candidates, it is necessary to further refine the candidate set in learner component, as not all usage-based constructions are worth storing (Jackendoff, 2003). There are four psychological principles considered essential for the organization of constructions in a
language. These principles can be integrated into our efforts to optimize the potential construction set (Goldberg, 2003; Goldberg et al., 2005). We separate the four principles into two groups of optimization objectives as follows: (1) The principles of Maximized Expressive Power and Maximized Economy. From the usage-based perspective (Dunn, 2019; Goldsmith, 2006), these two principles are interdependent, striking a balance between construction complexity (the encoding size of construction list) and descriptive adequacy (the encoding size of a corpus given the grammar). Thus, the minimum description length (MDL; Rissanen, 1978) is aptly suited for modeling these principles. The MDL metric can be defined as:
SMDL = γSCom(G)− ηSDes(T |G) (2) where G denotes the list of constructions, while T refers to the learner corpus. The γ and η serve to balance the two terms. The construction complexity, defined as the total information content within the construction list, is represented by the first term:
SCom(G) = − ∑
g∈G
|g|∑
s=1
1 |g| log 1 Ns (3)
Here, g denotes a construction comprising |g| slots. Ns indicates the number of instantiations for each slot: it is 1 for a lexical slot, and for the abstract slot, it refers to the number of tokens that can fill it. The second term is intended to assess descriptive adequacy and is defined as follows:
SDes(T |G) = 1
n
n∑
j=1
logRcovj − logRovej (4)
where n denotes the number of sentences in the corpus. The constructions within each sentence are scanned, with Rcovj representing the coverage while Rovej indicating the overlap of matched constructions in the j-th sentence.
(2) The principles of Maximized Motivation and No Synonymy. These principles also symbolize a pair of mutually dependent goals, indicating that if two constructions are syntactically related, they also bear a semantic relationship to a certain degree, and the converse applies as well. Thus, in a similar vein to contrastive loss (Chen et al., 2020), we propose the concept of contrastive scoring:
SC = − ∑
g∈G log
∑ g+ exp (σg,g+SIM(hg, hg+)/τ)∑
g− exp (SIM(hg, hg−)/τ)
(5)

heading: Benchmark Type # Num. Answer
text: syntactic distance σ is determined by the proportional overlap length between two constructions. For abstract slots, overlap includes the inclusion relation, but this is counted as half the length of the overlap. If the syntactic distance exceeds 0, g+ is a positive sample of g, otherwise negative g−. ASE is employed to provide the representations h of the constructions, while SIM is utilized to compute the semantic similarity (e.g., cosine similarity). τ is a temperature coefficient for adjusting the sensitivity to differences between positive and negative pairs.
We employ heuristic algorithms (e.g., Simulated Annealing; Kirkpatrick et al., 1983) to optimize both objective min{αSMDL + βSC}, where α and γ are weighting factors. Then the optimal construction inventory for each domain is obtained.

heading: D.6 Generic Construction Synthesization
text: Following the previous three stages, we obtain a proprietary set of constructions for each domain. In order to acquire the generic inventory, we synthesize the set of constructions across all domains. More specifically, we retain only those constructions that emerge in at least two domains. The fact that these constructions exist across domains is evidence of their general applicability. After that, the generic inventory of constructions is available.

heading: E Details of Benchmark Tasks
text: In Section 4, we conduct experiments for comparison on various benchmark tasks, including 10 tasks from Language Model Evaluation Harness (Gao et al., 2023) framework. As shown in Table 6, we present the statistics for these benchmarks.
Physical Interaction Question Answering (PIQA; Bisk et al., 2020) task primarily focuses on evaluating understanding of physical common sense. The Boolean Questions (BoolQ; Clark et al., 2019), Question-answering NLI (QNLI) and Winograd Schema Challenge (WSC; Levesque et al., 2012) tasks are all included in GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019). BoolQ is a question answering task that involves analyzing a brief passage from Wikipedia followed by a yes/no question pertaining to the text, while WSC presents a coreference resolution task where each example comprises a sentence that includes a pronoun and a list of noun phrases extracted from that sentence. Then the PLMs are requested to accurately identify the noun phrase that the pronoun refers to from the provided options. And QNLI (White et al., 2017; Demszky et al., 2018) is converted from questionanswering datasets (e.g., SQuAD). The task is to determine whether the context sentence contains the answer to the question. HellaSwag (Zellers et al., 2019) is designed to assess the ability of the models to understand both grounded and temporal aspects

heading: Model Params E-Params
text: of common sense. The AI2 Reasoning Challenge (ARC; Clark et al., 2018) contains two subsets, i.e., ARC-Easy (Arc-E) and ARC-Challenge (Arc-C), which are centered on science exam questions that necessitate complex reasoning under different levels of difficulty. OpenBookQA (OBQA; Mihaylov et al., 2018) is a question-answering dataset modeled after open book exams for assessing human understanding of a subject. LogiQA (Liu et al., 2020) is a task for testing human logical reasoning, covering multiple types of deductive reasoning. LAMBADA (Paperno et al., 2016) serves as a benchmark dataset for assessing the proficiency of language models in comprehending text through a word prediction challenge.

heading: F Hyper-parameters and Settings
text: In Table 7, we provide the full configuration details for pre-training our CoELM. We implement our CoELM using PYTORCH and LIT-GPT framework 6. Then CoELM is pre-trained with 250,000 steps from scratch on a 8 × NVIDIA GeForce RTX 4090 device with about 3,000 GPU hours.
As shown in Table 8, we compare the number of parameters of our CoELM with other baseline models. Though we utilize hyper-parameters that are harmonized with the other language models, it can be observed from the results that our CoELM has more parameters than GPT-2, OPT, and Pythia, but fewer than BLOOM. This is mainly due to two aspects: (1) the size of our embedding layer consists of two parts, the tokens in vocabulary and the inventory of constructions. Therefore, the embedding layer has a larger number of parameters. (2) Our CoELM exploit the Llama architecture, in which SwiGLU is used. Thus, each layer will have 4M more parameters than the baseline models, bringing about a gap in the number of parameters.
6https://github.com/Lightning-AI/lit-gpt

heading: Domain Corpus # Candidate # Inventory

heading: G Statistics of Construction Inventory
text: As described in Section D, the inventory of constructions is synthesized across various domains for acquiring the generic constructions. In order to be applicable to the pre-training process, we identify five primary domains for domain-specific construction acquisition, which are shown in Table 9. For each domain, we sample the corpus of 3B tokens, except for the commercial domain, where we exploit the entire corpus since the Mail corpus is relatively small. The candidate constructions generated by the extractor component and the inventory of pruned and optimized constructions are then reported. After that, we synthesize constructions across these domains and obtain 13,262 generic constructions in the inventory.
As shown in Table 10, we present some representative constructions for the five domains. Semiidiomatic and idiomatic constructions are used for demonstration as they could be observed more intuitively. For example, “mail” is widely used for contacting in the commercial domain. And the construction “quote–of–the–NOUN” is often exploited to make references to correspondence and matters, while “to–contact–PROPN” is utilized to indicate who can be contacted. Meanwhile, “reservation– of—rights” and “this–section–AUX–not–prior” are typical constructions from the legal and professional domain. These observations illustrate the validity of our framework.

heading: H Detailed Configuration of ASE
text: In our induction framework, we utilize ASE to simulate unidirectional association strength (Dunn, 2019) for assessing slot constraints, which is introduced in Section 2.2 and Appendix D. We employ GPT-2 architecture to implement ASE and pre-train it with a next slot prediction task. To explore the correlation between the output of ASE and the asso-

heading: Domain Example
text: Commercial quote–of–the–NOUN to–contact–PROPN please–VERB–DT–NOUN Knowledge AUX–serve–as sponsored–by–the–NOUN NN–directed–by–PROPN Academic estimated–from–DET–NOUN analytic–expression–IN uniquely–determined–by Reportorial commissioner–for–NOUN after–allegedly–VBG holdings–in–shares–of–NOUN
Professional
reservation–of–rights claimed–IN–NOUN–AUX discharge–of–NOUN this–section–AUX–not–prior
Table 10: Examples of constructions across five different domains.
0.68
1⇥
2.4⇥
1.6⇥
0.79 0.82
Figure 9: Comparison among different sizes of ASE.
ciation strength, we pre-train ASE on academic domain with ArXiv corpus (Table 9) and conduct comparative experiments for different sizes of ASE.
We compare three different sizes of the ASE and keep the other hyper-parameters the same, which are denoted as Tiny, Small, and Base, respectively. For the hyper-parameters, we set hidden size to 768 with 12 heads, while the intermediate size is set to 3072. Meanwhile, the numbers of layers of these three sized ASEs are set to 4, 6 and 12, respectively. Then we randomly select 10,000 pairs of slots in the encoded candidate corpus of the Arxiv for comparative experiment.
As shown in Figure 9, we demonstrate the correlation coefficients between the outputs of ASE and the association strengths, as well as the inference times for different size of ASEs. The correlation coefficients of Small and Base are relatively close to each other, both indicating a high degree of cor-
relation. The correlation coefficient for Tiny is significantly lower than Small and Base, which can be considered moderately correlated. However, larger size models also require longer inference times as well as more computational resource consumption. We can observe that the inference time of Base is 2.4 times that of Tiny. For the trade-off between high correlation and inference speed, the Small ASE with 6 layers is employed to assess the slot constraints. Meanwhile, the high correlation between the output of ASE and association strength can also demonstrate the feasibility of our ASE.

heading: I Optimization Process of Learner
text: As we described in Section D.5, the learner component is employed to optimize the potential construction candidates. Thus, we utilize MDL metric and contrastive score that correspond to two groups of psychological principles for optimization.
We illustrate the trend of each objective during the optimization process for academic domain in Figure 10. The first row of the figure presents the components of the MDL metric, which we introduced in Equation 2 ~ 4. In Figure 10, we can observe that as the optimization proceeds, the coverage ratio of the constructions over the learner corpus keeps increasing while the overlap between constructions decreases, which is aligned with our optimization objectives. Meanwhile, the construction complexity and contrastive score are also descending. Since the scales of the objectives are different, we introduce balancing coefficients to allow them to be close to each other in order of magnitude. Ultimately, the total objective score converges in the optimization procedure, demonstrating that our learner component can achieve the optimization goals for all psychological principles.

heading: J Few-shot Setting for Wug Test
text: In Wug Test (Section 4.3), for language models other than CoELM, we do not directly utilize the prompt from Language Model Evaluation when conducting few-shot setting. Instead, we exploit the prompts suggested by Weissweiler et al. (2023b). This is due to the discovery that utilizing other unseen samples from the Wug Test as prompts would lead to significantly diminished performance of the baselines. As shown in Table 11, both Pythia-1B and TinyLlama exhibit high perplexity and low accuracy in such a few-shot setting. In contrast, our CoELM significantly outperforms them. We believe this is due to the constructional information introduced during the pre-training, which can guide the model to recognize the part of speech of unseen words and to perform tense transformation. This illustrates that with the aid of constructional information, the language models may achieve greater robustness towards unseen words.

heading: Future Work