Enhancing Language Representation with Constructional Information for Natural Language Understanding
Year: 2023

Abstract:
Natural language understanding (NLU) is an essential branch of natural language processing, which relies on representations generated by pre-trained language models (PLMs). However, PLMs primarily focus on acquiring lexicosemantic information, while they may be unable to adequately handle the meaning of constructions. To address this issue, we introduce construction grammar (CxG), which highlights the pairings of form and meaning, to enrich language representation. We adopt usage-based construction grammar as the basis of our work, which is highly compatible with statistical models such as PLMs. Then a HyCxG framework is proposed to enhance language representation through a three-stage solution. First, all constructions are extracted from sentences via a slot-constraints approach. As constructions can overlap with each other, bringing redundancy and imbalance, we formulate the conditional max coverage problem for selecting the discriminative constructions. Finally, we propose a relational hypergraph attention network to acquire representation from constructional information by capturing high-order word interactions among constructions. Extensive experiments demonstrate the superiority of the proposed model on a variety of NLU tasks.

heading: 1 Introduction
text: Recent progress in natural language processing relies on pre-trained language models (PLMs) (Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020; Brown et al., 2020; Raffel et al., 2020; He et al., 2021), which generate word representations by capturing syntactic and semantic features from their context. Typically, PLMs (e.g., BERT) employ the Masked Language Model (MLM) as a pre-training objective, randomly masking some tokens from the input before predicting the original tokens. However, PLMs primarily focus on acquiring lexicosemantic information (Tayyar Madabushi et al.,
∗ Corresponding author.
2020) while the meaning is not attached to the words instantiating the construction, but rather to the abstract pattern itself (Weissweiler et al., 2023). Learning the representation of more complex and abstract linguistic units (called constructions) can be more challenging, such as collocations, argument structures and sentence patterns. For example, the ditransitive construction (Goldberg, 1995), which involves the form “Subject–Verb–Object1– Object2”, denotes the meaning of transfer or giving of an entity (Object2) from an agent (Subject) to a recipient (Object1), such as “John gave Mary a book”. Linguistic experiments have proved that constructions can substantially contribute to sentence comprehension in multiple languages, such as English (Bencini and Goldberg, 2000) and German (Gries and Wulff, 2005).
As the meaning of a construction is assigned to a language pattern rather than specific words (Hoffmann and Trousdale, 2013), the constructional information is rarely acquired by MLM and requires large bulk training data, which may lead to failure in natural language understanding tasks with constrained low resource data. As shown in Table 1, we illustrate three examples from aspectbased sentiment analysis (ABSA) tasks. The attention scores are derived from BERT-SPC (Devlin et al., 2019), a simple baseline by fine-tuning
4685
aspect-specific representation of BERT. In the first example, the model focuses mainly on the opinion word “friendly” while ignoring the modality construction of “should be”, resulting in the misclassification. In the second case, “too hard to” expresses a negation for the “fancy food”. However, the model ignores the collocation of “too...to” structure, and the “food” is wrongly considered to be positive. The third example illustrates that lack of understanding of conditional sentence (if clause) causes the model to make an incorrect prediction.
Observations from both linguistics and NLU standpoints motivate us to exploit construction grammar (CxG; Goldberg, 1995, 2006) as the inductive bias to complement language representation with constructional information. Construction grammar refers to a family of linguistic approaches which regard constructions as the basic unit of language. Composed of pairings of form and meaning, constructions pervade all levels of language along a continuum (e.g., morphology, lexicon and syntax).
However, when bridging the gap between CxG and NLU, we face three critical questions:
(Q1) Which CxG approach is applicable for NLU? (Q2) How can typical constructions be identified? (Q3) How can constructions be encoded?
To answer (Q1), we investigate different variants of CxG. Instead of formal methods (e.g., fluid construction grammar and embodied construction grammar), we adopt usage-based approaches which assume that grammar acquisition involves statistical inference. Therefore constructions can be induced as frequent instances of linguistic units. This assumption makes usage-based approaches highly compatible with statistical models such as PLMs (Kapatsinski, 2014). Specifically, we follow the efforts of Dunn (2019) with a computationally slotconstraints approach that formulates constructions as the combinations of immutable components and fillable slots. As shown in Table 1, the modality construction “NOUN–AUX–be” in the first example contains an immutable component “be” and two fillable slots (NOUN and AUX) with a meaning of expressing advice or suggestion.
As for (Q2), from the usage-based perspective, constructions are often stored redundantly at different levels of language. Therefore constructions can overlap with each other, which results in redundancy and imbalance. Consequently, we formulate the problem of selecting discriminative constructions as multi-objective optimization for con-
ditional maximum coverage. To alleviate the computational complexity, we adopt a heuristic search algorithm, i.e., simulated annealing (SA) to determine the composition of constructions.
In order to address (Q3), we propose a relational hypergraph attention network (R-HGAT) to capture high-order word interactions inside the constructions, so as to acquire representation from constructional information. R-HGAT generalizes hypergraph attention network using its flexible labeled hyperedges. We refer to the entire framework which involves construction extraction, selection and encoding as hypergraph network of construction grammar (HyCxG)1.
Extensive experiments have been conducted to illustrate the superiority of HyCxG on NLU tasks, while multilingual experiments further indicate the constructional information is beneficial across languages. Additionally, based on the constructional representations acquired by our model, we conduct an empirical study of building a network of constructions, which provides meaningful implication and in turn contributes to usage-based CxG.

heading: 2 Construction Extraction and Selection
text: In this section, we elaborate on the details of extraction (Q1) and selection of constructions (Q2).

heading: 2.1 Computational Construction Grammar
text: Construction grammar is a branch of cognitive linguistics which assumes that syntax and patterns have specific meanings. A construction is a formmeaning pair whose structure varies in different level of abstractness, including partially or fully filled components and general linguistic patterns. According to CxG, speakers can recognize patterns after coming across them a certain number of times, comparable to merging n-grams (i.e., constructions) at different schematic or abstract levels (Goldberg, 2003, 2006; Tayyar Madabushi et al., 2020). However, the methodology for acquiring the constructions is always labor-intensive and requires careful definition, while the computational generation of CxG is a relatively new research field.
As our answer to (Q1), the usage-based approach is employed to obtain constructions following the work of Dunn (2019). They propose a grammar induction algorithm, which can learn the patterns of constructions with a slot-constraints approach from
1Our code is publicly avaliable at https://github.com/ xlxwalex/HyCxG.
the corpus. Therefore, we adapt Dunn’s (2019) system in our work for construction extraction.
Constructions are represented as a combination of slots and are separated by dashes (Table 1). The slot fillers are drawn from lexical units (words), semantic categories (the discrete semantic domains are clustered with fastText embeddings (Grave et al., 2018) via K-Means) and syntax (Universal Part-of-Speech tags; Petrov et al., 2012), which are progressively higher in abstraction level.

heading: 2.2 Conditional Max Coverage Problem
text: Since constructions are often stored redundantly at different levels of abstractness, overlapping constructions can be captured by the grammar induction algorithm (Dunn, 2017, 2019). We summarize the phenomenon of overlap into two scenarios: Inclusion and Intersection. Inclusion refers to the case where one construction is a subpart or instantiation of another construction, while Intersection indicates that the constructions have some common slots. Redundancy and imbalance encoding problem can be brought by overlap phenomenon. The redundancy of the constructions can cause high computational complexity. And imbalanced construction distribution may introduce interference. This is due to the fact that the words in high dense parts are updated frequently, while this seldom happens in low dense parts. There is a virtual example to illustrate these phenomena in Figure 1.
Therefore, the key issue is to select the discriminative constructions from all candidates (Q2). We formulate this as multi-objective optimization for conditional maximum coverage (CondMC) problem. Specifically, we seek an optimal subset CO = {c1, c2, · · · , cm} from the set C = {c1, c2, · · · , cn} containing all n constructions. The constructions in CO are supposed to reach the following three objectives:
1. The constructions cover as many words as possible in sentences.
2. Intersection among constructions is minimal. 3. Constructions preferably contain more slots
from concrete levels. Objectives 1 and 2 spread the constructions throughout the entire sentence to ensure balanced distribution, while the more discriminative constructions can be selected by objective 3. To unify the objective functions for optimization, we formalize each objective as a specific score function and weight them together to generate the total score. Therefore, Cond-MC can be converted
Algorithm 1: SA algorithm for Cond-MC Input: The set of all constructions C. Output: The optimal set CO.
1 Initialize the feasible solutions CO ← Cf 2 Initialize temperature T ← T0, step k ← 0 3 while k <kmax do 4 T ← COOL(T , k, kmax) 5 Ck ← RANDFLIP(CO, T) 6 dE ← SCORE(Ck) − SCORE(CO) 7 if dE > 0 or RAND() ≤ P (dE, T ) then 8 CO ← Ck 9 end
10 k ← k + 1 11 end 12 return CO
to the problem of maximizing the score for CO. We define each construction ci as a set of r slots ci = {w1, w2, · · · , wr}, while D(·) is utilized to compute the number of elements in a set. Then the score function of objectives 1 and 2 can be formulated as D(c1 ∪ · · · ∪ cm) and D(c1 ∩ · · · ∩ cm), with Sob1 and Sob2 referring to them respectively. And the score function of objective 3 is written as:
Sob3 =
m∑
i=1
r∑
j=1
sr r , sr ∈ {ssyn, ssem, slex} (1)
where three scores ssyn, ssem and slex are assigned for slots in syntax, semantic and lexical level. The score of each construction is calculated by averaging the slot scores, while Sob3 computes the total score of all constructions in CO. Then Cond-MC can be formulated as:
{ max w1Sob1 − w2Sob2 + w3Sob3
s.t. CO ⊂ C (2)
where w1, w2 and w3 are the weighting factors for balancing the scores of three objectives.

heading: 2.3 Solution for Cond-MC Problem
text: As we formulate a multi-objective optimization problem for selecting typical constructions, it is complicated to solve Cond-MC since it is an NP problem. When the solution space is large, i.e., there is a significant amount of constructions in the sentence, which leads to an unacceptable computational complexity. To alleviate this issue, we employ the Simulated Annealing (SA; Kirkpatrick et al., 1983) algorithm in heuristic search of CO, where the procedure is stated in Algorithm 1.
We define CO as a binary set to indicate whether a construction is selected. RANDFLIP(·) is applied to reverse t elements stochastically, while step k is inversely related to temperature T . SCORE(·) is the overall score function in Equation 2. Besides, we utilize RAND() to compare with the transition probability P to estimate whether to accept the new solution under metropolis criterion (i.e., accepting a new solution with a certain probability to increase the perturbation). A cooling schedule is used to control the evolution of the new solution with a reduced temperature function COOL(·) to regulate the temperature in an exponential form.

heading: 3 Model
text: After extracting and determining the optimal set for constructions, the next challenge is to encode constructions in an effective way (Q3). In this section, we first introduce hypergraph for modelling complex data correlation within constructions. Then a relational hypergraph attention network (R-HGAT) is proposed to encode the high-order word interactions inside each construction. Finally, enhanced by the constructional information, the language representation is ready for NLU tasks.

heading: 3.1 Construction Hypergraph Generation
text: Since the associations within constructions are not only dyadic, but could be triadic, tetradic or even higher-order, we adopt hypergraph for modelling such data correlation. Hypergraph is a generalization of conventional graph that the hyperedge can connect arbitrary number of nodes. A hypergraph can be defined as G = {V, E}, which contains a node set V = {v1, v2, · · · , vm} and a hyperedge set E = {e1, e2, · · · , en}. As each hyperedge can connect two or more nodes, the hypergraph G can be denoted by an incidence matrix H ∈ R|V|×|E|, with entries being defined as:
Hij = { 1, if vi ∈ ej 0, if vi /∈ ej (3)
Given a sentence with m words (i.e., nodes in G), we can model high-order word interactions through n typical constructions. In specific, each construction is regarded as a hyperedge, while the words contained in the construction are considered as the member nodes of certain hyperedges. Thus, we can generate the hypergraph for each sentence as illustrated in Figure 1.

heading: 3.2 Hypergraph Attention Network
text: Different from the simple graph that nodes are pairwise linked, the information of nodes cannot be directly aggregated from neighboring nodes in hypergraph. Thus, hypergraph attention network (HGATT) is employed to learn the node representations, which separates the computation procedure into two steps, i.e., node aggregation and hyperedge aggregation. It first aggregates the information of nodes for generating the hyperedge representation. Then the information is fused back to the nodes from hyperedges via hyperedge aggregation. In general, the procedure is formulated as follows:
glj ← AGGRNode ({ hl−1s | ∀vs ∈ ej }) hli ← AGGREdge ( hl−1i , { glj | ∀ej ∈ Ei }) (4)
where AGGRNode and AGGREdge denote the twostep aggregation functions for nodes and hyperedges. hli and g l j denote the representations of node vi and hyperedge ej in l-th layer, while Ei is the set of hyperedges connected to node vi.
The HGATT is mainly implemented based on graph attention mechanism (Veličković et al., 2017), such as HyperGAT (Ding et al., 2020a). However, this attention mechanism applies the same weight matrices for different types of information within hyperedges, which inhibits the ability for the models to capture high-order relations in hypergraphs (Fan et al., 2021).

heading: 3.3 R-HGAT Network
text: To tackle the limitation of HGATT and exploit the information of constructions, we propose a relational hypergraph attention network (R-HGAT). The mutual attention mechanism, i.e., node-level attention and edge-level attention, is adopted to aggregate the information from nodes and hyperedges. The entire architecture is described as follows:
Node-level attention. Node-level attention is utilized to encode representation of hyperedges that aggregates information from the nodes. Given the node vi and the set of hyperedges Ei that connect to vi, we first embed each hyperedge (i.e., construction) into vector space zj by looking up an embedding matrix Ec ∈ R|V|×d, while |V| is the size of construction set within a certain language and d refers to the dimension of the vectors. After that, the attention mechanism is applied to compute the importance score for spotlighting the nodes that
1
1
are crucial to the hyperedge according to zj . Then the aggregation procedure can be formulated as:
glj = ∑
vs∈ej αjsWnh
l−1 s (5)
where Wn is the weight matrix and αjs is the attention score between the representation of node hl−1s and hyperedge zj that can be computed by:
rl−1js = σ((Wczj) TWsh l−1 s )
αjs = exp(rl−1js )∑
vs∈ej exp(r l−1 js )
(6)
where Wc and Ws are trainable matrices, while σ(·) is the non-linear activation function (e.g., ReLU).
In particular, the information of constructions is injected to hyperedges with trainable matrix Wg as:
g ′l j = g l j +Wgzj (7)
Edge-level attention. As an inverse procedure, we fuse the information of the hyperedges back to each node via the edge-level attention. Formally:
hli = ∑
ej∈Ei βijWeg
′l j
tl−1ij = σ((Woh l−1 i ) TWrg ′l j )
βij = exp(tl−1ij )∑
ej∈Ei exp(t l−1 ij )
(8)
where We, Wo and Wr are trainable matrices. βij is the attention score of hyperedge ej on node vi.
After the enhanced representation Hl is acquired with mutual attention mechanism in R-HGAT network, we apply the feed-forward network (FFN)
consisting of two fully-connected layers coupled with residual connections to generate final node representation H ′l, it can be formulated as:
H ′l = LN(Hl−1+σ(W2(W1Hl+b1)+b2)) (9)
where W1 and W2 refer to trainable weight matrices, while b1 and b2 are the biases. Besides, LN(·) denotes the layer normalization operation.

heading: 3.4 Model Training
text: To apply representations to downstream tasks, we utilize average-pooling to obtain the task-specific representation hz , which retains most of the information on node representations. For classification tasks (e.g., sentiment analysis and paraphrase tasks), hz is passed to a fully connected softmax layer, where the objective function to be minimized is the cross-entropy loss. As for regression task (i.e., similarity task), mean squared error loss is adopted as the objective function for optimization.

heading: 4 Experiments

heading: 4.1 Experiments Setup
text: Datasets. Experiments are conducted on a variety of NLU tasks for evaluation. We adopt ABSA datasets from SemEval and MAMS (Pontiki et al., 2014, 2015, 2016; Jiang et al., 2019) as well as GLUE (Wang et al., 2018) benchmark to demonstrate the effectiveness of our HyCxG. GLUE contains a broad range of datasets, including natural language inference (MNLI, QNLI, RTE), sentencelevel sentiment analysis (SST-2), paraphrase and similarity (MRPC, QQP, STS-B), and linguistic ac-
ceptability (CoLA). The detailed statistics of these datasets are provided in Appendix B.
Implementation. To solve Cond-MC problem, we set the weight scores to 1.0, 1.2 and 1.5 for ssyn, ssem and slex (Equation 1), while w1, w2 and w3 are set to 1.0, 0.4 and 0.3 (Equation 2). Meanwhile, the COOL(·) function is formulated as T0 · ak, while a is the factor calculated as − ln(T0/Tf ).
HyCxG is optimized with AdamW (Loshchilov and Hutter, 2018) optimizer. The optimal hyperparameters are selected when the model achieves the highest performances on the development set via grid search. More detailed setups are described in Appendix E, and the pre-trained model weights are obtained from Hugging Face (Wolf et al., 2020).
Baselines. We compare HyCxG with BERTbased state-of-the-art baselines on ABSA datasets: (1) BERT-SPC denotes the fine-tuning BERT with aspect-specific representation. (2) LCFS-ASC (Phan and Ogunbona, 2020) employs the syntactic distance to alleviate the interference from unrelated words. (3) R-GAT (Wang et al., 2020) utilizes a relational graph attention network to encode the pruned dependency tree. (4) KumaGCN (Chen et al., 2020) synthesizes the dependency tree and latent graphs to enhance representation. (5) DGEDT (Tang et al., 2020) provides a dependency enhanced dual-transformer for classification. (6) DualGCN (Li et al., 2021) links the syntactic structure and semantic relevance to generate features. (7) dotGCN (Chen et al., 2022) builds the induced tree with syntactic distances to encode opinion information.
For other NLU tasks from the GLUE benchmark, we compare HyCxG with the base and large versions of the RoBERTa (Liu et al., 2019) model to demonstrate the performance improvement.

heading: 4.2 Experimental Results
text: Results on ABSA tasks. Experiments on ABSA tasks are conducted firstly, which are intuitive to analyze the function of constructions with aspects. Meanwhile, the results can be extended to multilingual settings as well as accessing to case studies. The model performances are shown in Table 2, from which several observations can be obtained.
First, compared to BERT-SPC model, most of syntax-based baselines achieve higher performances on five datasets, since they can alleviate interference introduced and build the relationship between aspects and their corresponding opinion words via synthesizing syntactic knowledge (e.g., dependency tree). Second, the baseline models (i.e., DualGCN and dotGCN) with more information, such as semantic and opinion representations, gain better performances than others.
Third, our HyCxG significantly outperforms all baselines with constructional information incorporated, which demonstrates the effectiveness of HyCxG. Furthermore, the perspective that construction grammar can contribute to sentence comprehension for NLU tasks can also be substantiated.
Results on GLUE tasks. As shown in Table 3, we conduct performance comparison among HyCxG with the base (B) and large (L) version of RoBERTa on GLUE development sets. And the results on test set are reported via online leaderboard.
The average results over tasks illustrate that our HyCxGB outperforms RoBERTaB, and HyCxGL is also better than RoBERTaL on both development and test sets. It indicates the validity of HyCxG and the benefits of constructional information.
As a branch of grammar, HyCxG has huge improvement on CoLA task, which shows its poten-
tial capability for evaluating linguistic acceptability. Meanwhile, HyCxG also significantly outperforms RoBERTa on RTE and MRPC tasks. Especially on the MRPC task, HyCxGB even surpasses RoBERTaL, which further validates the language representation can be enhanced by constructional information. Besides, HyCxG also boosts the performances on all other NLU tasks as well.

heading: 4.3 Comparative Analysis
text: Ablation study on model complexity. Though the efficiency of our HyCxG model is illustrated from the results on NLU tasks, we conduct ablation study to further investigate whether the performance enhancement is due to an increase in model complexity. We implement OML and OLL+HyCxG to address this concern. As increasing the depth of the PLMs can improve performance, OML adds an additional transformer layer on BERT, while OLL+HyCxG removes the last layer of BERT in HyCxG. Besides, we continue pre-training OML model (OML+Pretrain) on the massive corpus to achieve its optimal performance.
As shown in Table 4, OML achieves higher re-
sults than BERT-SPC, indicating that more complicated models tend to yield better performance. OML+Pretrain performs better compared to OML, which indicates the effect of pre-training. However, OLL+HyCxG surpasses OML with lack of two transformer layers, while it also overtakes OML+Pretrain on Rest 14 and 16 datasets. Meanwhile, HyCxG significantly outperforms OML and OML+Pretrain. As the details of computation complexity comparison between HyCxG and other baselines shown in Appendix D, the results illustrate that HyCxG achieves the highest performance at a relatively low computational complexity. These evidences all prove the efficacy of HyCxG and constructional information. The ablation study on GLUE benchmark is conducted in Appendix G.
Comparison of construction selection strategies. In section 2, we formulate the Cond-MC problem for selecting typical constructions. Table 5 illustrates the performances for different strategies.
The strategy of w/o Cond denotes that constructions cannot overlap with each other, while w/o Selection employs all constructions. As redundancy and imbalance can occur in w/o Selection and w/o
Cond, w/o Cond performs better on Rest 15 and 16 datasets, while fails on Rest 14 and Lap 14 datasets compared to w/o Selection. In contrast, Cond-MC strategy consistently outperforms w/o Selection and w/o Cond on all datasets. The results illustrate the necessity of selecting typical constructions and the validity of Cond-MC strategy.
Multilingual results. Since construction grammar is applicable to multilingual analysis and can be learned via a unified framework for different languages (Dunn, 2022), we can further discuss the performance improvements of our HyCxG model in multilingual settings. The ABSA datasets in SemEval 2016 (Pontiki et al., 2016) with multiple languages are conducted for our experiments (statistics are shown in Table 8). Meanwhile, we also adapt baselines to multilingual environment with their official implementations. For models that involve dependency tree parsing (e.g., R-GAT, DualGCN), we maintain the same parsing tools that they employ (spaCy or Stanford Parser (Manning et al., 2014)). The results are shown in Table 6. We can observe that the inclusion of syntactic information can also improve model performances on multilingual settings compared to BERT-SPC. DualGCN has relatively high performance on different languages, since it incorporates the semantic features via SemGCN to the syntactic foundation with regularizers. Moreover, HyCxG outperforms these baselines across all four languages, indicating that constructional information can also enhance semantic understanding for other languages.
More empirical studies. We conduct experiments to investigate the potential capacity of HyCxG. First, experiments in Appendix F demonstrate the superior ability of HyCxG in perceiving statements and patterns. Second, we conduct ablation study in Appendix I to examine the benefits for each component in HyCxG. We also investigate the results on colloquial datasets in Appendix H.
As the constructions are derived from a formal language corpus, which has a different register (Dunn, 2023) than the colloquial dataset, some of the edge constructions are not extracted to enhance the representation, resulting in suboptimal performances. It illustrates the necessity of obtaining constructions on a diverse corpus. Besides, we present an approach in Appendix J to build the construction network with the representations of constructions acquired in HyCxG. It depicts the inheritance relations between constructions, which provides meaningful implications to usage-based CxG.

heading: 4.4 Case Study
text: The case study is utilized to illustrate the mechanism inherent in HyCxG for NLU tasks. We first visualize the learned representations via tSNE (Van der Maaten and Hinton, 2008) in Figure 2, while the representations of these five randomly selected constructions are obtained via applying average-pooling in BERT-SPC and HyCxG. The results demonstrate that the representations in HyCxG form clusters with distinct boundaries, while the representations in BERT-SPC are diffuse among different clusters. It proves that HyCxG can model higher-order interactions and synthesize constructional information.
Meanwhile, we visualize the attention scores to illustrate the benefits of constructional information in Figure 3. As discussed in Section 1, BERTSPC mainly focuses on the opinion word “friendly”, leading to misclassification. In contrast, HyCxG can produce correct prediction, since it captures the modality pattern (i.e., NOUN–AUX–be).

heading: 5 Related Work
text: Applications of CxG in NLP. CxG theories have been explored to NLP applications. Doubleday et al. (2017) exploit embodied CxG to parse the text of requests from users, while Nevens et al. (2019) employ fluid CxG for semantic parsing in
BERT-SPC
# Positive
CxG: [NOUN AUX be]
! "
visual question answering. Besides, a knowledge network is generated (Kiselev., 2020) with CxG on winograd schema challenge. These approaches all rely on hand-made definition with prior knowledge and are not compatible with PLMs, thus we employ the usage-based CxG in our work. However, we observe the problems of redundancy and imbalance in usage-based approach, which motivates us to formulate the conditional max coverage problem to acquire discriminative constructions.
Probing CxG in PLMs. Recent work has conducted empirical studies to probe whether PLMs acquire constructions. CxGBERT (Tayyar Madabushi et al., 2020) finds that constructional information is accessed by BERT but cannot be exploited for lack of proper methods. CxLM (Tseng et al., 2022) shows that while PLMs are aware of the construction, they are confused at the variable slots. Weissweiler et al. (2022) discover that PLMs can recognize the comparative correlative constructions but fail to utilize their meaning. Furthermore, Weissweiler et al. (2023) argue that fine-tuning on downstream tasks necessitates explicit access to constructional information. Our work is the first attempt to exploit the constructional information for representation enhancement on NLU tasks.
Hypergraph neural networks. Recent research has extended conventional graph to hypergraph (Feng et al., 2019; Ding et al., 2020a; Cai et al., 2022), which can model high-order data correlation. However, when the networks ignore the representation of hyperedges, the capability of capturing higher-order relationships between nodes can be inhibited (Fan et al., 2021). It motivates us to propose R-HGAT to encode constructional information.
Aspect-based sentiment analysis. There has been much work on syntax-based methods (Wang et al., 2020; Tang et al., 2020; Phan and Ogunbona, 2020; Li et al., 2021; Tang et al., 2022; Xu et al., 2023), which establish syntactic connections between each aspect and their corresponding opinion
words via dependency and constituency parsing.

heading: 6 Conclusion
text: In this work, we introduce usage-based construction grammar (CxG) to enrich language representation with constructional information. Then a hypergraph framework of CxG named HyCxG is proposed to integrate constructional information for language representation enhancement. In HyCxG, we extract constructions through a slot-constraint method, and formulate the conditional max coverage problem for selecting the discriminative constructions. Then we propose a relational hypergraph attention network to encode high-order word interactions within constructions. Extensive experiments illustrate the validity of our HyCxG on natural language understanding tasks.
Limitations
In this study, the limitations can be summarized into two major aspects:
(1) The usage-based approach (Dunn, 2017, 2019) being employed in our work has the ability to extract most of constructions, while a small portion of non-contiguous constructions (e.g., comparative correlative constructions) are neglected. These noncontiguous constructions are probably fragmented into multiple independent constructions. We will investigate the approaches to capture these noncontiguous constructions via incorporating more syntactic knowledge in future work for language representation enhancement.
(2) As discussed in Appendix H, the performances are not significant improved on the tasks that contain a large amount of colloquial expressions. Since our constructions are mainly learned from the formal corpus, which has less colloquial expressions. It causes fewer constructions to be accessed in these tasks, which encourages us to learn constructions in more diverse corpus to enhance the language representation for natural language understanding tasks in the future.

heading: Acknowledgements
text: We thank all anonymous reviewers for their insightful comments and suggestions. And we also appreciate High-Flyer for providing us with computational cards for pre-training. This research is supported by the Science and Technology Project of Zhejiang Province (2022C01044) and the National Natural Science Foundation of China (51775496).

heading: A Background
text: A.1 Construction Grammar Construction Grammar (CxG) theory is a branch of cognitive linguistics. It assumes that grammar is a meaningful continuum of lexicon, morphology and syntax rather than solely relying on a system composed of stable but arbitrary rules for generating well-formed sequences. Therefore, constructions (i.e., symbolic elements that connect a certain morphosyntactic form to a meaning) are thought to be the primary objects of grammatical description (Langacker, 1987; Goldberg, 1995, 2006). Constructions can be defined as linguistic patterns that store different form and meaning pairs. They cannot be strictly predictable by their components. (Goldberg, 1995; Langacker, 2005). Besides, patterns that occur with sufficient frequency can also be considered as constructions (Goldberg, 2006).
In terms of the form for CxG, their syntactic structure varies in level of abstractness, including partially or fully filled components (e.g., idioms), and general linguistic patterns (Goldberg, 2003, 2006; Dunn and Wong, 2022). In another perspective, each construction has immutable components and unfilled slots that the element in the slots can be altered with certain words to boost productivity.
Meanwhile, constructions can also be regarded as linguistic knowledge. This knowledge is based on cognitive results and the product of empirical generalization and abstraction. In the evolution of human language experience, the knowledge is formed, acquired and applied by our human (Goldberg, 2003; Hilpert, 2014). Analogy to Knowledge Graph (KG), the language systems are considered as a network of constructions (Goldberg, 1995). This network consists of two fundamental components: (1) nodes in the graph (i.e., specific constructions). (2) the edges between nodes. These edges refer to diverse relationships between constructions (e.g., polysemy link, metaphorical extension and instance link). It is worth mentioning that there may exist multiple connections between constructions in the network.
A.2 Usage-based Construction Grammar The CxG paradigm has already developed a variety of implementations, including formal approaches, i.e., fluid construction grammar (FCG), embodied construction grammar (ECG) and sign-based construction grammar (SBCG) as well as usagebased approaches. However, the FCG, ECG and
SBCG rely heavily on hand-made definition with prior knowledge (Steels and de Beule, 2006; Goldberg et al., 2005; Rambelli et al., 2019). Though these methods can provide high-quality representations, they cannot automatically mine the constructions from the data with the emergence of slotconstraints (Dunn, 2017, 2019).
We follow the efforts of Dunn (2019) that propose a computationally slot-constraints approach for CxG extraction via the data-driven pipeline. First, three types of slots (i.e., lexical, syntactic and joint semantic-syntactic) are defined according to the different levels of abstraction. Note that the syntactic slots refer to the POS tags, while the semantic-syntactic slots are derived from word embeddings that are clustered in discrete semantic domains based on K-Means algorithm (Dunn and Wong, 2022). Secondly, they generate the potential construction templates from the corpus. Then the statistics of frequency and association strength are calculated for pruning templates. Finally, the tabu search is applied to determine the optimal set of constructions with Minimum Description Length (MDL) metric (Dunn, 2019). Based on this pipeline, they develop the c2xg2 toolkit.

heading: B Data Statistics
text: We conduct experiments on a variety of natural language understanding tasks in main experiments, including eight tasks from the GLUE benchmark (Wang et al., 2018) and five aspect-level sentiment analysis tasks (Pontiki et al., 2014, 2015, 2016; Jiang et al., 2019). Table 7 presents the
2https://pypi.org/project/c2xg/
data statistics for GLUE benchmark tasks. We divide the GLUE benchmark tasks into two categories: single-sentence task and sentence-pair task. In single-sentence task, the language acceptability task (CoLA) and the sentence-level sentiment task (SST-2) are included. As for sentence-pair task, it contains similarity task (STS-B), paraphrasing tasks (MRPC, QQP) and several natural language inference (NLI) tasks (MNLI, QNLI and RTE). All tasks except STS-B are classification tasks, while STS-B is a regression task.
The statistics for aspect-based sentiment analysis datasets are shown in Table 8. They are all annotated with aspects and corresponding polarities (i.e., positive, neutral or negative). In multilingual settings, we adopt four different language datasets from SemEval 2016 (Pontiki et al., 2016). Meanwhile, Twitter (Dong et al., 2014) and GermEval (Wojatzki et al., 2017) datasets, which collect tweets and messages from social media, are employed in the colloquial expression experiments.

heading: C Learning and Alignment of CxG
text: In order to utilize constructional information, the inventory of construction grammar needs to be learned at first. As we present a brief description of construction learning procedure in Section A.2, prior work has shown that construction grammars can converge on stable representations with sufficient training data (Dunn, 2022). Therefore, the construction grammar in English is learned from sampling of multi-source corpus (WIKIPEDIA, BOOKCORPUS and CC-NEWS) (Devlin et al., 2019; Liu et al., 2019) which contains about 1,200 million words. As for other languages utilized in multilingual experiments, constructions are learned from sampling the multilingual portion of the C4 corpus (Raffel et al., 2020). Since the constructions on a variety of languages are learned in c2xg toolkit, we validate that our HyCxG can achieve comparable performances on NLU tasks with the construction grammar list in c2xg.
After obtaining the construction grammar list, we are able to derive all possible constructions from a given sentence via c2xg toolkit. However, the toolkit can only detect which construction patterns are present in the sentences and cannot provide specific position indexes of them. Besides, there are discrepancies between c2xg and pre-trained language models in tokenization procedure. The c2xg toolkit utilizes the basic white-space tokenizer, while pre-trained language models adopt more complex algorithms (e.g., WordPiece (Devlin et al., 2019), SentencePiece (Kudo and Richardson, 2018)). Therefore, we have to align the position indexes of the constructions under different tokenization algorithms with specific mapping function.
To tackle these imperatives, we develop a wrapper for c2xg. It can be adapted to different tokenization methods with the output of the start and end position index for construction spans. We hope that it can facilitate future research on construction grammar in natural language processing tasks.

heading: D Analysis of Computational Complexity
text: In Table 4, we demonstrate that the effectiveness of our model does not result from the increased complexity of the model. As shown in Table 9, we further analyze the computational complexity of the models via DEEPSPEED to illustrate more intuitive results. The number of model parameters and the multiply–accumulate operations (MACs) are utilized to compare complexity on aspect-based
sentiment analysis models. To ensure a fair comparison, we unify the public hyper-parameters of all baselines while preserving other unique hyperparameters consistent with their official implementations. Besides, the additional embedding parameters for constructions in HyCxG are not counted. Then three observations can be derived.
First, LCFS-ASC is significantly more complex than other models due to its dual BERT architecture, while other models all use a single PLM. Second, adding an additional transformer layer to BERT (OML) indeed improves performances but is far more complex than our HyCxG. However, our model outperforms OML, which proves the improvement of our HyCxG is not just due to the increased complexity of the model structure. Third, our model has lower computational cost and memory consumption than most of baselines. Meanwhile, our HyCxG achieves even better performances without dependency information injection, which empirically validates the effectiveness of constructional information.

heading: E Hyper-parameters and Settings
text: For fine-tuning our HyCxG on GLUE and aspectbased sentiment analysis datasets, we search for the optimal task-specific hyper-parameters with the range of values in Table 10 (We implement our model in PYTORCH and use GeForce RTX 3090 devices for experiments):
In order to pre-train OML models in Table 4 and 12, we follow the same pre-training procedure of BERT and RoBERTa. OML models are implemented in PYTORCH and pre-trained on 8 × Tesla A100 devices. We employ the officially released PLM checkpoints from Hugging Face3 to initial-
3https://huggingface.co/models
ize basic parameters (12 layers) in OML, while the parameters in additional transformer layer are initialized to the average of the preceding 12 layers. Besides, we set the batch size to 64 per GPU device, and the training steps to 400,000 during pretraining procedure with AdamW optimizer (learning rate is 5e-5 and warmup ratio is 0.1) and the corpus of WIKIPEDIA and BOOKCORPUS.

heading: F Pattern Recognition Capability of CxG
text: In this work, we demonstrate that the construction grammar is capable of improving natural language understanding via infusing constructional information. However, can CxG only serve to enhance language representation? We explore more application scenarios for construction grammar based on this concern and a viable scene is pattern recognition. The identification of this scenario can be viewed as an investigation into whether the text contains particular specific structures (e.g., negation and counterfactual expression). We take counterfactual detection task as an example to verify the feasibility of the scenario.
The counterfactual dataset in SemEval 2020 (Yang et al., 2020) are adopted in this experiment with train (13,000 instances) and test (7,000 instances) sets. We employ Precision, Recall and F1 score as the metrics to evaluate the performance. To better evaluate our method, we compare the performances with the best system in SemEval.
HIT (Ding et al., 2020b) is the state-of-the-art method that employs the ensemble model (combining the large version of BERT, RoBERTa and XLNet via weighted average for their probability predictions) with pseudo-labeling (Lee et al., 2013) strategy and 10-fold cross-validation. Table 11 shows the results and the above system is marked as HIT+PL. HIT is the single model of RoBERTalarge without pseudo-labeling. Our HyCxG model is also trained with 10-fold cross-validation based on RoBERTa-large, while HyCxG+PL incorporates pseudo-labeling strategy to HyCxG.
Surprisingly, we observe that our HyCxG significantly outperforms HIT system by 4.47 F1 points. Even for HIT+PL (applying ensemble and pseudolabeling), there is still a massive gap with HyCxG. Furthermore, our HyCxG+PL also has a 3.82 F1 points boost compared to HIT+PL. It endorses the ability of our model in pattern recognition.
As discussed by Yang et al. (2020), the main challenge in counterfactual detection task is that existing models focus excessively on token level features while neglecting to understand statements. Most counterfactual cases are expressed with subjunctive mood to convey wishes, suggestions and demands. There is a practical example “if I were asked to, I would be happy to talk to anyone.”, which is misclassified by HIT model, while HyCxG can predict correctly. It is a counterfactual expression that regards the part guided by the conjunction “if ” as the antecedent. Based on such hypothesis, the possible consequent is stated. In this sentence, our system can extract two critical construction “if –PRON–were–VERB” → “if I were asked” and “AUX–be–ADJ”→ “would be happy” for capturing the counterfactual statements. Thus, HyCxG is capable of recognizing specific patterns via encoding constructional information.
Similarly, negation detection is also a recognition task relying on a series of special patterns. There are more application scenarios of CxG, which we leave those discussions to future work.

heading: G Ablation Study on GLUE Datasets
text: Similar to the ablation study on ABSA tasks (Table 4), we conduct experiments on GLUE benchmark with four datasets (CoLA, SST-2, RTE and MRPC). As discussed previously, the PLM (here is RoBERTa-base) with an additional layer on top is called OML. OML+Pretrain performs an extra pretraining procedure on OML (details of pre-training are shown in Section E). OLL+HyCxG is the model that removes the last transformer layer from PLM in our HyCxG model. For the results in Table 12, OML+Pretrain always outperforms OML on four tasks, which illustrates the utility of pre-training procedure. Besides, our OLL+HyCxG achieve better performances compared to OML, even though it has two fewer transformer layers. It is worth mentioning that the result of OLL+HyCxG is even higher than OML+Pretrain on SST-2 dataset.
Moreover, HyCxG performs better than OML and OML+Pretrain, while it has a smaller number of parameters with lower complexity. These results indicate the benefits of constructional information in natural language understanding tasks and high cost performance of our HyCxG.

heading: H Colloquial Expression Results
text: As we discuss in Limitations, the performance improvement of HyCxG is not significant for datasets with informalized expressions (i.e., colloquial expressions). In this work, constructions are learned from the standard corpus (e.g., WIKIPEDIA and BOOKCORPUS) with formal language. This causes some of edge constructions not being extracted for language representation enhancement. We conduct experiments on Twitter (Dong et al., 2014) and GermEval (Wojatzki et al., 2017) datasets to investigate the issue for English and German. These datasets are collected from social media which consist of
numerous colloquial sentences (statistics are shown in Table 8). As shown in Table 13, the results on Twitter are retrieved from the papers of baselines. Then we adapt all the models with multilingual settings (Section 4.3) to the GermEval dataset. From the results, we observe the performance of our HyCxG on Twitter is only in the middle compared to baselines. As for GermEval, HyCxG exceeds most of models but also falls below DualGCN.
To further analyze the evidence for supporting our conclusion, two metrics are proposed to evaluate the sparsity of constructions: average ratio of construction (AoC) and average coverage ratio (ACR). We define AoC as the average proportion of extracted constructions in each sentence, while ACR means the average ratio for the total length of the constructions to sequence length in each sentence under maximum coverage (Section 2.2). Metrics can also be formulated as follows:
AoC = 1
N
N∑
i=1
CxG (si)
L (si)
ACR = 1
N
N∑
i=1
∑M j=1 L (cj) L (si) , cj ∈ MC(si)
(10) where N is the number of sentences in each dataset while si is the i-th sentence. L(·) is the function, that computes the length of the input and CxG(·) is employed to calculate the total number of constructions in a sentence. Besides, MC(·) refers to the set of constructions in the sentence that satisfy the maximum coverage (w/o condition), while M is the size of the set and cj is the j-th construction.
We compare these metrics between formal group (Rest14, Lap14) and colloquial group (Twitter, GermEval) and the results are shown in Figure 4. There is a huge gap, where colloquial group is significantly lower than formal group on AoC and ACR. It demonstrates that constructions are much sparser
in Twitter and GermEval. In such circumstances, constructional information learned by our model has been inadequate so far to enhance the representation. For this limitation in our work, we believe that it can be improved via adding more colloquial corpus to the construction learning (Section C), which requires further investigations in the future.

heading: I Ablation Study on Model Components
text: To further investigate the influence of different components in our HyCxG, we conduct extensive ablation studies. As shown in Table 14, w/o HGATT and FFN indicate that we temporarily remove each of these components from our model. The experimental results illustrate the importance of the HGATT network, since its removal can cause significant performance degradation. Meanwhile, it also demonstrates the validity of injecting constructional information for language representation enhancement. Overall, our HyCxG with all components can achieve the highest performance.

heading: J Network of Constructions
text: Construction grammar is not an unordered set in a language, but rather a network linked by inheritance relations. The whole network of constructions in a particular language is called the Constructicon (Evans, 2007). As briefly introduced in Section A.1, Constructicon consists of nodes (constructions) and edges (inheritance relations) (Goldberg, 2003). There are four types of relations between constructions: polysemy link (IP ), subpart link (IS), instance link (II ) and metaphorical extension (IM ). Previous efforts are taken by linguists using case studies to manually build the networks (Boogaart et al., 2014). To the best of our knowledge, there is no attempt to construct Constructicon from the perspective of distributional representation. In this work, we can acquire representation of each construction via the embedding matrix in HyCxG. Therefore, our HyCxG can contribute to linguistics in turn to provide interpretability for the network of constructions.
Algorithm 2: Compute MD between constructions
Input: The set of constructions G. Output: MD matrix for G.
1 G = [g1, g2, · · · , gn] 2 Initialize MD ∈ Rn×n matrix with 0 3 for i = 0 to n− 1 do 4 for j = 0 to n− 1 do 5 if i = j then 6 MD[i,j]←0 7 else 8 MD[i,j]←Compute_md(gi, gj) 9 end
10 end 11 end 12 return MD 13 14 Function Compute_md(A, B): 15 A = [c1, c2, · · · , cp], B = [e1, e2, · · · , eq] 16 Initialize dp ∈ R(p+1)×(q+1) matrix with 0 17 for i = 1 to p+ 1 do 18 for j = 1 to q + 1 do 19 if Is_Match(A[i-1], B[j-1]) then 20 d←0 21 else 22 d←1 23 end 24 dp←min(dp[i-1][j]+1,dp[i][j1]+1, dp[i-1][j-1]+d) 25 end 26 end 27 dist← dp[m][n] 28 return dist 29 end function
We discuss three types of inheritance relationships (IP , IS and II ) in this work, while metaphorical extension is more abstract and complex. IP characterizes the semantic relationship between a specific meaning of a construction and its extended meaning. IS is referred to the connection that a construction is an independent subpart of another construction. Besides, when a concrete construction is a special instance of another construction, the relationship is defined as II . For the purpose of modeling inheritance relationships, we measure the distances between different constructions from semantic and morphological perspectives. Then we weight the sum of the two distances and the top-k closest constructions are selected for each construction to determine the relationships. Eventually, the network of constructions is generated.
Since the representation of each construction is obtained, semantic distance (SD) can be com-
puted via cosine similarity. Meanwhile, we propose multi-level edit distance (MED) algorithm to evaluate the morphological distance (MD) between two constructions with different abstract level. In this setting, semantic distance can provide evidence for polysemy link, while morphological distance constrains variation in structures for subpart link and instance link. Thus, these distances are complementary on modeling inheritance relations. Under definition, semantic distance can be formulated as:
SD (gi, gj) = 1− −→vgi · −→vgj ∥−→vgj∥∥−→vgj∥
(11)
where −→vgi and −→vgi are the embedding vector of construction gi and gj , which can be looked up from construction embedding matrix Ec, respectively.
MED is proposed to compute the morphological distances. As slots in constructions are defined for different levels (Lexical, Syntactic and Semantic) of abstraction, we need to match slots across the levels. We define a function Is_Match(·, ·) to indicate whether two slots match or not. The matching criteria can be summarized as follows:
• (Lexical↔Lexical) Match when the slots are identical with the same POS tags or belong to common semantic cluster.
• (Syntactic↔Syntactic, Semantic↔Semantic) Match if and only if the slots are identical.
• (Lexical↔Syntactic) Match when the slot of lexical has the same POS tags as the other.
• (Lexical↔Semantic) Match when the semantic cluster of the lexical slot is identical to the slot in semantic level.
Based on such matching strategy, MED algorithm is illustrated in Algorithm 2. After obtaining semantic and morphological distances, we synthesize them to compute the construction distance.
Then the top-k closest constructions for each construction can be selected as the potential inheritance set. It can be derived as:
{Gg ⊂ G|argmin |Gg |=k (γ SDg +MDg)} (12)
where Gg is the potential inheritance set for construction g, while G is the set of all constructions. SDg and MDg refer to semantic and morphological distances between g and the other constructions in G, respectively. γ is the coupling co-efficiency that regulates the two distances. Then we can determine the inheritance relationship within the set based on the definition of IP , IS and II .
In order to analyze Constructicon, the embedding matrix Ec of constructions is extracted after training on CoLA task as an example. We set k to 15 and γ to 10 in Equation 12. In such settings, we are able to illustrate the effectiveness of our approach with practical examples for each inheritance relationship. As shown in Table 15, the relationship between “think–PRON–AUX”, “believe–PRONAUX” and “know–PRON–AUX” belongs to polysemous inheritance. They can all be considered as instantiations of construction “VERB–PRON–AUX”. Thus, the subjective belief that someone has done or can do something is the primary meaning of this type of constructions. After that, different verbs are filled in the slots to inject expanded meanings. As for subpart inheritance, “SCONJ–PRON–AUX” is the smallest independent construction in the example. It is embodied in “SCONJ–PRON–AUX– VERB”, while “SCONJ–PRON–AUX–VERB” is also a subpart of “SCONJ–PRON–AUX–VERB– to”. The instance inheritance establishes the con-
nection between the construction and its completely or partially filled constructions . “if –PRON–can” is a specific construction that fills “if ” and “can” into “SCONJ–PRON–AUX” as a substitute for “SCONJ” and “AUX”. Besides, it is worth mentioning that IP is undirected relationship, while IS and II are directed relationships in our Constructicon.
Furthermore, we develop a visualization toolkit to demonstrate the network of constructions. As shown in Figure 5, a sub-network that depicts the three inheritance relationships is constructed as an example under our settings.

heading: K Detailed Results on ABSA Tasks
text: Since all four ABSA datasets (Rest14, Lap14, Rest15 and Rest16) of SemEval do not have official validation sets, randomness may be introduced for performance comparison. Therefore, we randomly divide the training set into a new training set and a validation set with the ratio of 9:1 for these ABSA datasets. Then we conduct the experiments with baseline models and our HyCxG. Meanwhile, the hyper-parameter searching is conducted for each model for fair comparison.
As shown in Table 16, we can observe that the performance comparison of the model is almost consistent with that in Table 2. First, the baseline models that inject syntactic information always achieve better performances than BERT-SPC. Second, R-GAT outperforms the other baseline models on the Rest14 and Rest15 datasets, while KumaGCN and DualGCN have better performance on Lap14 and Rest16, respectively. Third, HyCxG outperforms all baseline models with constructional information incorporated on these datasets, which illustrates the validity of HyCxG. Meanwhile, it can also substantiate that construction grammar can be regarded as the inductive bias to enhance the language representation in NLU tasks.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations are presented in a standalone Limitation section (after the Conclusion and before the References).
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Main claims are stated in Abstract and Introduction (Section 1).
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Scientific artifacts are illustrated in Section 2 and 3.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Citations are all shown in the References and footnotes.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? The information of artifacts is shown in Section B of the Appendix.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. The statistics are included in Section B of the Appendix.
C 3 Did you run computational experiments? The computational experiments are included in Section 4 and Section D, F, G, H, I, K in the Appendix.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? The computational statistics are illustrated in Section D of the Appendix.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? The experimental setup is shown in Section 4.1 and Section E of the Appendix.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? The descriptive statistics are included in Section 4.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? The existing packages in our work are described in Section 4 and Appendix F.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: For this limitation in our work, we believe that it can be improved via adding more colloquial corpus to the construction learning (Section C), which requires further investigations in the future