StructSP: Efficient Fine-tuning of Task-Oriented Dialog System by Using Structure-aware Boosting and Grammar Constraints
Year: 2023

Abstract:
We have investigated methods utilizing hierarchical structure information representation in the semantic parsing task and have devised a method that reinforces the semantic awareness of a pre-trained language model via a two-step fine-tuning mechanism: hierarchical structure information strengthening and a final specific task. The model used is better than existing ones at learning the contextual representations of utterances embedded within its hierarchical semantic structure and thereby improves system performance. In addition, we created a mechanism using inductive grammar to dynamically prune the unpromising directions in the semantic structure parsing process. Finally, through experiments1 on the TOP and TOPv2 (low-resource setting) datasets, we achieved state-of-the-art (SOTA) performance, confirming the effectiveness of our proposed model.

heading: 1 Introduction
text: Task-oriented dialog systems are computer systems designed to perform a specific task (Bai et al., 2022). These systems have a wide range of applications in modern business (Zhang et al., 2020b) and daily lives (Yan et al., 2017), and so on. The semantic parsing model at the core of these systems plays an essential role in converting user utterances into machine-understandable representations for use in capturing the semantic meaning and returning appropriate responses. The introduction of hierarchical representation by Gupta et al. (2018) demonstrated the importance of nested sublogic composition in a task-oriented dialog system. Although this representation is flexible enough to capture the meaning of complicated queries, it also challenges models to identify labels and select the
*Equal contribution +Corresponding author 1 The source code of this work is released at https://
github.com/truongdo619/StructSP
corresponding spans of constituent semantics in the natural sentence.
Pre-trained language models (e.g., BERT (Devlin et al., 2019)) have recently achieved impressive results on many tasks, including semantic parsing (Ziai, 2019; Herzig and Berant, 2021; Rubin and Berant, 2021). Among them, the Recursive INsertion-based Encoding (RINE) model (Mansimov and Zhang, 2022) produces SOTA results by using recursive insertion-based mechanism that directly injects the decoded tree from the previous steps as input to the current step. Although many semantic parsing systems have shown impressive results, there is still much room for improvement in their overall performance.
A promising way to improve the performance of a task-oriented dialog system is to use a pretrained language model based on the characteristics of this task. The language models is pre-trained on large-scale unstructured text, which means that it does not explicitly learn the logical structure of any sentences. Since information about sentence structures is crucial in the semantic parsing task. Besides, the hierarchical semantic representation in a task-oriented dialog system usually follows a grammar based on a specific task or domain. For
10206
example, the parent node of slot SL:NAME_EVENT is always the intent IN:GET_EVENT, etc. However, existing methods ignore this information (Mansimov and Zhang, 2022). We thus focused on making two improvements: strengthening the hierarchical structure awareness of the pre-trained language model and using dynamically pruning the unpromising decoding directions by using inductive grammar (Figure 2).
In this paper, we introduce StructSP, a framework for effectively embedding the structured logic information of utterances into a pre-trained language model and thereby enhancing the performance of the semantic parsing system. Our method is generalized and can be adapted to any logical structure, such as a logical form, by using λcalculus or Prolog syntax (Dong and Lapata, 2016). In particular, we exploit the hierarchical representation, a deep fine-grained structure widely used in semantic parsing (Zhao et al., 2022b; Desai and Aly, 2021; Louvan and Magnini, 2020). Our method consists of two phases: structure-aware boosting and grammar-based RINE. The structure-aware boosting phase aims to enhance the structured information of the pre-trained language models and includes two subtasks. The first is structurefocused mask language modeling (MLM), which extends the standard MLM task (Devlin et al., 2019) by focusing more on the logical units in the linearized hierarchical representation. The second subtask is relative tree agreement where relative trees are the trees parsed in the middle steps of the parsing process (Figure 1). This subtask enables the model to represent the relative trees with closer hidden vectors and learn the hidden relationship
between them. The second phase, grammar-based RINE uses the grammar extracted from annotated data to support node label prediction. Incorporating structure information into the model enabled our StructSP to achieve an exact match score on the TOP dataset 0.61 points better than with a SOTA method and demonstrated similar potential results on the TOPv2 datasets (low-resource setting).
This work makes three contributions: (1) an effective fine-tuning approach is introduced for incorporating hierarchical semantic structured information into a pre-trained language model; (2) the use of grammar is introduced in the parsing process to reduce unpromising node label prediction; (3) the StructSP framework is shown to outperform existing models in task-oriented semantic parsing on two datasets, TOP and TOPv2.

heading: 2 Related Work
text: Overview Task-oriented parsing (TOP) (Gupta et al., 2018) and its variant TOPv2 (Chen et al., 2020) were created as benchmarks for assessing the performance of task-oriented semantic parsing models. Various approaches have been proposed for tackling the semantic parsing task on these datasets. Zhu et al. (2020) introduced a non-autoregressive sequence-to-sequence semantic parsing model, which is based on the Insertion Transformer model (Stern et al., 2019). Aghajanyan et al. (2020) introduced an extension of the hierarchical representation, “decoupled representation”, and used sequence-to-sequence (S2S) models based on the pointer generator architecture to parse this representation. Rongali et al. (2020) introduced a unified architecture based on S2S mod-
els and a pointer-generator network for semantic parsing. Einolghozati et al. (2019) introduced a shift-reduce parser based on recurrent neural network grammars (RNNGs) with three improvements to the base RNNG model: incorporating contextualized embeddings, ensembling, and pairwise reranking based on a language model. Additionally, Zhao et al. (2022a) showed that compositional TOP could be formulated as abstractive question answering (QA), where the parse tree nodes are generated by posing queries to a QA model. The RINE model (Mansimov and Zhang, 2022) splits the work of parsing an utterance into multiple steps, where the input of each step is the output of the previous step. However, even SOTA methods ignored the hierarchical structure information during the parsing process. We thus focused on utilizing the hierarchical structure information by using inductive grammar extracted from annotated data to guide the node label predictions.
Pre-trained Language Model Adaptation Several recent studies have demonstrated the value of adapting pre-trained language models to specific tasks with different training objectives, such as in summarization (Zhang et al., 2020a) and knowledge inference (Sun et al., 2019; Liu et al., 2020). In the realm of semantic parsing, the SCORE pretraining method (Yu et al., 2020b) focuses on inducing representations that capture the alignment between dialogue flow and structural context in conversational semantic parsing tasks. Grappa (Yu et al., 2020a) is a pre-training approach designed for table semantic parsing and seeks to learn an inductive compositional bias through the joint representation of textual and tabular data. Bai et al. (2022) introduced a semantic-based pre-training approach that uses abstract meaning representation (AMR) as explicit semantic knowledge to capture the core semantic information of utterances in dialogues. In comparison, to strengthening hierarchical structure information, we continue training the model by using only annotated data for the finetuning task instead of generating (or augmenting) a large artificial dataset to learn structure information (Yu et al., 2020b,a). This enables the model to effectively performs the hierarchical semantic parsing task using fewer computational resources.
Grammar-Constrained Neural Network Models Incorporating constraint grammar in text generation tasks using neural networks is a topic of
interest to researchers. Yin and Neubig (2017) proposed an approach using a grammar model to generate an abstract syntax tree by a series of actions. Krishnamurthy et al. (2017) presented a typeconstrained semantic parsing model, which ensures that the decoder only generates well-typed logical forms. Shin et al. (2021) demonstrated that the constrained decoding process with an intermediate sub-language mapped by a large language model can support parsing the target language. More recently, Baranowski and Hochgeschwender (2021) extracted context-free grammar from the target logical form of the semantic parsing task. These grammar constraints were then enforced with an LR parser to maintain syntactically valid sequences throughout decoding. In comparison, our approach utilizes grammar as additional structured information during training with a conditional loss function. This key difference sets our approach apart from previous works.

heading: 3 Method

heading: 3.1 Overview
text: Our StructSP framework consists of two finetuning phases: structure-aware boosting and grammar-based RINE (Figure 2). In the first phase, Structure-Aware Boosting, we use a pre-trained RoBERTa model (Liu et al., 2019) as a backbone and continue to train it in a multi-task setting. This improves the model’s ability to express structured information of hierarchical representation by using two sub-tasks: structure-focused MLM (section 3.2.1) and relative tree agreement (section 3.2.2).
In the next phase, we use grammar-based RINE, which uses the RINE approach (Mansimov and Zhang, 2022) augmented by grammar rules to tackle the problem of the hierarchical semantic parsing task. The parsing process is split into multiple steps, where the input of each step is the output of the previous one. The structure-aware model from the previous phase is used as an encoder to make predictions. Especially, an inductive grammar synthesized using training data is used to prune the unpromising decoding directions. The grammar rules not only correct the parsing process but also reduce resource usage by eliminating unnecessary predictions.

heading: 3.2 Structure-aware Boosting

heading: 3.2.1 Structure-focused MLM
text: Before introducing our proposed structure-focused masking strategy, we describe the original MLM (Devlin et al., 2019).
Original MLM Given a sequence of tokens X = [x1, x2, ..., xm], probability T = [t1, t2, ..., tm] corresponding to the input sequence is generated, where ti indicates the probability that token xi will be chosen for masking. Following Devlin et al. (2019), all ti are equal to 15%, meaning that every token has the same probability of being chosen in the masking process. Let D be the set of tokens randomly chosen for masking (e.g., D = {x2}), and let X ′ be the sentence with masked tokens (e.g., X ′ = [x1, [MASK], ...xm]). The cross-entropy loss function is computed for all masked tokens:
Lmlm = − 1
|D|
|D|∑
i=1
|V|∑
j=1
xij log p(xij | X ′) (1)
where |D| is the total number of masked tokens in a sentence, |V| is the size of vocabulary of language model, and p(xij | X ′) is the probability that the model assigns to the token xi belong to token jth in vocabulary given the context provided by the masked sequence. The model is trained to minimize this loss function, which measures how well it can predict the masked tokens on the basis of the context provided by the unmasked ones
Structure-focused MLM In the linearized hierarchical representation of an utterance (Table 1), there are two types of tokens: normal and logical. The logical tokens can be divided into two sub-categories: bracket tokens ("[" or "]") indicate the span of a label, and label tokens represent the label itself (e.g., IN:GET_EVENT). The correct prediction of a masked bracket token demonstrates the model’s understanding of the label span. Similarly, the correct prediction of a masked label token demonstrates the model’s understanding of the label type. Therefore, masking logical tokens to train the model for learning the hierarchical structure is a reasonable approach (Bai et al., 2022). However, the original MLM treats all tokens equally, regardless of whether they are normal or logical.
In our approach, we modify the original MLM approach by assigning a higher masking probability to logical tokens, thereby pushing the model to pay more attention to the hierarchical structure (the
logical token) information. This enables the model to learn the structural characteristic of natural sentences while retaining the knowledge learned from the original MLM (Yu et al., 2020a). More specifically, our structure-focused MLM strategy gives logical tokens a higher masking probability, defined as α (with α > 15%). Let T ′ = [t′1, t ′ 2, ..., t ′ m] be the revised sequence of masking probabilities, where t′i = α if the corresponding i
th token is a logical one. The structured-focused MLM loss (Lstr_mlm) function of masked tokens is computed similarly to the original MLM loss.

heading: 3.2.2 Relative Tree Agreement
text: In hierarchical representation, leaf nodes are words, while non-terminal nodes are semantic tokens: either intents or slots. We define a non-terminal list as an ordered list of nonterminal nodes by using a breadth-first search. For example, using the tree shown in Figure 2, we obtain the non-terminal list [IN:GET_EVENT, SL:CATE_EVENT, SL:NAME_EVENT]. This list is then used to form positive training samples.
The goal of this task is to improve the ability of the model to produce similar representations for trees in the same parsing process, enabling it to effectively perform the recursive insertion-based semantic parsing task. To achieve this goal, a pruned tree is generated at each training iteration by randomly selecting a node from the non-terminal list of the full tree (i.e., the annotated tree) and pruning all non-terminal nodes to the right of it in the list. This pruned tree is relative to the full tree. Formally, we denote the linearized representation of the full tree as Pfull and that of the pruned tree as Ppruned. The hidden vector representations of these two trees (hfull and hpruned) are encoded using a pre-trained language model:
hfull = RoBERTa(Pfull) hpruned = RoBERTa(Ppruned)
(2)
We use contrastive learning (Frosst et al., 2019; Gao et al., 2021; Bai et al., 2022; Luo et al., 2022) to train our model for this task. The aim is to align the representations of positive pairs in the latent space by minimizing the distance between them while simultaneously maximizing the distance between negative pairs. Specifically, we define a contrastive loss function that measures the similarity between the hidden states hfull and hpruned. This information is then used to update the model’s parameters. Minimizing this loss function enables
our model to learn how to produce more similar representations for the full and pruned trees. In particular, given a training batch B, the positive pair at the ith position is (h(i)full, h (i) pruned), and the negative pairs are the other samples in the batch. The training objective is given as follows:
Lcontrastive = −log exp(sim(h(i)full, h (i) pruned)/τ)∑
j∈B exp(sim(h (i) full, h (j) pruned)/τ)
(3)
where sim is the similarity function based on cosine distance, h(i)full and h (i) pruned are the hidden states for the full and pruned trees, respectively, at the ith position in the batch indexes. τ is a temperature hyperparameter (Gao et al., 2021), and B is the set of samples in the mini-batch.

heading: 3.2.3 Objective Function
text: We combine the objectives of the two sub-tasks to formulate the structure-aware boosting loss (Lsab).
Lsab = Lcontrastive + λ ∗ Lstr_mlm (4) where λ is a hyperparameter that balances the contributions of the two objectives. As proposed elsewhere (Lee et al., 2020; Nandy et al., 2021; Bai et al., 2022), we do not train our model from scratch. Instead, we use a pre-trained language model to initialize its parameters. This enables us to leverage the knowledge contained in the language model.

heading: 3.3 Grammar-based RINE
text: RINE In this phase, we use a recursive insertionbased approach (Mansimov and Zhang, 2022), with our structure-aware boosted model created in the previous phase serving as the backbone. The parsing process can be formally represented as a chain of incremental sub-parsed trees, denoted as P = [P0,P1, ...,Pgold] (Table 1).
At the ith step, the model receives the tree in the previous step Pi−1 as input and predicts the current node label and its span to decode the tree Pi. The tree Pi is updated to include a new prediction node
and processing recursively moves to the next step i+ 1 until a special end-of-prediction signal (EOP) is found.
Grammar constraint Observation revealed that the relationships between nodes in the semantic tree are important information in the parsing process. For example, the intent IN:GET_EVENT typically comes with slots for event information, SL:CATE_EVENT or SL:NAME_EVENT. Therefore, we introduce a grammar integration mechanism into RINE to reduce movement in unpromising decoding directions. Particularly, we construct grammar G = {A → B | A,B are non-terminal nodes} on the basis of training data (e.g., IN:GET_EVENT → SL:CATE_EVENT ). Integrating grammar into the RINE model means that the parser considers only the potential nodes in a candidate set (e.g., C = G(A)) on the basis of the parent node (e.g., B ∈ C) at each step of node label prediction.
Modeling For mathematical modeling, given subtree Pi at the ith step, our model first encodes this tree as a sequence and obtains the corresponding hidden states fused by context. Following Mansimov and Zhang (2022), we use hidden states of the [CLS] token in the last transformer encoder layer to predict the label of the next node. Furthermore, to predict the range of this node, we use all hidden states in the last two layers to compute the probabilities of the start and end positions.
pnodeLb = softmax(Wlbh (l) [CLS] + blb) (5)
pstartk = softmax(Wsh (l) wk + bs) (6)
pendk = softmax(Weh (l−1) wk + be) (7)
where W · and b· are the learnable parameters, l is the number of transformer encoder layers in the pre-trained language model, and wk is the kth token (word) index of the input sequence. We use a special penalty score (spenalty) injected into the loss function to cause the unpromising label node predictions to be ignored:
spenalty = { 0 if nodeLb ∈ C +∞ otherwise
(8)
where C is the candidates generated from the grammar and the parent node. Finally, the cross-entropy (CE) losses are synthesized in accordance with the
gold label (gnodeLb) and these output probabilities:
LnodeLb = CE(gnodeLb, pnodeLb) + spenalty (9) Lstart = ∑
k
CE(gstartk , p start k ) (10)
Lend = ∑
k
CE(gendk , p end k ) (11)
Lfinal = LnodeLb + Lstart + Lend (12)
Inference challenge In the inference step, the challenge with the approach is a lack of information about which node is the parent node at the current step to be used to generate the candidate set. We propose a simple strategy to meet this challenge: use the result of span prediction to determine the parent node. For example, considering the 3rd step in Table 1 and given input P2, we have no information about the next node to be predicted and do not know whether the parent node is IN:GET_EVENT or SL:CATE_EVENT. We predict the span first and obtain the position of “Chris Rock”. We thus determine that the parent of the current node is IN:GET_EVENT, and can generate candidates C by using this node.

heading: 4 Experiment
text: To verify the effectiveness of our proposed method, we conducted experiments on datasets TOP2 and TOPv23 (low-resource setting). Following Mansimov and Zhang (2022), we conducted three random seeds for each experimental setting and reported the average results with standard deviation.

heading: 4.1 Datasets and Evaluation Metric
text: The TOP dataset (Gupta et al., 2018) is a collection of utterances divided into two domains: navigation and event. Twenty-fine intents and 36 slots are used to represent the utterances. The mean depth of the trees in the dataset is 2.54, and the mean length of the utterances is 8.93 tokens. Following previous works (Einolghozati et al., 2019; Rongali et al., 2020; Zhu et al., 2020; Mansimov and Zhang, 2022), we removed utterances with the UNSUPPORTED intent, resulting in 28,414 training, 4,032 valid, and 8,241 test utterances.
The TOPv2 dataset (Chen et al., 2020) focuses on scenarios with low resources. We followed Mansimov and Zhang (2022) by using low-resource versions of the reminder and weather domains. The
2 Provided under the CC-BY-SA license 3 Provided under the CC BY-NC 4.0 license
reminder domain contains 19 intents and 32 slots while the weather domain contains 7 intents and 11 slots. The low-resource data were created by selecting a fixed number of training samples per intent and slot (SPIS) from the original dataset. If an intent or slot occurred fewer times than the specified number of samples, all of the parse trees containing that intent or slot were included in the low-resource data. Specifically, we used the training, validation, and test data at 500 and 25 SPIS. At 500 SPIS, the reminder domain has 4788 training and 1871 validation samples, and the weather domain has 2372 training and 1202 validation samples. At 25 SPIS, the reminder domain has 493 training and 337 validation samples, and the weather domain has 176 training and 147 validation samples. For both SPIS settings, the test sets for the reminder and weather domains contained 5767 and 5682 test samples, respectively.
Identical to previous studies (Einolghozati et al., 2019; Rongali et al., 2020; Zhu et al., 2020; Mansimov and Zhang, 2022), we used exact match (EM) as performance metric. The EM score was calculated as the number of utterances for which fully parsed trees were correctly predicted.

heading: 4.2 Experimental setting
text: Structure-aware boosting: We used the RoBERTa (Liu et al., 2019) model as our backbone. With the TOP dataset, we used a peak learning rate of 1e-05 and continued training for ten epochs using the Adam optimizer (Kingma and Ba, 2014) with epsilon 1e-08 and a batch size of 16 sequences with 512 tokens. For the hyperparameters, we set τ = 1.0 and selected λ from {0.5, 1.0}4 . With the TOPv2 dataset, we used the same settings as for the TOP dataset except we increased the number of training epochs to 50 for the 25 SPIS setting. The training process on a single NVIDIA A100 GPU.
Grammar-based RINE: we used the model trained from the structure-aware boosting phase as an encoder. For the TOP dataset, we set the number of warmup steps to 1000, selected the learning rate from {1e-05, 2e-05}, and performed training for 50 epochs. The best checkpoint was chosen on the basis of the model’s performance on the validation set. We utilized the Adam optimizer and a batch size of 32 sequences containing 512 tokens. For the TOPv2 dataset, we used the same settings
4 The values resulting in the best performance are bold.
except for adjusting the batch size to 16 for the 500 SPIS setting and 8 for the 25 SPIS setting.
Baseline: We reproduced the RINE model (Mansimov and Zhang, 2022) as a strong baseline model and kept the same hyperparameter values as the original paper (Mansimov and Zhang, 2022).

heading: 4.3 Main Results
text: TOP dataset We evaluated the performance of our proposed StructSP method on the TOP dataset and compared it with that of the other works. The results are shown in Table 2.
Our model outperformed all the other models. Specifically, it achieved a higher score than the autoregressive seq2seq model with pointer (Rongali et al., 2020) by 1.51 EM and outperformed the best method of previous works, RINE (Mansimov and Zhang, 2022) by 0.61 EM scores. These results show the effectiveness of our proposed method in injecting prior semantic hierarchical structure information of a natural sentence into the pre-trained language model. Besides, our result outperformed the result of Bai et al. (2022) by a large margin, although the authors also strengthened the structure information of pre-trained models using a general AMR structure. Additionally, using grammar improved the performance of our model by 0.29 points compared with a version of the model trained without using grammar. This demonstrates the value of incorporating grammar into our model when using the TOP dataset.
TOPv2 dataset We evaluated the performance of our proposed StructSP method on the low-resource data of the reminder and weather domains of the TOPv2 dataset and compared it with that of the other methods. The results are shown in Table 3.
Our model outperformed the others for all SPIS settings. At the 25 SPIS setting, our StructSP outperforms the baseline by 0.93 EM and 1.02 EM in the weather and reminder domains. Notice that, with these low-resource settings, in the structure-aware boosting phase, the model also only continues to train with limited data but still achieves impressive improvements.
In addition, at the 25 SPIS setting of the weather domain, we observed that the performance of the StructSP was improved when it was trained without using grammar. We argue that this was due to the size of the training set at this setting being extremely small (176 samples), and the extracted grammar from this training set is not expressive enough to cover the grammar in the validation and the test set.

heading: 5 Analysis

heading: 5.1 Ablation Study
text: To evaluate the effect of using each component in our framework, we compared the model’s performance for each combination of component settings with that of the RINE baseline model on the validation set of the TOP dataset (Table 4).
We found that using grammar in the second phase grammar-based RINE led to improved performance on the TOP dataset (EM score 0.12 points higher than with the baseline model). Additionally, using the structure-aware boosting phase substantially improved the EM score compared with the baseline (0.52 points higher). To further understand the contributions of each training subtask in the structure-aware boosting phase, we conducted two additional experiments. The results show that using the structure-focused mask language modeling subtask improves EM performance by 0.36 points compared with the baseline while using the relative tree agreement subtask leads to only a 0.05 improvement.
Furthermore, we conducted a t-test (Lehmann and Romano, 1986) with the null hypothesis that the expected values of our full-setting model (StructSP) and the baseline model (RINE) are identical (Table 4). Based on the experimental outcomes, the p-value was found to be below 0.05, which suggests that the proposed method outperforms the baseline model significantly.

heading: 5.2 Effect of masking probability α
text: In another experiment, we analyzed the effect of the logical-token masking probability (α in Section 3.2.1) in the structured-aware boosting phase on overall performance (Figure 3). High performance was achieved when α was set to 0.3 or 0.4. We attribute this to our mechanism pushing the model to pay more attention to the logical tokens, helping it to better capture the structure. In sum-
mary, these results show that our StructSP method achieved better performance than the baseline for all values of α, which demonstrates the robustness of our approach.

heading: 5.3 Case Study
text: Table 5 presents several example utterances from the TOP dataset. In the first example, our model
5 Please refer to Appendix C for tree representation.
predicted the slot SL:POINT_ON_MAP after the intent IN:GET_LOCATION, whereas the baseline model predicted the slot SL:NAME_EVENT. This difference occurred because the extracted grammar does not contain the constraint (IN:GET_LOCATION => SL:NAME_EVENT). This demonstrates the effectiveness of incorporating grammar into our model. The second example presents a ground-truth tree with a complex structure, requiring the model to identify the span "after a Pacers game" as a date time slot (SL:DATE_TIME) and then correctly parse the structure within this span. Our model was able to correctly return the tree, whereas the baseline model was not. The final example is a particularly challenging one, the tree has a depth of 5, indicating that the tree structure is highly complex. Both models failed to return the correct predictions, suggesting that learning to handle such complicated queries is an interesting topic for future work.

heading: 6 Conclusion
text: We have presented a novel approach to improving the performance of SOTA semantic parsing models on hierarchical representation datasets. In this approach, a model is created that incorporates knowledge of the utterance structures into the semantic parsing process. This is achieved by learning contextual representations from the hierarchical representation of utterances with objective functions targeted at the semantic parsing task as well as by using grammar rules containing knowledge about the structure of the data for training and label prediction. In experiments on the TOP and TOPv2 datasets, our model outperforms previous SOTA approaches.

heading: 7 Limitations
text: There are two main limitations to our works. (1) Grammar constraint: The results of the StructSP method at the 25 SPIS setting in the TOPv2 dataset (Table 3) suggest that the results
of using grammar with low-resource data can be uncertain. The reason is that the extracted grammar from training data for low-resource setting is not general enough to capture the grammar of the new coming data (validation or test set). Therefore, for our StructSP method to work effectively, the provided grammar should cover all grammar rules if possible.
(2) Prediction time: A recursive insertion-based strategy is used for prediction. This means that the output of the previous parsing step is used as input for the current parsing step, and this process continues until a terminal signal is encountered. As a result, parsing a complex tree with multiple intents/slots (labels) can be a lengthy process due to the recursive nature of this method. Future work includes improving parsing prediction time by predicting all labels at the same level in the parsed tree rather than predicting them one by one.

heading: Acknowledgement
text: This work is partly supported by AOARD grant FA23862214039

heading: A Data Pre-processing
text: Before training our models, we perform the following data processing steps:
• Step 1: Extract the grammar using the full trees of the training samples, with the constraint "parent node -> child node" (Eg. IN:GET_LOCATION -> SL:NAME_EVENT).
• Step 2: Convert the ground-truth full trees into multiple sub-parsed trees, each represented as a triple (Current parsed tree, Label, Label span) (Figure 1).
• Step 3: Extract the set of label types from the training samples.
With the above data, we then proceed to train our models in two phases.

heading: B Model Hyper-Parameters
text: The hyper-parameters used in our models in the TOP dataset are shown in Table 6. In the TOPv2 dataset, we use the same hyper-parameters with the following exceptions: for the 25 SPIS setting in the structure-aware boosting phase, the training epoch was adjusted to 50; for the 500 SPIS and 25 SPIS settings in the label prediction phase, the batch size was adjusted to 16 and 8, respectively. The hyperparameters used in our models in the TOP dataset Our framework is implemented using Pytorch6 and HuggingFace Transformers.7 . Our source code and extracted grammar can be found at: [masked] (access will be granted upon acceptance).

heading: C Case study outputs with tree representation
text: Figure 5 presents the example of the parsing process described in Section 1 using tree representation. Additionally, Figure 5 displays the outputs of examples discussed in Section 5.3 using tree representation.
6 https://github.com/pytorch/pytorch 7 https://github.com/huggingface/transformers
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
7
7 A2. Did you discuss any potential risks of your work? Our research focuses on improving the performance of the semantic parsing task, which is experimented on well-known published datasets.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 4
3 B1. Did you cite the creators of artifacts you used? 4
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? 4

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 4
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? This data is used by previous works.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 4
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 4
C 3 Did you run computational experiments? 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 4

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Future work includes improving parsing prediction time by predicting all labels at the same level in the parsed tree rather than predicting them one by one