Syntax Matters: Towards Spoken Language Understanding via Syntax-Aware Attention
Authors: Yifeng Xie
Zhihong Zhu
Xuxin Cheng
Zhiqi Huang
Dongsheng Chen
Venue: EMNLP 2023 Findings (2023)
Keywords: Spoken Language understanding
Syntactic Dependency Parsing
Feature Fusion

Abstract: Spoken Language Understanding (SLU), a crucial component of task-oriented dialogue systems, has consistently garnered attention from both academic and industrial communities. Although incorporating syntactic information into models has the potential to enhance the comprehension of user utterances and yield impressive results, its application in SLU systems remains largely unexplored. In this paper, we propose a carefully designed model termed Syntax-aware attention (SAT) to enhance SLU, where attention scopes are constrained based on relationships within the syntactic structure. Experimental results on three datasets show that our model achieves substantial improvements and excellent performance. Moreover, SAT can be integrated into other BERT-based language models to further boost their performance.

Syntax Matters: Towards Spoken Language Understanding via Syntax-Aware Attention
Yifeng Xie
Zhihong Zhu
Xuxin Cheng
Zhiqi Huang* Dongsheng Chen
Guangdong University of Technology, PPkkng University
evfxie@gmail.com
Abstract
Intent
Utterance
NUM
How many watts make kilowatt
Spoken Language Understanding (SLU), cru- cial component task-oriented dialogue sys- tems, has consistently garnered attention from both academic and industrial communities. Al- though incorporating syntactic information into models has the potential to enhance the compre- hension user utterances and yield impressive results, its application in SLU systems remains largely unexplored. In this paper. we propose carefully designed model termed Syntax-aware attention (SAT) enhance SLU, where atten- tion scopes are constrained based on relation- ships within the syntactic structure. Experi- mental results three datasets show that our model achieves substantial improvements and excellent performance. Moreover, SAT can be integrated into other BERT-based language models to further boost their performance.
ABBR
What is the abbreviation for micro" ?
DESC
What caused the death of Bob Marley
ENTY
What kind of creature is coot ?
LOC
Where is the Valley the Kings ?
HUM
Who was the second man to walk on the moon ?
What Adion city bossts the world Nigestt bowling alley
Atinn worch bniding
Figure I: Intent classification examples TREC6 dataset (Top) and syntax parsing process (Bottom).
training with utterances alone. However, relevant research on SLU is lacking.
To address the deficiency, we propose Syntax- aware attention (SAT) for SLU. Specifically, we first utilize pre-trained dependency parse tree structure to generate the related nodes for each word in a sentence. These related nodes, known syntax-aware attention mask, consist of all ances- tor nodes and the word itself in the dependency parsing tree, treating the word as a child node. Examples intent detection and syntax parsing are shown Figure 1. To effectively incorporate Syntax- aware Attention Module (SATM), we also design the Dual Context Fusion Module (DCFM) to merge the original Transformer (Vaswani et al., 2017) with the syntax-award Transformer. This en- ables SAT provide more linguistically inspired representations for SLU.
1 Introduction
In recent years, there has been surge of research interest in dialogue systems from academia and in- dustry, where SLU (Chen et al., 2017; Huang et al., 2020; Chen et al., 2022b) is crucial component, re- sponsible for identifying user intents based on their natural language utterances (Huang et al., 2021c c, 2022; Chen et al., 2022a; Zhu et al., 2023c.b). Many works have been proposed to enhance the understanding of utterance semantics, leading to remarkable successes.
While language models (LMs) have the ability to capture contextual semantics to some extent, their performance can be further improved by integrat- ing external knowledge (Cheng et al., 2023c). The inclusion of additional knowledge in LMs has be- come growing trend, and researchers have exten- sively explored the integration of syntax informa- tion into pre-trained LMs (Bai al., 2021: Zhang al., 2020, 2022; Gong et al., 2022). Incorporat- ing syntax information through parser dependen- cies into PLMs improves performance compared to
To summarize, our contributions are three-fold: (1) S AT can capture the syntactic structure informa- tion from utterances for enhancing SLU. It can be straightforward integrated into BERT-based mod- els. (2) In order effectively merge syntactic in- formation, various feature fusion methods were explored with the aim of enhancing linguistic rep- resentation. (3) Experiments demonstrate the effec- tiveness on three SLU benchmarks.

---

Mull-tta
2.31 Stam
Input
Foor
Dual Confext Fusion Module
Embedding
MLP Layer
sentence
Ao0t hom
Syntex- Awore Attention Module
Figure 2: The architecture SAT. The utterance first passed to the utterance encoder ($2.1) and SATM ($2.2) to obtain feature representations, which then fused DCFM ($2.3) obtain the final representation ($2.4).
2 Methodology
of S AT mask M E Ru from the parser:
The main architecture mainly consisting of SATM and DCFM illustrated Figure 2. We elaborate on each component below.
1, qj E P. or i-j
M[i
(1)
0, otherwise
From the formulation, if M[i.j] = 1, it means that the node qj is the ancestor of q: or qi itself.
2.1 Backbone Model
Afterward, we project the outputs of the back- bone model HELS into separate query Q, key K, and value V representations. The syntax-aware attention scores be calculated follows:
Following previous works, select two back- bone models, i.e., BERT (Devlin al., 2019) and RoBERTa (Liu al., 2019h). Specifically, for a given utterance x = {.) "..,., "n), we prepend [CLS] to represent the entire sequence and append [SEP] to separate non-consecutive token sequenees (Devlin et al., 2019). Subse- quently, we obtain the encoded feature representa- tion H {HOLS, H1, H2 .... Hn, HSEP} and feed HOLS E 1xd into the backbone model, where L represents the length of the utterance and d the hidden dimension. Finally, HeLs is passed through a simple multilayer perceptron (MLP) neu- ral network to predict the intent.
M (QKT)
W SATM = softmax
V
(2)
ddn
where dy is the dimension of key K. Concatenating all heads W Watm and passing them through a feed- forward layer, we obtain the syntax-aware attention output HSATM.
2.3 Dual Context Fusion Module (DCFM)
We obtain original and syntax-aware feature rep- resentations HELS and HSATM to enhance the model's performance. Different fusion methods have varying impacts on the representations, mo- tivating us to explore diverse techniques to im- prove effectiveness. We then explore several rep- resentative fusion methods, including concatena- tion and more complex mechanisms that consider the correlation between the two outputs. The fu- sion process of two embeddings is defined as: Y HCLS + HSArM where represents fu- sion method, H is the final representation for in- tent detection. Specially, We choose three typical works (Wang et al., 2018: 8; Liu al., 2020; Li al., 2019) 9) to compare their effectiveness: a way that extracts the weights of features to assign
2.2 Syntax-Aware Attention Module (SATM)
Unlike previous approaches that treat addi- tional syntactic features as a separate part of PLMs (Strubell et al., 2018; Kasai et al., 2019; Duan et al., 2019), we implement SATM by apply- ing an attention mask on self-attention to enforce syntactic constraints. Specifically, we begin by deriving the syntactie structure from dependency parsing, treating it directed tree. Each token node qi
h, is then mapped to a corresponding For a given node qi we define the its syntac- tie ancestors as P;. To focus attention on the node and its syntactic ancestors, we obtain a sequence
(1) LEAM is

---

coefficient, (2) aimNet symmetrical multi- modal attention module, (3) SKnet is gate-based fusion method applied to images.
3.2 Main Results
Table I presents the main results on three bench- mark datasets. SAT has shown significant improve- ments across various performance metrics. For ex- ample, on the TREC6 dataset, SAr demonstrates notable enhancement of 0.6% in accuracy and 0.8% in Fl-score compared to the gate unit based on RoBERTa. Thus, wc analyze the factors contribut- ing to the success of SAT, which are elaborated as follows:
We further propose a Dual Context Fusion Module (DCFM). Firstly, we introduce gaterase that involves extracting useful features from both the original embedding and the syntax embedding. The data is then mapped to range of 0 to 1, serv- ing filter gate. This gate selectively filters the information from the syntax embedding. mitigat- ing the errors caused by ASR and parser to some extent. Subsequently, the filtered data undergoes additional processing using an activation function and fully connected layer to retain only the rele- vant and useful features:
(1) Incorporating syntactic information improved results by capturing keywords with dependency re- lationships. The utilization of syntactic information proves to be beneficial, particularly in cases where such information is lacking. These findings align (3) with previous studies (Sachan et al., 2021).
gatebase: a ( HELS: HSATM] + bi)
(2) Considering the non-robust nature of syntax embedding, we explore various fusion methods, finding that the gated unit or gate-based fusion method outperforms other fusion methods. We attribute this effectiveness to SAT's ability to ef- fectively leverage two distinct features, leading to enhanced information utilization, more reasonable weight allocation, and improved modeling of syn- tactic relations.
H W,sTTanb(aaggatee W2(gatthase
HSATM) b2)) + b3
(4)
where [. denotes concatenation operation, W. and b, trainable weights, is sigmoid activa- tion (Han and Moraga, 1995) and Tanh is Tanh activation (Kalman and Kwasny, 1992). To ensure the robustness the model's performance, it is nec- essary to combine it with the original embedding. Inspired by Self-Adaptive Scaling Approach (Liu et al., 2019a), we utilize control gate that automat- ically learns the most relevant and useful features to obtain the final embedding in robust manner.
(3) Errors in ASR and parsing can distort or ren- der token relationships irrelevant. For instance, when the verb "is" is the keyword in an utterance, its relationship may not be the most crucial one. One of the notable advantages of SAT is its ability to mitigate such errors to certain extent.
H = LN(HoLS + Ho) = a HCLS 8 H + +11- a) 9) -))-H H,
(5)
(6) 3.3 Integration with Other Models
1
(7)
We compare SAT with previous works (Sundarara- man et al., 2021; Chang and Chen, 2022) on the same benchmark datasets. In Table 2, we find that our proposal SAT outperforms the other models. Moreover, one of the key advantages of SAT is its seamless integration with BERT-based models. This integration has significantly enhanced the over- all performance, highlighting the versatility and potential impact of our proposed model in driving major advancements in the field.
where and B are hyperparameters and leam by training, and LN stands for layer normalization. remains the ultimate outcome.
2.4 Training Objective
Following Cheng et al. (2023a,b), we employ a fully connected network as a task classifier, and optimize SAT using Cross-Entropy.
3
Experiment
3.4 Ablation Study on Parser
3.1 Datasets and Experimental Settings
We also extract syntax information in the form of dependency trees using high-precision parser, such as the HPSG parser (Zhou and Zhao, 2019). coupled with two different pre-trained backbone models: XL-Net (Yang et al., 2019) and BERT (De- vlin et al., 2019). Notably, the HPSG parser with
We conduct experiments on three widely used benchmark datasets, namely TREC6, TREC50 and ATIS (Hovy al., 2001; Li and Roth, 2002; Shiv- akumar et al., 2019). Detail analysis and experi- mental settings can be seen in Appendix A.

---

Pre-trained Model
Syntax Information
Fusion Module
Fusion Method
highlighting SAr's ability identify relevant fea- Dataset: TREC6
Dataset: TREC50
Datasct: ATIS
Acc
Ace
Acc
83.9
79.9
82.6
75.8
57.8
62.5
82.8
80.1
78.4
82.7
76.2
58.3
61.2
58.6
79.4
77.2
add
84.4
82.3
82.3
83.4
75.2
59.1
60.9
58.3
94.6
77.8
76.4
BERT
LEAM
84.2
82.4
81.5
83.7
75.9
57.5
59.7
60.1
94.9
80.1
81.2
86.8
aimiNet
84.8
82.3
81.2
83.8
77.0
58.5
58.0
58.7
94.8
79.6
76.5
87.2
SKaet
85.2
82.5
81.4
83.9
78.0
60.8
62.1
63.4
95.1
79.8
76.3
87.4
gate unit
85.6
83.8
83.8
84.8
78.2
61.8
64.2
64.6
95.2
80.8
77.8
88.5
SAT
86.2
85.6
86.5
85.0
78.9
62.6
65.0
64.6
95.3
82.7
81.5
88.8
X
x
84.1
82.1
82.5
83.4
76.0
59.9
60.1
61.5
94.5
79.8
77.8
86.3
83.2
81.5
79.1
82.7
76.4
60.1
62.2
64.0
94.8
81.5
79.2
87.0
add
84.6
84.3
84.9
83.9
75.4
61.1
62.9
63.7
94.9
81.5
78.8
87.9
RoBERTa
LEAM
84.6
83,5
83.1
84.1
76.6
57.8
62.4
60.0
95.1
81.2
78.5
87.8
aimNet
85.2
83.9
83.7
84.4
77.2
59.8
61.1
63.7
95.0
80.7
78.0
87.2
SKnct
85.4
84.0
84.2
83.9
78.6
60.6
64.1
61.6
95.2
80.1
76.3
88.0
gate unit
85.8
85.0
86.3
84.5
78.2
62.1
64.7
64.5
95.3
81.8
81.0
88.4
SAT
86.4
85.8
87.0
85.0
79.0
62.8
65.2
64.6
95.4
83.0
81.2
89.5
Table Results on three datasets. "Syntax Information" refers to the inclusion of SAT mask the backbone model.
Dalasel
the capacity to recognize crucial information. For example, the keyword "describe" shows high at- tention scores with the tokens "term" and "word",
Method
TREC6
TREC50
ATIS
RoBERT' (Liu 2019b)
84.1
76.6
94.4
Phoneme-BFRT (Sundararaman et ol., 2021)
85.9
77.8
94.8
SpokenCSB (Chang and Chen, 2022)
86.4
76.2
95.1
Proposed (S)
86.4
95.4
tures that might overlooked by the original at- tention. This finding provides a compelling expla- nation for the effectiveness of SAT.
Phoneme-BEKT SAT)
86.6
79.4
95.6
SpokenCSB (finetune SAT)
87.2
76.6
95.8
Table 2: Comparing results accuracy on benchmarks.
Fusion
Metric: Accurncy
Method
HPSSG-Paree-BIIT HPSSG-Parse--L. XL-NNe
ablition
B4.4
84.6
0.2
LEAM (Wang 2018)
simNar (L et all, 20200)
K)
K5.1
012
SKnet (Li 2019)
8488
85.2
0.4
gate 2018)
DCFM
863
86.4
0.1
Table 3: Accuracy on TREC6 about using different parser based RoBERTa.
Figure 3: Visualization attention score selected from
last layer and averaged over all heads. Left is the origi- nal attention, while right is the syntax attention.
XL-Net outperforms the HPSG parser with BERT in terms of performance. The results presented in Table clearly demonstrate that a highly ac- curate parser yields superior outcomes. In other words, there is a direct correlation between the ac- curacy of the parser and the performance achieved, Furthermore, these findings further emphasize the importance of incorporating syntax dependency in- formation to enhance the intent detection task.
Conclusion
In this paper, we propose a novel Syntax- aware attention (SAT) to improve SLU by incorporat- ing syntactic information, specifically dependency structures, into PLM. Experimental results demon- strate the superiority of our method over the back- bone BERT/ROBERT: models on three benchmark datasets. Moreover, we experiment with multiple parsers and find that parsers with stronger syntactic parsing abilities lead more substantial improve- ments SLU systems. Detailed analysis confirms the effectiveness and power of SAT in enhancing SLU systems.
3.5 Visualization
In Figure 3, we compare the syntax attention and the ordinary attention using heat-map of attention scores. The heat-map excludes [CL.] and [SEP] tokens to establish a clearer correlation among the other tokens. We observe that the SAT exhibits

---

Limitations
Devlin, M. Chang, Lee. and K. Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACI.
While the Syntax aware attention (SAT) exhibits significant improvements by incorporating syn- tactic information, it still possesses certain draw- backs. Firsdl, our method can only be applied in Transformer architectures that rely on self-attention mechanisms. Without the attention mechanism as foundation, the utilization of our method would be infeasible. Secondly, SATM hinges the gram- matical relation of "all ancestor nodes". We be- lieve that exploring the relationships among "all ancestor nodes + limited child nodes" could yield valuable insights that deserve further investigation. Thirdly, as a gated mechanism, the introduction of additional parameters by DCFM may potentially increase the risk of overfitting and further compli- cate the clear explication of its function within the framework.
S. Duan, H. Zhao, J. Zhou, and R. Wang. 2019. Syntax- aware transformer encoder for neural machine trans- lation. In Proc. IAL.P
Z. Gong, K. Zhou, X. Zhao, J. Sha, S. Wang, and Wen. 2022. Continual pre-training of language models for math problem understanding with syntax-aware memory network. In Proc. of ACL.
Han and C. Moraga. 1995. The influence of the sig- moid function parameters on the speed of backpropa- gation learning. In From Natural Artificial Neural Computation, International Workshop on Artificial Neural Networks.
C. Hemphill, Godfrey, and G. Doddington. 1990. The ATIS spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of Workshop Held at Hidden Valley.
L. Hou, Z. Huang, L. Shang, X. Jiang, X. Chen, and Q. Liu. 2020. Dynahert: Dynamic hert with adaptive width and depth. In Proc. of NeurlPS.
References
J. Bai, Wang. Y. Chen, Yang. J. Bai, J. Yu, and Y. Tong. 2021. Syetax-bert: Improving pre-trained transformers with syntax trees. In Proc. of EACL.
E. Hovy, Gerber. U. Hermjakob, C. Lin, and D. Ravichandran. 2001. Toward semantics-based answer pinpointing. In Proc. of HLT.
Y. Chang and Y. Chen. 2022. Contrastive learning for improving ASR robustness in spoken language un- derstanding. Proc. of INTERSPEECH.
Z. Huang, D. Chen, Z. Zhu, and X. Cheng. 2023. Mclf: A multi-grained contrastive learning framework for asr-robust spoken language understanding. In Proc. of EMNLP Findings.
D. Chen, Z. Huang, X. Wu, S. Ge, and Y. Zou. 2022a. Towards joint intent detection and slot filling via oigher-order attention. In Proc. of IJCAI.
Z. Huang, L. Hou, L. Shang, X. Jiang, X. Chen, and Q. Liu. 2021a. Ghostbert: Generate more features with cheap operations for BERT. In Proc. of ACL.
D. Chen, Z. Huang, and Y. Zou. 2022b. Leveraging bilinear attention to improve spoken language under- standing, In Proc. of ICASSP.
Chen, Liu, D. Yin, and Tang. 2017. survey on dialogue systems: Recent advances and new frontiers. SIGKDD.
Z. Huang, F. Liu, X. Wu, S. Ge, H. Wang, W. Fan, and Y. Zou. 2021b. Audio-oriented multimodal machine comprehension via dynamic inter- and intra-modality attention. In Proc. of AAAI
Y. Chen, D. Hakkani-Tür, G. Tür, A. Celikyilmaz, J. Gao, and L. Deng. 2016. Knowledge as teacher: Knowledge-guided structural attention net- works. Computing Research Repository.
7. Huang, F. Liu, P. Zhou, and Y. Zou. 2021c. Sentiment injected iteratively co-interactive network for spoken language understanding. In Proc. of ICASS.
X. Cheng, B. Cao, Q Ye, Z. Zhu, H. Li, and Y. Zou. 2023a. MI-Imel: Mutual learning and large-margin contrastive learning for improving asr robustness in spoken language understanding. In Proc. of ACI. Findings.
7. Huang, F. Liu, and Y. Zou. 2020. Federated learn- ing for spoken language understanding. In Proc. of COLING.
Z. Huang, M. Rao, A. Raju, Z. Zhang, B. Bui, and C. Lee. 2022. MTL-SLT: multi-task task learning for spoken language tasks. In Proc. of ConvAI@ALL.
X. Cheng, W. Xu, Z. Zhu, H. Li. and Y. Zou. 2023b. Towards spoken language understanding via multi- level multi-grained contrastive lcarning. In CIKM.
B. Kalman and S. Kwasny, 1992 Why tanh: choosing sigmoidal function. In Proc. IJNN.
Cheng. Z. Zhu, B. Cao, Q. Ye, and Y. Zou. 2023c. Mrrl: Modifying the reference via reinforcement learning for non-autoregressivv joint multiple intent detection and slot filling. Proc. of EMNLP Find- ings.
J. Kasai, D. Friedman, R. Frank, D. Radev, and O. Ram- bow. 2019. Synta neural semantic role label- ing with supertags. In Proc. of NAACI.

---

L. Li, and Qi. 2018. self-attentive model with gate mechanism for spoken language understanding. In Proc. of EMNLP.
Z. Zhang, Y. Wu, J. Zhou, S. Duan, H. Zhao, and R. Wang. 2020. Sg-net: Syntax-guided machine reading comprehension. In Proc. of AAAI.
Li and D. Roth. 2002. Learning question classifiers. In Proc. of COLING.
J. Zhou and H. Zhao. 2019. Head-driven phrase struc- ture grammar parsing on penn treebank. In Proc. of ACL.
X. Li, W. Wang, X. Hu, and J. Yang. 2019. Selective kernel networks, In Proc. of CVPR.
P. Zhou, Z. Huang, Liu, and Y. Zou. 2020. PIN: A novel parallel interactive network for spoken lan- guage understanding. In Proc. of ICPR.
F. Liu, M. Gao, Y. Liu, and K. Lei. 2019a. Self-adaptive scaling for learnable residual structure. In Proc. of CoNLI
Zhu, X. Cheng, Z. Huang, D. Chen, and Zou. 2023a. Enhancing code-switching for cross-lingual slu: A unified view of semantic and grammatical coherence. Proc. of EMNLP.
F. Liu, X. Wu, S. Ge, W. Fan, and Y. Zou. 2020. Fed- erated learning for vision-and-language grounding problems. In Proc. of AAAL
Zhu, X. Cheng, 7. Huang, D. Chen, Y. Zou. 2023b. Towards unified spoken language understand- ing decoding via label-aware compact linguistics rep- resentations. Proc. of ACL Findings.
Y. Liu, M. Ou, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, Zettlemoyer, and V. Stoyanov. 2019b. RoBERTa: A robustly optimized BERT pre- training approach. CoRR, abs/1907.11692.
Z. Zhu, W. Xu, X. Cheng. T. Song, and Zou. 2023c. A dynamic graph interactive framework with label- semantic injection for spoken language understand- ing. In Proc. of ICASSP.
T. Nguyen, 2022. Improving spoken language under- standing by enhancing text representation. In Proc. of ICASSP.
Sachan, Zhang, Qi, and W. Hamilton. 2021. Do syntax trees help pre-trained transformers extract information? In Proc. of EACL.
P. Shivakumar, M. Yang, and Georgiou. 2019. Spoken language intent detection using confusion2vec. In Proc. of INTERSPEECH.
E. Strubell, P. Verga, D. Andor, D. Weiss, and A. Mc- Callum. 2018. Linguisticalll-informe self- att-ntttntion for semantic role labeling. Proc. of EMNLP.
M. Sundararaman, A. Kumar, and Vepa. 2021. Phoneme-BERG Joint language modeling of phoneme sequence and ASR transcript. Proc. of INTERSPEECH.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, Jones, A. Gomez, L.. Kaiser, and Polosukhin. 2017. Attention is all you need. In Proc. of NeurlPS.
G. Wang, C. Li, W. Wang, Y. Zhang, D. Shen, X. Zhang, Henao, and Carin. 2018. Joint embedding of words and labels for text classification. In Proc. of ACL.
T. Xia, Y. Wang. Y. Tian, and Y. Chang. 2021. Using prior knowledge to guide bert's attention in semantic textual matching tasks. Proc. of Www.
Z. Yang, 2. Dai, Y. Yang. J. Carbonell, R. Salakhutdinov, and Q. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Proc. of NewrIPS.
S. Zhang, L. Wang, X. Xiao, and H. Wu. 2022. Syntax- guided contrastive learning for pre-trained language model. In Proc. of ACL Findings.

---

A Experiments
in each metric and believe that module's effect mains substantial and meaningful.
A.1 Datasets
We conduct experiments on three widely used benchmark datasets, namely TREC6, TREC50 and ATIS (Hovy al., 2001; Li and Roth, 2002; Shiv- akumar al., 2019; Hemphill al., 1990). The statistics of these datasets are shown in Table Noted that we employ the original text to generate SAT mask. For training, we utilize the ASR tran- script of the speech version of the original texts, following the format and partitioning scheme de- scribed in (Sundararaman et al., 2021).
Model
ABC
F1
Backlone
83.9
Attenionn
84.2
(f03) 81.2 (11.3) 82.8 (102)
(lestzad Attention)
(104)
83.2(119)
83. 8388
SATM DCFM
(616)) ().6()2.4 86.3 (12.9) 85.0 (1L.)
Table 5: Results of ablation test the TREC6 dataset.
Performance Comparison
To clarify the novelty, our work represents one of the first attempts at incorporating a syntax-aware attention method into spoken language understand- ing research. Furthermore, integration technique, DCFM (Dual Context Fu- sion Module). This is a unique contribution of our work which differentiates our approach from existing ones. In Table 6, we compare SAT with approaches such as other methods that incorporate knowledge attention or syntactic information. The results provide evidence that our proposed SATM (Syntax-Aware Attention Module) and DCFM tech- nique successfully improve the task's performance.
Dataset
#Class
Avg.
Length
Train
Test
introduce novel
TREC6
8.89
5,452
500
TREC50
50
8.89
5,452
500
ATIS
11.14
4,978
893
Table 4: Data statistics on TREC6, TREC50 and ATIS.
A.2 Experimental Settings
Considering the inference speed, the following work utilizes the BERT base (Devlin et al., 2019) and RoBERTaeas (Liu et al., 2019b) as the back- bone models. As for the parser, we employ Head- Driven Phrase Structure Grammar (HPSG) (Zhou and Zhao, 2019) for the syntax analysis of the input utterances. During training, we explore different batch sizes from range of 32, 64 and 128, and con- duct a total of 10 epochs to ensure the convergence. Furthermore, we tune the hyperparameter a and 3 of the gated unit, which are initially set to 0.5, and dynamically adjust them during the training process to achieve optimal performance.
Model
TREC6 TREC50
ATIS
K-SAN (Chen et al., 2016)
84.6
76.4
94,2
BERT-SIM (Xin al. 2021)
85.1
76.9
94.6
CapuBERT (Nguyen, 2022)
85.3
77.8
94,9
SG-Nes (Zhang et al,, 2020)
85.2
77.6
94.7
Syalax-BERT (Bai et al., 2021)
85.5
78.2
95.0
86.4
'95.4"
Table 6: Results of comparison three datasets.
To evaluate the model's effectiveness, we em- ploy accuracy, Fi-score, precision, and recall as evaluation metrics.
Ablation Study on Different Components
To further verify effectiveness of each component in SAT, we conduct set of ablation experiments on the TREC6 dataset. As demonstrated in the Ta- ble 5, the proposed module presents considcrable efficiency. It important highlight that, instead of leaning on the syntactic information, we con- ducted experiments with a standard attention layer to verify the efficacy of the syntax-aa aware attention. Consequently, we can observe that an improvement


Reviews:
[{"review_id": "kuAoE4Nkmw", "reviewer": "Reviewer_ifvP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude for the insightful reviews you've provided. We value your input and would like to address your concerns as outlined below.\n\n__R1:__\nThank you for pointing out the reference [1]. We will include this reference in the revised version and highlight the differences between our approach and theirs, as well as compare it with other frameworks that utilize syntactic information. After examining this paper, we noticed that their primary focus was on using a mask with syntax information and combining it with a gated unit to derive the final attention scores. In contrast, our approach doesn't directly combine elements within the attention block. Instead, our method emphasizes the exploration of integrating the original embedding with an embedding that incorporates syntax information, which distinguishes it from prior research.\n\n__R2:__\nThank you for raising this concern. \n- Firstly, we would like to emphasize that although the gains may seem small, our framework holds the potential to be readily and seamlessly applied to other attention-based models, as demonstrated in the following Table, where the integration of our framework has led to favorable performance improvements.\n|Method|TREC6|TREC50|ATIS|\n|-|-|-|-|\n|RoBERTa [2]|84.1|76.6|94.4|\n|RoBERTa ( + __SAT__)|__86.4__|__79.0__|__95.4__|\n|Phoneme-BERT [3]|85.9|77.8|94.8|\n|Phoneme-BERT ( + __SAT__)|86.6|__79.4__|95.6|\n|SpokenCSE [4]|86.4|76.2|95.1|\n|SpokenCSE (finetune + __SAT__)|__87.2__|76.6|__95.8__|\n\n- Secondly, we acknowledge the importance of rigorously evaluating the statistical significance of these enhancements. To that end, we conduct experiments using various backbones on the TREC6 dataset and employ a t-test (p<0.05) to validate our claims on performance improvements.\n|Backbone|Acc|F1|P|R|\n|-|-|-|-|-|\n|ALBERT [5]|86.3|85.5|86.8|84.7|\n|ELECTRA [6]|86.8|86.0|87.2|85.4|\n|DeBERTaV3 [7]|87.5|86.7|87.8|85.9|\n\nIn our revised version, we will include these statistical measures to support the significance of the reported performance improvements.\n\n__References__\n\n[1] Li et al. 2021 \"Improving BERT with Syntax-aware Local Attention\"\n\n[2] Liu et al. 2019 \"RoBERTa: A Robustly Optimized BERT Pretraining Approach\"\n\n[3] Sundararaman et al. 2021 \"Phoneme-BERT: Joint Language Modelling of Phoneme Sequence and ASR Transcript\"\n\n[4] Chang et al. 2022 \"Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding\"\n\n[5] Lan et al. 2020 \"ALBERT: A Lite BERT for Self-supervised Learning of Language Representations\"\n\n[6] Clark et al. 2021 \"ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators\"\n\n[7] He et al. 2021 \"DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Paper is well written and clearly explains their methodology with mathematical formulation\n2. Their approach is flexible and can be integrated with any BERT-based PLM.\n3. I liked the ablation analysis with different fusion methods.", "reasons_to_reject": "1. Novelty seems incremental to me. What are the ways in which this paper differs from https://aclanthology.org/2021.findings-acl.57.pdf? Is it just applying a very similar methodology to new task?\n2. Performance gains seem small. There should be p-test or atleast confidence intervals to check statistical significance.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper suggests enhancing SLU performance by incorporating syntax information through parser dependencies.\n\nThe authors utilize a pre-trained syntactic dependence parse tree to create a syntax-aware attention mask, which includes all ancestors and the word itself. They further introduce a novel Dual Context Fusion Module to integrate the original transformer with their Syntax-aware Attention Module.\n\nThe authors observe performance improvements on three publicly available SLU datasets with their approach.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qZuDhs6d7u", "reviewer": "Reviewer_EFty", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the feedback provided by the reviewers, which highlighted the limitations of the model in the paper and the need to conduct a comprehensive analysis.\n\n__R1:__\nAddressing the issue of insufficient analysis, we intend to include a dedicated section on limitations within the chapter. This section will delve deeply into the potential constraints of our proposed model, elucidating scenarios where challenges might arise. It aims to provide a clear comprehension of the operational boundaries within which our approach functions. This augmentation is geared towards ensuring that readers are thoroughly informed about the practical implications and confines of our research.\n\nWhile the Syntax-aware attention (SAT) exhibits significant improvements by incorporating syntactic information, it still possesses certain drawbacks. Firstly, our method can only be used in Transformer architectures that rely on self-attention mechanisms. Without the attention mechanism as a basis, the utilization of our method would be rendered infeasible. Secondly, SAT hinges on the grammatical relation of \"all ancestor nodes\". We believe that an exploration into capturing related relationships among \"all ancestor nodes + limited child nodes\" may provide significant insights worthy of further investigation. Thirdly, serving as a gated mechanism, DCFM's introduction of additional parameters could potentially raise the probability of overfitting and further complicate the clear explication of its function within the framework.\n\n__R2:__\nWe value your observations regarding the necessity for a more comprehensive analysis of our design decisions.\n\nFirstly, we plan to extensively elaborate on the methodological analysis. Our aim is to provide an in-depth explanation of the rationale behind this design choice, with the goal of offering a more comprehensive and insightful analysis. This effort is intended not only to address the concerns you have raised but also to bolster the overall credibility of our proposed approach.\n\nSecondly, we will include an ablation study in our methodology. As demonstrated in the table, the proposed module presents considerable efficiency. It's important to highlight that, instead of leaning on the syntactic information, we conducted experiments with a standard attention layer to verify the efficacy of the syntax-aware attention. Consequently, we believe that module's effect remains substantial and meaningful.\n|Model|Acc|F1|P|R|\n|-|-|-|-|-|\n|Backbone (BERT)|83.9|81.1|79.9|82.6|\n|+ Attention|84.0 (↑0.1)|81.3 (↑0.2)|80.8 (↑0.9)|82.8 (↑0.2)|\n|+ SATM (Instead of Attention)|84.6 (↑0.6)|83.2 (↑1.9)|83.6 (↑2.8)|83.8 (↑1.0)|\n|+ SATM + DCFM|86.2 (↑1.6)|85.6 (↑2.4)|86.5 (↑2.9)|85.0 (↑1.2)|\n\nWe genuinely value the reviewer's input and believe that these revisions will significantly enhance the quality and comprehensiveness of our paper. We deeply apologize for forgetting to add this section. Thanks to the reviewer for pointing out this problem, we will fix it in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Performance Improvements over Phoneme-BERT : Authors claim that their proposed model achieves an F1-score of 85.8% on the TREC50 dataset, surpassing Phoneme-BERT's score of 76.6%. \n* Good ablation studies on different fusion mechanisms and backbones along with the effect of using different syntactic parsers to obtain the H_sat representation.\n* Detailed evaluation and analysis on multiple benchmark datasets. (ATIS, TREC)\n* An novel attempt to incorporate syntactic information into pre-trained language models to improve intent classification in SLU systems.", "reasons_to_reject": "* Insufficient Analysis: The paper lacks in-depth analysis and discussions of the model's limitations, potential failure cases, and practical challenges. A thorough examination of these aspects is essential to understand the boundaries and implications of the proposed approach.\n* Methodological Weaknesses: The paper does not provide a comprehensive analysis of the choices made in the model design, such as the dual context fusion module. A lack of in-depth analysis may raise questions about the robustness and reliability of the proposed approach.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces Syntax-aware Attention (SAT), to an intent detection model and claim that this can help Spoken Language Understanding (SLU) systems in task-oriented dialogue systems. By incorporating syntactic information and employing various feature fusion methods, the proposed SAT is shown to empirically achieve substantial improvements and outperform state-of-the-art models on benchmark datasets (ATIS, TREC).", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZUqvicHGxi", "reviewer": "Reviewer_ajUT", "summary": "", "questions": "1. From L77-L80, do you mean we obtain the encoded feature representation from the backbone model and feed [CLS] embedding in SATM module?\n2. Have you tried to include the mask in the first backbone model directly?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our heartfelt appreciation for the thorough reviews you have provided. We value your insights and would like to offer detailed responses to your concerns:\n\n__R1:__\nThank you for your feedback on the perceived limited novelty of our work and the need for a broader baseline comparison. We value your comments and agree that more comprehensive comparisons would strengthen the presentation of our work. To clarify the novelty, we would like to mention that our work represents one of the first attempts at incorporating a syntax-aware attention method into spoken language understanding research. Furthermore, we introduce a novel integration technique, DCFM (Dual Context Fusion Module). This is a unique contribution of our work which differentiates our approach from existing ones. The results presented in the table will provide evidence that our proposed SAT (syntax-aware attention) and DCFM technique successfully improve the task's performance, thereby asserting the effectiveness and benefits of our approach.\n|Method|TREC6|TREC50|ATIS|\n|-|-|-|-|\n|K-SAN [1]|84.6|76.4|94.2|\n|BERT-SIM [2]|85.1|76.9|94.6|\n|CapuBERT [3]|85.3|77.8|94.9|\n|SG-net [4]|85.2  |77.6|94.7|\n|Syntax-BERT [5]|85.5  |78.2|95.0|\n|__SAT__|__86.4__ |__79.0__|__95.4__|\n\n__R2:__\nWe appreciate the reviewer's feedback about the need for additional references to support the novelty and usefulness of our proposed framework. In response to this concern, we will include the provided references in the main table and perform analysis and comparison in the revised version. We will also enrich the referencing in our paper by including citations that highlight the foundational works in the field and contextualize our framework's positioning within the existing literature. We appreciate the reviewer's vigilance in evaluating the scholarly rigor of our paper and are committed to addressing this concern in the revised version.\n\n__Q1:__\nThank you for requesting clarification on the feature representation. We apologize if our initial explanation was unclear. Here, we provide a more detailed description of how we obtain and utilize the encoded feature representation in our proposed method:\n1. First, we pass the input through the backbone model, which is responsible for encoding the input data into a feature representation.\n2. From the encoded feature representation obtained in the previous step, we extract the [CLS] embedding, which serves as a summarization of the input's contextual information.\n3. This [CLS] embedding is then used as input to our proposed Syntax-Aware Transformer Module (SATM).\n\nBy following these steps, we ensure a structured and streamlined integration of feature representation in our approach. The SATM leverages the [CLS] embedding, which is a condensed representation of the input's syntactic and semantic information, to make informed decisions and improve model performance.\n\n__Q2:__\nThank you for raising this question. The integration of the mask in the backbone model can be found directly in the second row of each backbone model in Table 1, on Line 221 of Page 4. To address your concerns and provide a clear understanding of our contribution, we present the results here again.\n\nThese results demonstrate the effectiveness of integrating the mask directly into the backbone model and showcase the improvements our proposed framework brings to the attention-based models. By providing these findings, we aim to highlight the feasibility and practicality of our approach and its potential impact on the field.\n|Pre-trained Model|Syntax Information|Fusion Module|TREC6| | | |TREC|50| | |ATIS| | | |\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n| | | |Acc|F1|P|R|Acc|F1|P|R|Acc|F1|P|R|\n|BERT|✗|✗|83.9|81.1|79.9|82.6|   75.8|57.8|62.5|58.9|   94.4|77.6 |75.2|86.8|\n|BERT|✓|✗|82.8|80.1|78.4|82.7|   76.2|58.3|61.2|58.6|   94.6|79.4|77.2 |85.9|\n|RoBERTa|✗|✗|84.1|82.1|82.5|83.4|   76.0|59.9|60.1|61.5|   94.5|79.8|77.8 |86.3|\n|RoBERTa|✓|✗|83.2|81.5|79.1|82.7|   76.4|60.1|62.2|64.0|   94.8|81.5|79.2 |87.0|\n\n__References__\n\n[1] Chen et al. 2016 \"Knowledge as a Teacher: Knowledge-Guided Structural Attention Networks\"\n\n[2] Xia et al. 2021 \"Using Prior Knowledge to Guide BERT’s Attention in Semantic Textual Matching Tasks\"\n\n[3] Nguyen et al. 2022 \"Improving Spoken Language Understanding by Enhancing Text Representation\"\n\n[4] Zhang et al. 2019 \"SG-Net: Syntax Guided Transformer for Language Representation\"\n\n[5] Bai et al. 2021 \"Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is easy to follow and demonstrates the effectiveness of adopting syntactic information to aid the spoken language understanding tasks.\n2. Extensive experiments have shown how different fusion mechanisms and syntactic masks may affect the intent detection performance.", "reasons_to_reject": "1. The novelty is relatively limited where the benefits of external syntactic information and using it as a mask in attention have been explored in previous works such as [1-3]. The technique is the extensive use of combination of existing approaches. However, the baselines such as knowledge attention or other approaches of syntactic information inclusion are not compared. \n2. Some references are lacking to support the novelty and usefulness of the proposed framework.", "questions_to_authors": "1. From L77-L80, do you mean we obtain the encoded feature representation from the backbone model and feed [CLS] embedding in SATM module?\n2. Have you tried to include the mask in the first backbone model directly?", "paper_topic_and_main_contributions": "The paper proposes to introduce a syntax-aware mask to guide the attention of each word in transformers toward their syntactic ancestors from a dependency parsing tree. Syntactic information from a pretrained parser will produce a mask to constrain the attention between words and their corresponding ancestors only. The experiments show the benefit of syntactic information in BERT-based models in intent detection while more baselines and references are suggested to support the novelty and usefulness of the proposed framework.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Chen et al. 2016 “Knowledge as a Teacher: Knowledge-Guided Structural Attention Networks”\n[2] Xia et al. 2021 “Using Prior Knowledge to Guide BERT’s Attention in Semantic Textual Matching Tasks”\n[3] Nguyen et al. 2022 “Improving Spoken Language Understanding by Enhancing Text Representation”\n"}}]

Author rebuttal:

Meta-review: This paper proposes to restrain the attention of a tranformer LM based on syntax. They show that this improves performance across multiple SLU datasets. The approach is easily used in many LMs (as long as they use attention), and the paper is well written. The reviewers mentioned multiple competing approaches that aim to incorporate syntax into LMs; which are not compared to (both descriptive and performance). Furthermore, ablation studies and qualitative analysis are not included; limiting the conclusions we can draw from the current paper (it would probably fit better as a long paper including the missing parts).

PS: I think SLU is a very misleading name; as it is not speech, and it is not fully understanding either. I  would suggest to refer to this as intent detection or intent classification
Decision: Accept-Findings