DemoNSF: A Multi-task Demonstration-based Generative Framework for Noisy Slot Filling Task
Authors: Guanting Dong
Tingfeng Hui
Zhuoma GongQue
Jinxu Zhao
Daichi Guo
Gang Zhao
Keqing He
Weiran Xu
Venue: EMNLP 2023 Findings (2023)
Keywords: Noisy Slot Filling
Input Perturbations
Multi-task Learning
Generative Framework
Large Language Model
Demonstration Learning
TL;DR: In this paper, we propose a multi-task demonstration-based generative framework for noisy slot filling task, named DemoNSF, which includes three noisy auxiliary tasks and a novel noisy demonstration retrievel strategy.

Abstract: Recently, prompt-based generative frameworks have shown impressive capabilities in sequence labeling tasks. However, in practical dialogue scenarios, relying solely on simplistic templates and traditional corpora presents a challenge for these methods in generalizing to unknown input perturbations. To address this gap, we propose a multi-task demonstration-based generative framework for noisy slot filling, named DemoNSF. Specifically, we introduce three noisy auxiliary tasks, namely noisy recovery (NR), random mask (RM), and hybrid discrimination (HD), to implicitly capture semantic structural information of input perturbations at different granularities. In the downstream main task, we design a noisy demonstration construction strategy for the generative framework, which explicitly incorporates task-specific information and perturbed distribution during training and inference. Experiments on two benchmarks demonstrate that DemoNSF outperforms all baseline methods and achieves strong generalization. Further analysis provides empirical guidance for the practical application of generative frameworks. Our code is released at https://github.com/dongguanting/Demo-NSF.

DemoNSF: A Multi-task Demonstration-based Generative Framework for Noisy Slot Filling Task
Guanting Dong',, Tingfeng Hui Zhuoma GongQue', Jinxu Zhao', Daichi Guo',
Gang Zhao', Keqing He², Weiran Xul*
Beijjing University of Posts and Telecommunications Beijing, China Mcituan, Beijing, China (dongguanting, www.tiiiiiineeee e xuweiran edu. cn
Abstract
Original Umteranry
kppt:
Food New York City by 12:00
1dbel:
ORCANIAA ATOOLUI LLRARIIVE
Recently, prompt-based generative frameworks have shown impressive capabilities sequence labeling tasks. However, in practical dialogue scenarios, relying solely on simplistic tem- plates and traditional corpora presents a chal- lenge for these methods in generalizing to un- known input perturbations.
Portorhations
Teoearkmttdd Dodey System
lxpe: :
Ploe go NVC
Eabi :
NSC NAME, 15 ARRIVE
Smpply
Teed New Your City tedberihits OHUUANNAA AUN,
Label
M.NIEE
lnpec: laad antin Now Yoark City 12.30.
Label: 1h ARROOS in name
Typos
address this gap, propose multi-task demonstration- based generative framework for noisy slot fill- ing, named DemoNSF. Specifically, we in- troduce three noisy auxiliary tasks, namely noisy recovery (NR), random mask (RM), and hybrid discrimination (HD), to implic- itly capture semantic structural information of input perturbations at different granulari- ties. In the downstream main task, de- sign a noisy demonstration construction strat- egy for the generative framework, which ex- plicitly incorporates task-specific information and perturbed distribution during training and inference. Experiments on two benchmarks demonstrate that DemoNSF outperforms all baseline methods and achieves strong gener- alization. Further analysis provides empirical guidance for the practical application of gen- erative frameworks. Our code is released at http://gghhbb.com/ddoggunnngggg
Figure The impact of diverse input perturbations on the slot filling system in real scenarios.
When faced with the uncertainty and diversity of human language expression (Wu et al., 2021). these perturbations significantly impact the SF model's generalization ability, thereby hindering its appli- cation in practical dialogue scenarios.
In real dialogue systems, models often encounter a wide range of input perturbations and errors made by humans. As illustrated in Figure 1, users may in- teract with the dialogue system in ways that deviate from the standard input format and even simplify their queries convey the same intent, all due to di- verse human expression habits. Furthermore, errors originating from the upstream input system may in- troduce disturbances to the downstream model (e.g. Typos from keyboard input, Speech errors from ASR systems). Existing slot filling models are typ- ically pre-trained and fine-tuned on perturbation- free datasets, leading to decreased performance when confronted with such situations.
1 Introduction
The slot filling (SF) task in the goal-oriented dialog system aims to identify task-related slot types in certain domains for understanding user utterances. Recently, traditional discriminative and generative models (Liu and Lane, 2015, 2016; Goo et al., 2018; Niu al., 2019; He et al., 2020a; Yan et al., 2021a; Wang et al., 2022b; Hao et al., 2023) have shown remarkable ability in slot filling. Despite their powerful capabilities, the high performance of these models heavily depends on the consistency of data distribution between the training and test sets.
Recently, existing studies (Wu et al., 2021; Moradi and Samwald, 2021a; Gui et al., 2021) have explored the issue of robustness. However, these methods are mainly designed for particular pertur- bations, limiting generalization ability to unknown perturbations. To capture the noisy semantic struc- ture, PSSAT and CMDA (Dong al., 2022a; Guo et al., 2023) further introduce additional corpus and generative models. Nevertheless. this approach carries the risk of introducing extra noise and in- creasing computing resource consumption. While Large language models (Brown et al., 2020; Tou-
TThe first two authors contribute equally. Weiran Xu is the corresponding author.

---

vron et al., 2023; OpenAl, 2023) and prompt-based methods (Lu al., 2022; Xie al., 2022b) have achieved strong performance on information ex- traction, the exploration of these generative frame- works on diverse input perturbations remains a blank area, hindering their application in realistic task-oriented dialogue systems.
2 Method
In this section, we introduce the overall frame- work of our proposed DemoNSF. We first briefly describe the problem definition against input per- turbations in the slot filling task. Next, we propose three distinctive noisy auxiliary tasks for diverse perturbations. Finally, we present novel noisy demonstration construction strategy. We will intro- duce these the following subsections!.
To address this limitation, we propose a multi- task demonstration--asse generative framework for noisy slot filling tasks, named DemoNSF. Specifi- cally, we design three noisy auxiliary tasks, namely noisy recovery (NR), random mask (RM), and by- brid discrimination (HD), to boost the performance against input perturbations in different levels. NR aims to capture the mapping relationship between fine- fine-grained noisy and clean data. RM implic- itly Iearns the slot entity distribution of perturbed data during the process of mask infilling. HD as- sists generative models consider global information while implicitly capturing the semantic character- isties unique to perturbed data. In the downstream process, we formulate the SF task as a sequence-to- sequence generation guided by noisy task demon- strations. In detail, DemoNSF selects semanti- cally similar example for each query from noisy candidate pool, converts it into natural demonstra- tion sentence, and encodes the demonstration along with the input text by integrating noisy semantic in- formation. With the boost of noisy auxiliary tasks and demonstrations, DemoNSF learns the semantic structure of perturbations from both explicit and implicit levels. Our contributions are three-fold:
2.1 Problem Definition
Given an input utterance X and its corresponding slot type set S {s,, Sm , the slot filling task aims to extract all the entities in X. For the noisy slot filling task, we formulate the input perturbation process in the real scenario as (X', S') = P(X,S) The model's 's ro- bustness is evaluated on the perturbed test dataset {(X', Y}} but with no access to the input pertur- bation process (-)) or perturbed data during the training phase. In this paper, We use Delean D Dang. and Diest to denote clean data, augmented data, and test data.
INN
2.2 Multi-level Data Augmentation
Figure 2 demonstrates how we construct our noisy candidate pool using NLPAug (Ma, 2019), enabling the input utterance of the clean training set into an augmented dataset comprising three distinct levels: character-level. word-level, and sentence-level augmentation. Specifically, at the character level, incorporate random operations such as character addition, deletion, and substitution within token, governed by probability parameter denoted as p. Moving to the word level, we introduce random word deletion, insertion, and replacement, along with the substitution of words with homophones within a sentence, again governed by the proba- bility parameter p². Furthermore, at the sentence level, we substitute sentences with synonymous alternatives.
1) To the best of our knowledge, we are the first to comprehensively investigate the effects of di- verse input perturbations generative frameworks in slot filling tasks and further validate the vulnera- bility of existing prompt-based generative methods when confronted with different human expressions.
2) We propose a simple but unified multi-task demonstration-baset generative framework, which includes three novel noisy auxiliary tasks and a noisy demonstration construction strategy, to en- hance the model's robustness and adaptability to perturbed inputs in
2.3 Noisy Auxiliary Tasks
The performance of the noisy slot filling task highly depends on the prior knowledge of the distribution of input perturbations. In this section, we introduce three novel noisy auxiliary tasks:
dialogue scenarios.
3) Experiments on two benchmarks demonstrate that our method outperforms all baseline methods and achieves strong generalization. The extensive analysis also provides empirical guidance for the practical application of generative frameworks.
Noisy Recovery (NR). Given character-level
augmented utterance char {x1, 22, .... 2u1
'rainingg, and Inference can found Appendix 3p is an empirical parameter, we set it to 0.3

---

Data Augmentation
Noisy Auxiliary Task Pre-training
Original Sentence: How about masum??
Label: [museum) is [TYPE) slot.
Chenster-leeel
Recovery
Output:
Aug. sag.laput: How about a mAkspum?
How about museum?
Clean
Word-level
Augmeented
Instruction:
Randomn Mask
Generative Model
Outgut:
Ang.Input: How about a Sextra, Id=!
The mask token is muscum.
Instruction: Hybeid Discriminzionn
Output:
Aug.Iaput: What going the muscum?
sentenet-leve nois
Demonstrations:
Demonstration-based Fine-Tuning
Augmented cltls
SBERT
[cinema] is [TYPE) slot.
What about cinena!
Demonstrations: [einema] is [TYPE] slot.
Generative Model
[ouseeun]] is [TYPR] slot.
Clean
Input Utterance:
Input How about a museum!
dala
How about musoum??
Figure 2: The overall architecture of our proposed approach DemoNSF.
..., NN where Pm kay represents the augmented to- ken with character-level augmentation, the objec- tive of the NR task, as illustrated in Figure 2, is to restore X to its corresponding clean utter- ance X. This task enables the model to capture the mapping relationship between fine-grained input perturbations and their clean equivalents, thereby enhancing the model's capability to represent fine- grained noisy data. Hence, the loss function can be formulated as:
Daug to create a mixed dataset, denoted as Dmiz- We randomly select utterances from Dmir and as- sign distinct labels based on whether the chosen utterance is clean or has different levels of pertur- bation. As shown in Figure 2, the generative model can implicitly capture the unique semantic distri- bution of perturbed data while considering global information by discriminating between inputs with and without perturbation. The loss function LHD is the same as LNR-
N X.Sii,X Xoug change)
Therefore, the overall loss function 1. is defined
Σ LNR = B
CE(X,N,
- 3LRM + LLHD
(3)
where B and denote the batch size and sequence length, respectively.
Random Mask (RM). Inspired by the concept masked language modeling (MLM) introduced in BERT (Devlin et al., 2019), we present our random mask-filling task. Specifically, given an utterance in Daugi we randomly mask one en- tity with the special [MASK] symbol, resulting in Xaug
where 8, and y represent the weights of NR. RM, and HD task loss functions, respectively.
2.4 Noisy Demonstration Construction
Different from prior Semonstration-aassee work (Min et al., 2022), select examples from Daug instead of Delean for each input X to incorporate perturbed semantic information into the model. For retrieval, we employ SBERT (Reimers and Gurevych, 2019) which independently produces [CLS) embeddings for both X and s, and compute their similarity scores to rank s. Subsequently, we select the top-k examples to construct the noisy demonstrations X and concatenate them with the input X to form the complete input [X; X). Our demonstration template is shown below: "Demonstrations:: [Retrieved Noisy Utter- ances]. [Text Span] is [Slot Type]. Input Utter- ance: [Original Input]."
"osk = {z ..... [MASK], ...,,.;,, ...,..) We aim restore the [MASK] token to its original value. The mask-filling procedure enables the model to implicitly incorporate the semantic distribution of slot entities within the perturbed data. Hence, the loss function of the RM task can be defined as:
1 B
LRM = Σ CElym: P(IMASKI))
(2)
B
3=1
where ym denotes the original token and (MASK] represents the logits of [MASK] token.
Hybrid Discrimination (HD). To further ad- dress coarse gaasee-raiied input perturbations, we pro- pose the HD task. In detail, we combine Delean and

---

Sentence-level
Character-leve
Word-level
Methods
Clean
Perturbed-Arg.
Verbose
Paraphrase
Simplification
Typos
Speech
GPT2
95.37
80.52
85.66
82.98
60.19
77.78
77.43
BART
95.28
77.87
82.72
82.95
53,90
73.57
74.20
TS
95.49
81,34
89.13
83.73
62.43
81.13
79.55
BARTNER
94,88
78.00
88.55
85.04
65.37
72.65
77.93
LighNNER
87.65
84.90
71.61
76.14
InstroctionNER
95.67
81.37
88.45
85.29
65,34
80.13
80.16
DemoNSF(GPT2)
95.66(10.44
81 0..3)
(7.6331111
87 07.02((1022)
69.75(110.)
86.31(10.7)
02.53(00.55
DeMANSF(BART)
95.7
08.2)9(-088)
06.01((0.7)
05.60(10.3)
82.48(+00.)
80.24(=1.1)
DcmoNSF(T5)
95.72(=0.55
89.98(=1.1) (2.27
(9.79)
76.63(20..5
87.55(10.77
85,20(=0.9)
Table 1: scores with standard deviations under different input perturbations on RADDLE.
CharsWord
Word-Sent
Clur- WSrd
Ap Demsdanbi
Methods
Eat.4Suh.
App.
App-+Sal.
App.
BART
58.00
47:27
50.28
38,
$7.44
36.79
73.47
47.93
54.83
Lig
42.34
45.44
27.00
16 Dincc
InsractioNER
$7.87
38.59
74.45
50.75
(BRR(ART)
60.27
51
(1.39
68
(8.72(=10.5)
44.27(#999))
Ours(TS)
63.5%
63,94
(..7)
55.12(10..)
121
10
Table 2: Fl scores with standard deviations under kinds of mixed perturbations on SNIPS.
Nummbersa))
Figure 3: Performance comparison of different types of demonstrations
3 Experiment
(2) DemoNSF(T5) shows remarkable superior- ity under different kinds of input perturbations while maintaining the best performance on the clean test data. For the fine-grained perturbations our method achieves a significant improvement of 11.29 11.99 and 7.42% in Typos and Speech com- pared with InstructionNER. DemonNSF maintains strong performance for coarse sarrse-rraiedd perturba- tions, especially with a 4.2% improvement in Sim- plification. These results clearly demonstrate that DemonNSF effectively captures the mapping rela- tionship between fine-grained noisy and clean data while also considering the ability to generalize to coarse-grained global semantic perturbations.
3.1 Datasets
Based on RADDLE (Peng et al., 2021) and SNIPS (Coucke et al., 2018). we adopt the evaluation set provided by Dong et al., which includes two differ- ent perturbation settings. For single perturbations setting, we include five types of noisy utterances (character-level: Typos, word-level: Speech, and sentence-level: Simplification, Verbose, and Para- phrase) from RADDLE. For mixed perturbations setting, we utilize TextFlint (Gui et al., 2021) to introduce character-level perturbation (EntTypos), word-level perturbation (Subword), and sentence- level perturbation (AppendIrr) and combine them get mixed perturbations dataset?.
(3) DemonNSF is a plug-and-play method that can achieve good results on different back- bones. Specifically, we replace the backbone with BART/GPT and also get the good performances compared with the corresponding baseline. The re- sults of the backbone ablation further demonstrate our approach remarkably enhances the robustness of generative models when facing perturbations.
3.2 Main Results.
Table shows the main results of DemoNSF and comparison baselines under single perturbation setting. We make the following observations:
(1) When faced with perturbations of different granularities, generative models suffer severe per- formance degradation, especially in Typos (GPT2: 35.18% BART: 41.38% T5: 33.06%) and Speech (GPT2: 17.57% BART: 21.71% % T5: 14.36%), which indicates that generative models have poor robustness against fine-grained perturbations.
3.3 Mixed Perturbations Scenario.
In real dialogue scenarios, mixed perturbations of- appear in one utterance simultaneously. To fur- ther verify the effectiveness of DemoNSF in more realistic scenarios, we conduct the mixed perturba- tions experiment. As shown Table 2, DemoNSF significantly outperforms other baselines all
'Due to space limitations, detailed experimental settings (Baselines, Datasets..) can he found in the Appendix B.I.

---

Sentence-level
Character-leve
Word-level
Methods
Clean
Perturbed-Avg
Verbose
Paraphrase
Simplification
Typos
Speech
Text-davinci-003
43.09
34.26
39.34
38.42
40.12
37.18
38.54
ChaiGPT
71.43
40.65
60.00
55.56
65,54
55.56
57.21
CharGPT Clean Demos
71.31
61.01
57.81
53.43
65.03
61.71
62.32
CharGPT Aug Demos
68.21
65.04
70.56
58.82
73.03
63.77
68.34
CharGPT Mixed Demos
76.92
58.73
68.26
58.61
74.19
57.78
65.36
Table 3: The evaluation on ChatGPT under 5 different input perturbations on Raddle.
two-level perturbations, especially achieving over 63% Fl-score in fine-grained mixed perturbations. Even with the joint interference of 3 perturbations, DemoNSF can still maintain 4.37% improvement compared with baseline, which further validates the stability of DemoNSF in challenging settings.
Text-davinci-003 perform poorly on diverse input perturbations, which far behind the finetune SOTA methods (DemoNSF, Instruction NER) presented in Table 1. The possible reason is that large lan- guage models are usually pre-trained the large- scale general training corpus, making difficult adapt well specific domain perturbation data in a zero-shot setting. (2) Compared with baselines and traditional clean demonstration retrieval meth- ods, selecting instances from both the augmented and mixed demonstration candidate pools can sig- nificantly improve the overall performance. This finding is consistent with our conclusion in the sec- tion 3.4, proving the effectiveness of incorporating noisy semantic structures in addressing input per- turbations. (3) From the perspective of different perturbations, both two types of noisy demonstra- tion strategies show significant improvements fine-grained perturbations (over 8% improvement in Typos). However, the improvement is not ob- vious in coarse-grained perturbations, especially in speech errors. This phenomenon indicates that noisy demonstrations are more suitable for fitting the distribution of fine-grained perturbations, while there is still much room for improvement in coarse- grained perturbations that severely disrupt the con- textual semantics and slot mentions of the original input. This finding poses further challenges for exploring the robustness of large language models, which will also be the focus of our future research.
3.4 Impact of Different Demonstrations.
Figure 3 shows the impact of the number of dif- ferent types of demonstrations under single per- turbations. We have the following findings: (1) DemoNSF exhibits a significant performance gain with only two augmented samples while perfor- mance severely decreases as the number increases. This may be because diverse augmented instances can help the model explicitly fuse noisy seman- tic distribution (Xie et al., 2022a) while the sam- ple diversity exceeding certain threshold may even bring additional noise. (2) Clean demonstra- tions only bring slightly improves as the number increases, which indicates that clean samples only provide some task general information(e.g. entity distributions, slot-value mapping) for prompting. (3) Retrieved demonstrations from the mixed data pool show a stable performance gain, which fur- ther confirms the mutual promotion between noisy semantic distribution and task general information, and provides guidance for the robustness of prompt- based generative models.
3.5 The ICL Evaluation on ChatGPT.
order to further validate the effectiveness of our noisy demonstration strategy on the large-scale generative framework, we conduct experiments on ChatGPT and Text-davinci-003 (Brown et al., 2020). We directly use them to do inference based on in-context learning (ICL) (Dong et al., 2022b; Brown et al., 2020; Min et al., 2022) on RADDLE, which means language models make predictions only based the conditioned demonstration ex- amples without any training samples.
Conclusion
In this paper, we propose a unified multi-task demonstration-basee generative framework for noisy slot filling tasks. Specifically, we introduce three novel noisy auxiliary tasks and noisy demon- stration construction strategy for the generative framework, which aims to learn the semantic struc- ture of perturbations from both explicit and implicit levels. Experiments on two benchmarks show the effectiveness of DemoNSF, Further analysis pro- vides empirical guidance for the practical applica- tion of the generative framework.
Table 3 illustrates the overall results of Chat- GPT under 5 different single perturbations. We draw the following findings: (1) ChatGPT and

---

Limitations
perturbation-robut slot filling. In Proceedings of the 29th International Conference on Computational Linguistics, pages 5327-5334, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
order to capture semantic structural information of input perturbations at different granularities, we introduce three novel noisy auxiliary tasks in the pre-training stage, which may consume more GPU memory than traditional methods. This drives us to further improve the overall memory efficiency of the framework. Also, our method mainly focuses on the slot filling task. However, we believe is possible to extend our work to other scenarios, such as sho-shot settings, and zero-shot settings. We also reserve them for our future research.
Guanting Dong, Zechen Wang. Liwen Wang, Daichi Guo, Dayuan Fu, Yuxiang Wu, Chen Zeng, Xuefeng Li, Tingfeng Hui, Keqing He, Xinyue Cui, Qixiang Gao, and Weiran Xu. 2023a. prototypical semantic decoupling method via joint contrastive learning for few-shot named entity recognition. In ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5.
Guanting Dong. Zechen Wang, Jinxu Zhao, Gang Zhao, Daichi Guo, Dayuan Fu, Tingfeng Hui. Chen Zeng. Keqing He, Xuefeng Li, Liwen Wang. Xinyue Cui, and Weiran Xu. 2023b. A multi-task semantic decom- position framework with task-specific pre-training for few-shot ner.
References
Ankur Bapna, Gokhan Tur, Dilek Hakkani- Turr, and Larry Heck. 2017. Towards zero-shot frame se- mantic parsing for domain scaling, arXiv preprint 177iv:170..02363.
Guanting Dong, Jinxu Zhao, Tingfeng Hui, Daichi Guo. Wenlong Wang, Boqi Feng, Yueyan Qiu, Zhuoma Gongque. Keqing Hc, Zechen Wang, and Weiran Xu. 2023c. Revisit input perturbation problems for llms: A unified robustness evaluation framework for noisy slot filling task. In Natural Language Processing and Chinese Computing, pages 092-694, Cham. Springer Nature Switzerland.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1197-1901.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun. Jingjing Xu, and Zhi- fang Sui. 2022b. A survey for in-context learning. arXiv preprint arXiv:2301.000 0.0244..
Xiang Chen, Lei Li, Shumin Deng, Chuanqi Tan, Changliang Xu, Fei Huang. Luo Si, Huajun Chen, and Ningyu Zhang. 2022. Lightner: lightweight tuning paradigm for low-resource ner via pluggable prompting.
Anjie Fang, Simone Filice, Nut Limsopatham, and Oleg Rokhlenko. 2020. Using phoneme representations to build predictive models robust to asr errors. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval, pages 699-708.
Xiang Chen, Ningyu Zhang. Lei Li, Xin Xie, Shumin Deng, Chuanqi Tan, Fei Huang, Luo Si, and Hua- jun Chen. 2021. Lightner: A lightweight genera- tive framework with prompt-guided attention for low- resource ner.
Chih-Wen Goo, Guang Gao, Yun-Kai Hsu, Chih-Li Huo, Tsung- Chich Chen, Keng- Wei Hsu, and Yun-Nung Chen. 2018. Slot-gated modeling for joint slot filling and intent prediction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 2 (Short Papers), pages 753-757.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Calt- agirone, Thibaut Lavril, Maël Primet, and Joseph Dureau, 2018 Snips voice platform: an embedded spoken language understanding system for private- by-design voice interfaces.
Jacob Devlin, Ming- Mei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume 1 (Long and Short Papers), pages 4171- 4186.
Karthik Gopalakrishnan, Behnam Hedayatnia, Long- shaokan Wang. Yang Liu, and Dilek Hakkani- 2020. Are neural open-domain dialog systems robust to speech recognition errors in the dialog history? an empirical study.
Tao Gui, Xiao Wang, Qi Zhang, Qin Liu, Yicheng Zou, Xin Zhou Rui Zheng, Chong Zhang, Qinzhuo Wu, Ji- acheng Ye, Zexiong Pang, Yongxin Zhang, Zhengyan Li. Ruotian Ma. Zichu Fei, Ruijian Cai, Jun Zhao. Xingwu Hu, Zhiheng Yan, Yiding Tan, Yuan Hu, Qiyuan Bian, Zhihua Liu, Bolin Zhu, Shan Qin, Xi- aoyu Xing. Jinlan Fu, Yue Zhang, Minlong Peng, Xi- aoqing Zheng, Yaqian Zhou, Zhongyu Wei, Xipeng
Guanting Dong, Daichi Guo, Liwen Wang, Xuefeng Li, Zechen Wang, Chen Zeng, Keging He, Jinzheng Zhao, Hao Lei, Xinyue Cui, Yi Huang, Junlan Feng, and Weiran Xu. 2022a. PSSAT: A perturbed se- mantic structure awareness transferring method for

---

Qiu, and Xuanjing Huang. 2021. Textflint: Unified multilingual robustness evaluation toolkit for natural language processing.
Mingda Li, Weitong Ruan, Xinyue Liu, Luca Soldaini, Wael Hamza, and Chengwei Su. 2020b. Improving spoken language understanding by exploiting asr n- best hypotheses.
Daichi Guo, Guanting Dong, Dayuan Fu, Yuxiang Wu,
Chen Zeng. Tingfeng Hui, Liwen Wang, Xuefeng Li, Zechen Wang. Keqing He, Xinyue Cui, and Weiran Xu. 2023. Revisit out-of- vovcabulay problem for slot filling: A unified contrastive frameword with multi- level data augmentations. ArXiv, abs/2302.
Xiaozhuan Liang. Ningyu Zhang, Siyuan Cheng, Zhen Bi, Zhenru Zhang, Chuanqi Tan, Songfang Huang, Fei Huang. and Huajun Chen. 2022. Contrastive demonstration tuning for pre-trained language mod- els. arXiv preprint arXiv:2204.00 04392.
Xia Hao, Lu Wang, Hongmei Zhu, and Xuchao Guo. 2023. Joint agricultural intent detection and slot filling based enhanced heterogeneous attention mechanism. Computers and Electronics in Agricul- lure, 207:107756.
Bing Liu and Ian Lane. 2015. Recurrent neural network structured output prediction for spoken language derstanding. In Proc. NIPS Workshop on Machine Learning for Spoken Language Understanding and Interactions.
Keqing He, Yuanmeng Yan, and Weiran Xu. 2020a. Learning to tag oov tokens by integrating contextual representation and background knowledge. In Pro- ceedings of the 58th Annual Meeting of the Associa- tion for Compulational Linguistics, pages
Bing Liu and Ian R. Lane. 2016. Attention-based recur- rent neural network models for joint intent detection and slot filling. INTERSPEECH.
Jiachi Liu, Liwen Wang. Guanting Dong. Xiaoshuai Song. Zechen Wang, Zhengyang Wang, Shanglin Lei. Jinzheng Zhao, Keqing Hc, Bo Xiao, et al. 2023, Towards robust and generalizable training: An empir- ical study noisy slot filling for input perturbations. arXiv preprint arXiv:2310.033
Keging He. Jinchao Zhang. Yuanmeng Yan, Weiran Xu, Cheng Niu, and Jie Zhou. 2020b. Contrastive zero-shot learning for cross-domain slot filling with adversarial attack. In Proceedings the 28th Inter- national Conference on Computational Linguisties, pages 1461-1467, Barcelona, Spain (Online). Inter- national Committee Computational Linguistics.
Jiexi Liu, Ryuichi Takanobu, Jiaxin Wen, Dazhen Wan, Hongguang Li, Weiran Nie, Cheng Li, Wei Peng, and Minlie Huang, 2020a. Robustness testing of lan- guage understanding task-orienn dialog. arXiv preprint 2012: 15262.
Keqing He, Jinchao Zhang, Yuanmeng Yan, Weiran Xu, Cheng Niu, and Jie Zhou. 2020c. Contrastive zero-shot learning for cross-domain slot filling with adversarial attack. In Proceedings the 28th Inter- national Conference on Computational Linguistics. pages 1461-1467.
Zihan Liu, Genta Indra Winata, Peng Xu, and Pas- cale Fung. 2020b. Coach: A coarse-to coasse-to-fin fine ap- proach for cross-domain slot filling. arXiv preprint arXiv:2004.1177 11727.
Chao-War Wei Huang and Yun-Nung Chen. 2020. Learn- ing asr-robust contextualized embeddings for spoken language understanding.
Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao. Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2022. Uni- fied structure generation for universal information extraction. In Proceedings of the 60th Annual Meet- ing the Association for Computational Linguistics (Volume I: Long Papers), pages 5755-5772 5772, Dublin, Ireland. Association for Computational Linguistics.
Dong-Ho Lee, Mahak Agarwal, Akshen Kadakia, Jay Pujara, and Xiang Ren. 2021. Good exam- ples make a faster learner: Simple demonstration- based learning for low-resource ner. arXiv preprint 10.05:21
Sungjin
Lee and Rahul Jha. 2019, Zero-shot adaptive transfer for conversational language understanding. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 6642-6649.
Edward Ma. 2019. https///gtthb.coommmeeeewww/////
NIp augmentation.
Tingting Ma, Huiqiang Jiang. Qianhui Wu, Tiejun Zhao, and Chin- Yew Lin. 2022. Decompased meta- learning for few-shot named entity recognition. arXiv preprint arXivv2204.0575)
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: De- noising sequence-to-seqqencce pre-training for natural language generation, translation, and comprchension.
Sewon Min, Mike Lewis, Hannanch Hajishirzi, and Luke Zettlemoyer. 2021. Noisy channel language model prompting for few-shot text classification. arXiv preprint 208.
Mingda Li, Xinyue Liu, Weitong Ruan, Luca Soldaini, Wael Hamza, and Chengwei Su. 2020a. Multi-tas/ learning of spoken language understanding by inte- grating n-best hypotheses with hierarchical attention. In Proceedings of the 28th International Conference on Computational Linguistics; Industry Track, pages
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannanch Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work?
113-123.

---

Milad Moradi and Matthias Samwald. 2021a. Evaluat- ing the robustuess of neural language models to input perturbations. arXiv preprint arXiv:2108, 12237.
Hugo Touvron, Thibaut Lavril. Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
Milad Moradi and Matthias Samwald. 2021b. Evalu- ating the robustness of neural language models to input perturbations. Proceedings of the 2021 Con- ference Empirical Methods in Natural Language Processing.
Jianing Wang. Chengyu Wang, Chuanqi Tan, Minghui Qiu, Songfang Huang, Jun Huang, and Ming Gao. 2022a. SpanProto: A stage pan-based prototyp- ical network for few-shot named entity recognition. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 3466 3476, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Marcin Namysl, Sven Behnke, and Joachim Köhler. 2020. Nat: noise-aware training for robust neural sequence labeling, arXiv preprint arXiv:2005.071 07.000.......
Marcin Namysl, Sven Behnke, and Joachim Köhler. 2021. Empirical error modeling improves robust- ness of noisy neural sequence labeling. Findings of the Association for Computational Linguistics: ACL-
Liwen Wang, Rumei Li, Yang Yan, Yuanmeng Yan, Sirui Wang, Wei Wu, and Weiran Xu. 2022b. Instruc- tionner: A multi-task instruction- generative framework for few-shot ner.
IJCNLP 2021.
Peiqing Niu, Zhongfu Chen, Meina Song, et al. 2019, A novel bi-directional interrelated model for joint intent detection and slot filling. arXiv preprint DrXiv:1907.00390
Liwen Wang. Xuefeng Li, Jiachi Liu, Keging Hc, Yuan- meng Yan, and Weiran Xu. 2021. Bridge to target domain by prototypical contrastive learning and la- bel confusion: Re Re-explore zero-shot learning for slot filling. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9474-9480, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
OpenAl. 2023. Gpt-4 technical report.
Baolin Peng, Chunyuan Li, 7hu Zhang, Chenguang Zhu, Jinchao Li, and Jianfeng Gao. 2020. Raddle: An evaluation benchmark and analysis platform for robust task-oriented dialog systems. arXiv preprint arXiv:2012. 14666.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou, 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.1 11903
Baolin Peng, Chunyuan Li. Zhu Zhang, Chenguang Zhu, Jinchao Li, and Jianfeng Gao. 2021. RADDLE:
An evaluation benchmark and amalysis platform for robust task-oriented dialog systems. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pages 4418 4429, Online. Association for Computational Linguistics.
Di Wu, Yiren Chen, Liang Ding, and Dacheng Tao. 2021. Bridging the gap between clean data training and real-world inference for spoken language under- standing. arXiv preprint
Sang Michael Xie, Aditi Raghunathan, Percy Liang. and Tengyu Ma. 2022a. An explanation of in-context learning as implicit bayesian inference.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 18)::
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong. Torsten Scholak. Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang. Vic- tor Zhong, Bailin Wang, Chengzu Li, Connor Boyle. Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022b. UnifiedSKG:: Unifying and multi-tasking structured knowledge grounding with text-to- text language models. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 602-63 631, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former.
Nils Reimers and Tryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese nett-networks.
Weitong Ruan, Yaroslav Nechaev, Luoxin Chen, Cheng- wei Su, and Imre Kiss. 2020. Towards an asr error robust spoken language understanding system. In INTERSPEECH, pages 901-905.
Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang. and Xipeng Qiu. 2021a. A unified generative framework for various NER subtasks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing
Darsh Shah, Raghav Gupta, Amir A Fayazi, and Dilek Hakkani- Tur.. 2019. Robust zero-shot cross-domain slot filling with example values. arXiv preprint arXiv: 10Xiv:9906.068700

---

(Volume 1: Long Papers), pages 5808-5822, Online. Association for Computational Linguistics.
A Method Details
A.1 Training and Inference
Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang, and Xipeng Qiu. 2021b. A unified generative framework for various ner subtasks.
For the multi-level data augmentation, we utilize NLPAug (Ma. 2019) construct a noisy candidate data pool from the clean data pool at different levels. During the upstream pre-training stage, adopt multi-tast training strategy, where the overall loss function denoted as (13).
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing fhot performance of language models. In In- ternational Conference on Machine Learning, pages 12697-12706. PMLR.
In the downstream demonstration-baase fine- tuning stage, we directly feed the demonstration- based input [X: into the model for predic- tion. During inference,
also employ SBERT (Reimers and Gurevych, 2019) to retrieve task demonstrations for test data from Dangs ensuring consistency between the training and inference stages.
Experiment Details
B.1 More Details of Datasets
Based on RADDLE (Peng et al., 2020) and SNIPS (Coucke al., 2018), we adhere the evalua- tion set provided by PSSAT (Dong et al., 2022a), which includes two settings: single perturbation and mixed perturbation.
For a single perturbation setting, RADDLE crowd-sourced diagnostic evaluation dataset that covers a wide range of real-world noisy texts for dialog systems. PSSAT(Dong al., 2022a) extracts each type of noisy utterance (Typos, Speech, Simplification, Verbose, and Paraphrase) from RADDLE to build the test data pool. Specif- ically, Typos occur due to non-standard abbrevia- tions or keyboard errors, while Speech arises from recognition and synthesis errors produced by ASR systems. Simplification refers to users expressing their intentions using concise words, while Ver- bose represents users employing redundant words to convey the same intention. Paraphrase also prevalent among users who use different words or rephrase the text based on their language habits.
For the multi-perturbationn setting, utilize TextFilint (Gui et al., 2021) toolkit introduce character-level noise (EntTypos), word-level noise (Subword), and sentence-level noise (AppendIrr). We then combine different types of noisy data to construct a detailed introduction of these perturbations can be found in the GitHub repository of TextFlint
evaluation set. The
"htps://www.cctttin.iiccccrt

---

Sentence-level
Character-leve Typos
Word-level Speech
Methods
Perturbed-ANe
Verbose
Paraphrase
Simplification
DemoNSFFBackbone)
95.49
81.34
89.13
83.73
62.43
81.13
79.55
+NR
95.17
81.13
88.96
87.98
67.22
82.61
81.58
+RM
95.30
81.04
89.88
86.78
62.31
85.84
81.17
+MD
95.39
81.37
90.36
87.20
64.61
82.66
81.24
+AF2
95.82
80.79
89.28
86.89
73.74
84.40
83.02
#ChonnDence
95.37
80.28
88.91
85.94
69.00
84.62
81.75
+MixDemos
95,90
80.04
90.71
87.19
71.23
82.60
82.35
+NoisyDemos 95.08
80.04
90.34
85.39
76.41
85.14
83.46
DemoNSF(Fall)
95.72
82.37
89.98
89.49
76.63
87.55
85.20
Table 4: The ablation study results (average Fl score%) on RADDLE. "+" ddnnotes the backbone of DemoNSF with specific module.
B.2 Baselines
generation problem and introduces descriptive in- structions and an optional mechanism to enable the model to understand different tasks and constrain the output space.
In this paper, we foeus on comparing DemoNSF with multiple state-of-the-art baselines that use generative framework, as shown below:
GPT2 (Radford et al.. 2019) is a decoder-only framework model developed by OpenAl 5, It is designed to generate human-like text by predict- ing the next word in given sequence. GPT-2 has gained popularity for its impressive ability to gen- erate coherent and contextually relevant text across various domains and has been used for tasks like text completion, translation, and creative writing. BART is a sequence-tor to-seneee----eeeeeeeeeee model architec- ture introduced by Lewis et al. (2019). It combines both autoregressive and denoising objectives dur- ing training to learn robust representations of input sequences.
B.3 Implementation Details
In the upstream pre-training stage, we set the batch size to 32, and our pre-training process typically takes around I hour for 5 epochs. In this paper, we conduct all the experiments without any hy- perparameter search. For the multi-task training strategy, we assign equal weights to three noisy auxiliary tasks, i.e., a, B3, and to The responding learning rates are set le-5. For the demonstration- based fine-tuning stage, we also set the batch size to 32 and the training takes an av- erage of hours for epochs, while the learning rates are set to 5e-5. For the selection of demonstra- tions, we recall the top 2 instances with the highest similarity score from the noisy candidate pool.
T5 pre-training model proposed by Raffel al. (2020). utilizes the transformer architecture and transforms various natural language processing tasks into text-to-text transfer tasks.
BARTNER (Yan et al., 2021b) is pointer-based architecture designed for scquence-to-sequeee NER subtasks into unified NER tasks. It converts sequence generation task by predicting entities and their corresponding type indexes in the input sen- tence.
In all experiments, we train and test various methods using NVIDIA RTX A6000 GPU. To select the best model, we evaluate the perfor- mance on the validation set using the F1 metric every 400 training steps. Experiments in Table 2 and Table use base-version of T5 and BART, while we also adopt the large-version model in Ta- ble 5 on single perturbation setting. We retrieve Clean Demos from Delenn Noise Demos from D Daug and M Demos from Dmiz for the ablation study and the experiments on investigating the im- pact of the number of demonstrations. We will release our code after a blind review.
LightNER (Chen al., 2022)
phinter-bssed scquence-to-sequenee model which builds upon the BARTNER. introduces prompt tuning tech- nique that incorporates additional parameters into the attention mechanism.
InstructionNER (Wang al., 2022b) is multi- task instruction baased generative framework specif- ically designed for addressing few-shot NER tasks. It redefines the NER task as natural language
hhtps://opeaai.com

---

Sentence-level
Character-level
Word-level
Methods
Clean
Perturbed-AAgg
Verbose
Paraphrase
Simplification
Typos
Speech
BART
95.21
79.23
87.20
83.81
57.79
75.65
76.74
T5
95.58
82.12
88.36
85.98
68.25
81.31
BARTNER
95,30
79.96
90.44
87.24
75.32
76.22
81.84
LighNEER
96.02
80.32
90.40
87.97
67.28
75.57
80.31
InstructionNER
82.01
69.75
DemoNSF(BART)
95
81.25(=0.7)
10,56(==0.1)
88. 12(5:055)
74.20(=0.7)
84.58(2:0.3) 3..3
83. 83.74(=0.4)
DemoNSF(T5)
95.81(10.77)
03.77
91.66()10.6)
19.78(110.2)
1..3
87.96(110.9)
86.16(
Table 5: Fl scores with standard deviations under different input perturbations on RADDLE. All the models are in large versions
More Detailed Experiments
perturbed data, the NoisyDemos used in this pa- per focuses on introducing the distribution infor- mation of the perturbed data, so that the generative model can learn the perturbed sentence and slot en- tity distribution information the maximum extent and make it more robust.
C.1 Ablation Study
We conduct an ablation study investigate the characteristics the main components in DemoNSF. As shown in Table 4, we have the following observations: 1) The performance of the model improves when adding any component, which demonstrates that every part of our design is necessary. 2) For the three different granularities of perturbations, observe significant improve- ments in auxiliary tasks specifically designed for each. Specifically, the NR task learns the map- ping relationship between character-level pertur- bations and clean data, resulting in 4.79% im- provement in Typos. While the RM task implic- itly captures the semantic information of slot enti- ties during the mask-filling procedure. It achieved about 4.71 1.% improvement under word-level per- turbations (Speech). As for the HD task, it is able capture the unique distribution information of perturbed data and significantly improves the per- formance of the model under coarse-graie grariee per- turbations while maintaining generalization, espe- cially in Simplification (3.47%). 3) Adopting joint pre-training tasks (+TT) results in noticeable improvement compared with adding of them, which indicates that jointly pre-training objectives have a mutually reinforcing effect (obtain 3.47% improvement on average of perturbed data). 4) We explore the ablation study of three demonstration retrieval strategies. BleanDemon Mix Demos, and N NoisyDemos represent retrieve demonstra- tions from Detann Dmizz and Daug, respectively. As for Demos, We make sure to include both clean and noisy demonstrations. We find that con- catenating demonstrations does yield exciting re- sults on perturbed test data. Specifically, while M Demos is able to absorb more diverse data distributions and performs well on both clean and
C.2 Results on Large-version Model
We compare the performance of DemoNSF with other baselines the large-version model (i.e. T5- large and BART-large). Despite using model with larger parameter size, generative models still ex- perience a significant decline performance when faced with perturbed inputs, especially with fine- grained perturbations. As shown in Table 5, we can find that the model's performance declined by 37 17.42% for BART and 27.33% for T5 on Typos and 19.56% and 14.27% for BART and T5, respectively Speech. Our approach also achieves impres- sive improvements both fine-grained and coarse- grained perturbations. To be specific, DemoNSF introduces 5.18% Fl improvements on the average performance all the perturbations input com- pared with InstructionNER based on T5 and 1.63% improvements compared with BARTNER based on BART.
C.3 Details of Mixed Perturbations
For the mixed perturbations experiment on SNIPS, we also investigate the performance of DemoNSF on single perturbation (AppendIrr, Sub, and EntTy- pos). As shown in Table 6, we obtain similar con- clusions. Specifically,
approach introduces sig- nificant improvements in fine-grained perturbations (e.g. 5.86% on EntTypos). While our approach also maintains exciting performance on coarse-grained perturbations (c.g. 2.51% improvements on Ap- pendIr)..

---

Sent
Word
Char
Char- Word
Char4Sent Apt.+App-
Werd+Sent
Shar-Word-Sent
Methods
Clean
App-
Sub.
Ent.
Ent.4Sub.
App-4Suh.
App.
BART
79.43
63.95
71.20
57.84
$8.00
47.27
50.28
38.36
94.12
83.97
85.13
65.90
57.44
56.79
73.47
47.93
BARTNER
86.34
69.33
77.22
62.28
54.83
49.10
58.92
42.25
LightNER
81.39
60.63
70.18
52.59
42.34
35.82
45.44
27.00
InStructionNER
94.69
84.32
84,78
66.93
57.87
58.89
74.45
50.75
DemoNSF (BART)
87.16(+0.7)
76.05(+ 1.2)
79.56(
67.31(+).2)
61.27(+0.55
51. 21,26((0.9)
(8.7)(
44,27(+0.9)
DemoNSF (TS)
94.75(+1..7)
86.5
86.8(-0.2)
72.79(++0.4)
63.59(+0.3)
63.94(+1.2)
(7.6%(-0.7)
55.12(+00.3
Table 6: Fl scores with standard deviations under kinds of single perturbations and 4 kinds of mixed perturbations on SNIPS.
D Related Work
dering their application in realistic task-oriented dialogue systems.
D.1 Slot Filling
Sequence Labeling Paradigm. Initially, the slot filling task was commonly defined as sequence labeling problem. Previous methods can be cate- gorized into two types: one-stage and two-stage approaches. Specifically, one-stage approaches (Bapna et al.. 2017: Shah et al., 2019; Lee and Jha, 2019) conduct slot filling individually for each slot type. It first generates word-level representa- tions and the predictions nated features for each slot type. However, these methods only learn the surface mapping between entities and suffer from multi-prediction problems. To address these limitations, a branch of two-stage methods (Liu et al., 2020b; He et al., 2020b; Wang et al., 2021; Ma et al., 2022; Dong et al., 2023a; Wang et al., 2022a; Dong et al., 2023b) are pro- posed. Firstly, coarse-grained binary sequence labeling model is used to identify all slot entities in the utterances. Subsequently, the entity value is mapped the representation of the corresponding slot label in the semantic space in order to classify slot types effectively.
D.2 Input Perturbation Problem
Recently, there has been a growing interest in enhancing the resilience of NLP systems to in- put perturbations. Moradi and Samwald (2021b) present empirical evaluations of the robustness of various NLP systems against input perturbations on synthetically generated benchmarks. Namysl et al. (2020, 2021) focus on the robustness the NER model against Optical Character Recognition (OCR) noise and misspellings. Compared to other NLP systems, dialogue systems would face more diverse input noise due more frequent interac- tions with users. Fang al. (2020); Gopalakr- ishnan al. (2020) investigate the robustness of dialogue systems on ASR noise, and Ruan et al. (2020); et al. (2020b); Huang and Chen (2020); Li et al. (2020a) mainly focus on the ASR-noise- robustness SLU models in dialogue systems.
based the concate-
Most previous studies (Moradi and Samwald, 2021a; Wu et al., 2021; Liu et al., 2020a) that in- vestigated this robustness problem predominantly focused on rule-based synthetic datasets, which do possess certain limitations. Meanwhile, real-world dialogue systems encounter wider range of per- turbations due to frequent interactions with users. To further explore this direction, RADDLE (Peng et al., 2020) offers a crowd-sourced robustness evaluation benchmark for dialog systems, which includes various noisy utterances that existed in real dialogue scenarios. Liu et al. (2020a) intro- duced Language Understanding Augmentation, a methodology that incorporates four data augmen- tation techniques to simulate natural perturbations. To cope with more complex noisy scenarios, Dong et al. (2022a); Liu al. (2023) investigate input per- turbation problems on discriminative neural mod- cls. In this paper, we mainly focus on the perfor- mance of generative frameworks on input perturba-
Generative F Framework. Recently, some works (Wang et al., 2022b) have started to reformulating NER and slot filling tasks to loquence-to-sqquee (seq2seq) tasks and integrate generative methods. BARTNER (He al., 2020c) proposes pointer- based seq2seq architecture, which converts the NER task to unified sequence generation task and predicts entities from the input sentences and the corresponding type indexes LightNER(Chen et al., 2021) introduces prompt-tuning to the attention mechanism of BARTNER and achieves promising improvement in low-resource scenarios. Moreover, some prompt-based Generative methods (Lu et al., 2022; Xie al., 2022b) have achieved strong per- formance in information extraction. Nevertheless, their exploration of generative frameworks on di- verse input perturbations remains a blank area, hin-

---

tion problems.
D.3 Demonstration-Dast based learning
Demonstrations are first introduced by the GPT series (Radford et al., 2019; Brown et al., 2020), where few examples are sampled from train- ing data and transformed with templates into appropriately-filed prompts. Based on the task reformulation and whether the parameters are up- dated, the existing demonstration--aase learning research can be broadly divided into three cate- gories: In-context Learning (Brown al., 2020; Zhao al., 2021; Min et al., 2021; Wei et al., 2022), Prompt-based Fine-tuning (Liang et al., 2022; Dong et al., 2023c), Classifier-based Fine- tuning (Lee et al.. 2021). However, these ap- proaches mainly adopt demonstration- learn- ing in the fine-tuning that cannot make full use of the effect of demonstration based learning.


Reviews:
[{"review_id": "Xju5dQWWSq", "reviewer": "Reviewer_8JBt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable appraisal and acknowledgment of our work. We are committed to enhancing and enriching our study with additional experiments and analyses, in accordance with the recommendations from all the reviewers, to further substantiate the robustness of our findings.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces an effective multi-task demonstration-based generative framework for the noisy slot-filling task in practical dialogue scenarios.\n2. The paper is well-written and easy to follow. The authors provide clear explanations of the proposed framework and the experiments conducted.", "reasons_to_reject": "No particular reason to reject.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents DemoNSF, a multi-task demonstration-based generative framework for the noisy slot-filling task in practical dialogue scenarios. It proposes three novel noisy auxiliary tasks in the pre-training stage to capture semantic structural information of input perturbations at different granularities. Moreover, it conducts extensive experiments on two benchmarks and demonstrates that DemoNSF outperforms several baseline methods by a large margin. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JNrwRVJEd0", "reviewer": "Reviewer_vQ9E", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you greatly for your valuable suggestions and your insights are truly appreciated. To validate the scalability and adaptability of our proposed method, we have conducted additional experiments using GPT-2 small. The outcomes of these experiments are presented in the subsequent table:    \n\n| Methods | Clean | Verbose | Paraphrase | Simplification | Typos | Speech | Perturbed-Avg. |\n| :---- | :----: | :----: | :----: | :----: | :----: | :----: | :----: |\n| BART | 95.28 | 77.87 | 82.72 | 82.95 | 53.90 | 73.57 | 74.20 |\n| T5 | 95.49 | 81.34 | 89.13 | 83.73 | 62.43 | 81.13 | 79.55 | \n| GPT-2 | 95.37 | 80.52 | 85.66 | 82.98 | 60.19 | 77.78 | 77.43 |\n| DemoNSF(BART) | 95.71 | 78.83 | 88.29 | 86.01 | 65.60 | 82.48 | 80.24 |\n| DemoNSF(T5) | 95.72 |  82.37 | 89.98 | 89.49 | 76.63 | 87.55 | 85.20 |  \n| DemoNSF(GPT-2) | 95.66 | 81.95 | 87.63 | 87.02 | 69.75 | 86.31 | 82.53 |\n\nThe experimental results demonstrate that our approach achieves state-of-the-art results on GPT-style models as well. This also indicates the plug-and-play, scalable nature of our method within the generative framework.  \nOnce again, we express our gratitude for your valuable suggestions. We will continue to refine our article's content and strive to enhance the quality and comprehensiveness of our research and experiments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The three main contributions of this paper can be summarized as follows:\n\t• Investigating the Effects of Input Perturbations: This paper is the first to comprehensively investigate the effects of diverse input perturbations on generative frameworks in slot filling tasks. It highlights the vulnerability of existing prompt-based generative methods when faced with different human expressions.\t\n\t• Multi-task Demonstration-based Generative Framework: The authors propose a simple yet unified multi-task demonstration-based generative framework called DemoNSF. It includes three novel noisy auxiliary tasks and a noisy demonstration construction strategy, which enhance the model's robustness and adaptability to perturbed inputs in real-world dialogue scenarios.\n\t• Improved Performance and Generalization: DemoNSF outperforms all baseline methods and demonstrates strong generalization. The extensive analysis provided in the paper also offers empirical guidance for the practical application of generative frameworks.", "reasons_to_reject": "The experiment is not enough. The authors build T5 and BART as baseline models, however, The GPT-style models are not considered in this paper. Maybe it is interesting to include some GPT-style models.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a framework called DemoNSF to address the issue of input perturbations in slot filling tasks in dialogue systems. \n\nThe existing models are trained on perturbation-free datasets, which leads to decreased performance when faced with different human expressions and input perturbations. \n\nTo tackle this limitation, the authors propose the DemoNSF framework, which includes three noisy auxiliary tasks: noisy recovery, random mask, and hybrid discrimination. The framework formulates the slot filling task as a sequence-to-sequence generation guided by noisy task demonstrations. It selects semantically similar examples for each query, converts them into natural demonstration sentences, and encodes them along with the input text to learn the semantic structure of perturbations. \n\nExperimental results demonstrate that DemoNSF outperforms baseline methods, achieves strong generalization, and effectively addresses fine-grained and coarse-grained perturbations, as well as mixed perturbation scenarios.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MVtZKsq2eO", "reviewer": "Reviewer_zUxk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your valuable suggestions and questions.    \n**Regarding your first question**:  \n \"The novelty seems a bit limited. If you introduce noises in pretaining and fine-tuning, it is expected to perform well on perturbed examples.\"    \n**Our response**:    \nWe greatly appreciate the question you've raised. In order to examine whether the augmented data in pre-training and fine-tuning stages significantly enhances the model's robustness, we conduct an experiment where augmented data was mixed with the training data for joint training. The experimental results are presented in the table below:    \n\n| Methods | Clean | Verbose | Paraphrase | Simplification | Typos | Speech | Perturbed-Avg. |\n| :---- | :----: | :----: | :----: | :----: | :----: | :----: | :----: |\n| T5 (baseline) | 95.49 | 81.34 | 89.13 | 83.73 | 62.43 | 81.13 | 79.55 | \n| DemoNSF (T5) | 95.72 |  82.37 | 89.98 | 89.49 | 76.63 | 87.55 | 85.20 |  \n| Data Aug. | 95.11 | 81.49 | 89.62 | 85.38 | 65.13 | 82.88 | 80.90 (-4.3) |  \n\nThe results indicate that solely performing data augmentation does not lead to a substantial improvement in the model's robustness against perturbed data, instead, it compromises the performance on clean data. We believe that rule-based data augmentation (such as random character augmentation and random deletion) deviates significantly from real-world test data, causing the model trained on mixed data to learn only superficial predictive patterns from the augmented data, without truly grasping the essence of perturbations. Consequently, when faced with real-world data, the presence of a gap prevents significant performance improvement.    \nIn contrast, our proposed approach is capable of learning the semantic structural information and inherent forms of perturbed data at three different levels of granularity (character, word, sentence). This enables the model to better capture intrinsic details such as perturbation granularity and patterns when dealing with real-world perturbed data, resulting in enhanced predictive outcomes.    \n\n**Regarding your second question**:  \n\"I personally think the ablation study/ChatGPT experiments/Ablation study are quite important, not sure why they appear in the appendix.\"    \n**Our response**:    \nDue to the limitations in the length of the paper, we have placed the primary analytical experiments in the main body of the text, while the extensible experiments (such as the ChatGPT experiment and ablation study) have been included in the appendix. We sincerely appreciate your valuable suggestion, and we will optimize the structure of the paper based on your recommendation.    \nOnce again, we express our gratitude for your two suggestions. We are committed to further enhancing the structure and content of this paper, incorporating additional experiments, and striving to elevate the quality of our research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and easy to read.\n2. The empirical study demonstrates the improvement of the proposed methods, especially on character-level and word-level perturbations.", "reasons_to_reject": "1. The novelty seems a bit limited. If you introduce noises in pretaining and fine-tuning, it is expected to perform well on perturbed examples.\n2. I personally think the ablation study/ChatGPT experiments/Ablation study are quite important, not sure why they appear in the appendix.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work proposes multi-task training to augment slot-filling training to capture input perturbations' features. It also constructs noisy demonstrations for the generative framework. The proposed approach performs better on diverse kinds of perturbations and various models (BART, T5, ChatGPT.)", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary Evaluation:

The paper introduces DemoNSF, a multi-task demonstration-based generative framework for the noisy slot-filling task in practical dialogue scenarios. It proposes three novel noisy auxiliary tasks in the pre-training stage to capture semantic structural information of input perturbations at different granularities. Experimental results demonstrate that DemoNSF outperforms baseline methods and achieves strong generalization.

Pros:
- Addresses an important issue of input perturbations in slot filling tasks in dialogue systems.
- Proposes a novel multi-task demonstration-based generative framework called DemoNSF that improves performance and generalization.
- Clear and well-written paper with thorough experiments conducted.
- First comprehensive investigation of the effects of diverse input perturbations on generative frameworks in slot filling tasks.

Cons:
- The novelty seems to be limited, and the improvements could be expected.
- While it uses T5 and BART as baseline models, GPT-style models are not considered.
- Some important experimental results such as ablation studies and ChatGPT experiments are placed in the appendix instead of the main paper.

After considering the updated scores of the reviewers for soundness and excitement, this paper presents a valuable contribution to the field of NLP and dialogue systems. The proposed framework, DemoNSF, shows improved performance and generalization over existing models in addressing input perturbations in slot filling tasks. However, the authors could consider incorporating GPT-style models in their comparison and moving some experimental results from the appendix to the main paper to strengthen the work further.
Decision: Accept-Findings