PromptAttack: Probing Dialogue State Tracker with Adversarial Prompts
Year: 2023

Abstract:
A key component of modern conversational systems is the Dialogue State Tracker (or DST), which models a user’s goals and needs. Toward building more robust and reliable DSTs, we introduce a prompt-based learning approach to automatically generate effective adversarial examples to probe DST models. Two key characteristics of this approach are: (i) it only needs the output of the DST with no need for model parameters, and (ii) it can learn to generate natural language utterances that can target any DST. Through experiments over state-ofthe-art DSTs, the proposed framework leads to the greatest reduction in accuracy and the best attack success rate while maintaining good fluency and a low perturbation ratio. We also show how much the generated adversarial examples can bolster a DST through adversarial training. These results indicate the strength of prompt-based attacks on DSTs and leave open avenues for continued refinement.

heading: 1 Introduction
text: Task-oriented dialogue systems aim to help users with tasks through a natural language conversation. Example tasks include booking a hotel or completing a do-it-yourself project. A key component for enabling a high-quality task-oriented dialogue system is the Dialogue State Tracker (or DST) which plays an important role in understanding users’ goals and needs (Wu et al., 2019; Hosseini-Asl et al., 2020; Li et al., 2021b; Dai et al., 2021; Feng et al., 2021; Zhao et al., 2021; Balaraman et al., 2021). For example in Figure 1a, given the user utterance “I am looking for a cheap restaurant in the center of the city”, the DST extracts the user’s preference for booking a restaurant, which is typically represented as slot-value pairs such as (restaurant-price range, cheap) and (restaurant-area, center). The current state
∗ Equal Contribution
Dialogues
R: Is there anything else I can help you with today? U: I am looking for a cheap restaurant in the center of the city.
Belief States (Domain-Slot, Value)
DST restaurant-price range, cheap restaurant-area, center
(a) Dialogue state tracking task. R represents system response, and U represents user utterance.
User Utterances
U: I am looking for a cheap restaurant. U1: Can you give me the address of a cheap restaurant please? U2: Oh, I am looking for an uh, expensive, cheap restaurant.
Belief States (Domain-Slot, Value)
DST U: restaurant-price range, cheap ✓ U1: restaurant-price range, none ✗ U2: restaurant-price range, expensive ✗
(b) Adversarial examples. U1 and U2 are adversarial examples based on U which maintain ground-truth values but lead DST models to wrong predictions.
Figure 1: Dialogue examples and adversarial examples.
of the conversation is a primary driver of the subsequent dialogue components (e.g., what is the next action to take? what is the appropriate response to generate?).
For a conversational system designer, it is critical that a deployed DST be robust and reliable, even in the presence of a wide variety of user utterances. Many of these systems are trained over previous user utterances and so may have only limited coverage of the space of these utterances. Further, beyond these benign users, there is also a long history of spammers, trolls, and malicious users who aim to intentionally undermine deployed systems.
Indeed, recent work has demonstrated that careful construction of adversarial examples can cause failures in the DST (Li et al., 2021b; Liu et al., 2021a), leading to incorrect slot-value pairs and degraded user experience. These approaches, however, are mainly hand-crafted or based on heuristics. As a result, there is a research gap in learningbased methods for probing DSTs centered around three key questions: (i) How can we systematically learn effective adversarial examples? (ii) What impact do such discovered examples have on the quality of state-of-the-art DSTs? and (iii) Can we
10651
build more robust DSTs even in the presence of such adversarial examples? Further compounding these questions are the inherent challenges of adversarial examples in the context of a DST: that is, the examples should preserve the semantics of a non-adversarial input while leading to an incorrect prediction even in the presence of the correct slot-value in the adversarial input as illustrated in Figure 1b. For example, an adversarial example based on the user utterance “I am looking for a cheap restaurant” that maps to the slot-value pair (restaurant-price range, cheap) should preserve the user intent for “cheap” while leading to the incorrect prediction (restaurant-price range, expensive).
Hence, in this paper, we propose a novel promptbased learning approach called PromptAttack to automatically generate effective adversarial examples to probe DST models. Our approach builds on recent advances in prompt learning, which has demonstrated a strong ability in probing knowledge in pre-trained language models for many NLP tasks (Gao et al., 2021; Li and Liang, 2021; Liu et al., 2021b; Zhu et al., 2022). Concretely, we first show how to find effective adversarial prompts in both a discrete and a continuous setting. In both cases, our approach needs only the output of the DST (e.g., (restaurant-price range, cheap)) with no need for model parameters or other model details. Second, we use the adversarial prompts to generate adversarial examples via a mask-andfilling protocol, resulting in natural language utterances that can be targeted at any DST. As a result, such a prompt-based attack can be widely applied.
Through experiments over four state-of-the-art DSTs and versus competitive baselines, we find that the prompt-based framework leads to the greatest reduction in accuracy for all DSTs, ranging from a 9.3 to 31.0 loss of accuracy of the DST making a correct slot-value prediction. Further, we observe that PromptAttack results in the best attack success rate (that is, how many of the adversarial examples lead to incorrect predictions). Moreover, the generated adversarial examples maintain good fluency and low perturbation ratio, evidence that they are close to legitimate non-adversarial user inputs. We also show how such a prompt-based attack can be used to bolster a DST by augmenting the original training data with adversarial examples, leading to a significant increase in accuracy (from 61.3 to 67.3). These and other results indi-
cate the strength of prompt-based attacks on DSTs and leave open avenues for continued refinement.1

heading: 2 Related Work
text: Adversarial examples have been widely explored to investigate the robustness of models (Goodfellow et al., 2015). Recent work in the NLP domain has targeted tasks like text classification and inference (Pruthi et al., 2019; Ren et al., 2019; Morris et al., 2020; Jin et al., 2020; Li et al., 2020; Yang et al., 2022; Lei et al., 2022), reading comprehension (Jia and Liang, 2017; Bartolo et al., 2021), named entity recognition (Simoncini and Spanakis, 2021), and machine translation (Belinkov and Bisk, 2018). These works typically aim to construct examples that are imperceptible to human judges while misleading the underlying model to make an incorrect prediction, while also maintaining good fluency and semantic consistency with original inputs (Li et al., 2020). Only a few works have begun to explore adversarial examples in DSTs like CoCo (Li et al., 2021b), which aims to test the robustness of models by creating novel and realistic conversation scenarios. They show that DST models are susceptible to both unseen slot values generated from in and out of the slot domain. Liu et al. (2021a) propose a model-agnostic toolkit to test the robustness of task-oriented dialogue systems in terms of three aspects: speech characteristics, language variety, and noise perturbation. The adversarial examples are based on heuristics and it is unclear how to adapt such an approach to new victim models effectively without more hand-crafted templates. In contrast, we explore in this paper the potential of a learning-based approach to generate effective adversarial examples.
Prompt learning is a recently proposed paradigm for using prompts to better probe and adapt large pre-trained language models (PLMs) to a variety of NLP tasks, e.g., text classification and inference (Gao et al., 2021; Yang et al., 2022; Wang et al., 2022), factual probing (Zhong et al., 2021), summarization (Li and Liang, 2021), and dialogue systems (Madotto et al., 2021; Lee et al., 2021; Zhu et al., 2022; Yang et al., 2023). With the increase in the size of PLMs, prompt learning has been shown to be parameter-efficient (Liu et al., 2021b; He et al., 2022; Lu et al., 2023). There are two types of prompts: discrete (or hard) prompts and continu-
1Our code is publicly available at https://github.com/ dongxiangjue/PromptAttack.
Change Fill in
belief states: [s] = [v]
belief states: restaurant-price range = expensive
Slot-Value Dict.
restaurant-price range, cheap
restaurant-price range, expensive
Template
Discrete PromptBelief States
(a) Discrete prompt construction. The discrete prompt is constructed by filling pre-designed templates with slots extracted from the DST model and corresponding random values.
DST Model loss Prompt Dialogue
(b) Continuous prompt tuning. The continuous prompt is prepended before the dialogue context embeddings and tuned by optimizing the loss while keeping DST model parameters fixed.
Prompt Dialogue
Black-Box DST Model
Adversarial Utterances
MLM Model
mask
(c) Adversarial example generation. The adversarial prompt (discrete or continuous) is prepended before the masked dialogue context (or embeddings) to generate perturbations via mask-and-filling. After removing the adversarial prompt, the generated adversarial example is used to attack victim models.
Figure 2: Overview of PromptAttack.
ous (or soft) prompts. Discrete prompts are humandesigned text strings (Brown et al., 2020) while continuous prompts are continuous embeddings. Soft prompts proposed by Lester et al. (2021) prepend a sequence of continuous vectors to the input, freeze the language model parameters, and then backpropagate the error during tuning. In this paper, we explore both approaches in the design of our prompt-based attack framework.
Recent works have begun to explore how prompts can be helpful in exposing fundamental flaws in large language models. Yang et al. (2022) shows how to manually design prompts to flip the output of a model for classification tasks. However, it is time-consuming to design and find prompts that are most effective to generate adversarial examples capable of successfully attacking victim models. It is an open question how to leverage prompts for uncovering effective adversarial prompts.

heading: 3 PromptAttack
text: Our prompt-based learning approach proceeds in two stages. First, our goal is to identify adversarial prompts that can effectively probe a DST to reveal gaps in its robustness. In the second, we use these prompts to create adversarial examples that can attack DSTs successfully while maintaining good fluency. Figure 2 shows an overview of the proposed approach. In the following, we first formalize DSTs and the problem of probing a DST. Then, we introduce the details of PromptAttack.

heading: 3.1 Task Formulation
text: DST Task. Let CT = {(r1, u1), . . . , (rT , uT )} represent a T -turn dialogue, where ri and ui(1 ≤ i ≤ T ) are the system response and user ut-
terance at the i-th turn, respectively. Each turn (ri, ui) contains several slots (e.g., arrive by, leave at) in a specific domain (e.g., taxi), where we denote the N domain-slot pairs as S = {s1, . . . , sN}. At turn t, we denote current user utterance ut and previous dialogue context Ct = {(r1, u1), . . . , (rt−1, ut−1), rt}. A DST model aims to extract the dialogue belief state Bt = {(s1, v1), . . . , (sN , vN )} for ut, where vj is the associated value for each slot sj(1 ≤ j ≤ N). For example, given a dialogue (“. . . . I am looking for expensive Mediterranean food.”), the DST model aims to extract expensive for the slot restaurant-price range and Mediterranean for the slot restaurant-food.
Attacking a DST. Given dialogue history Ct, current user utterance ut, and dialogue belief states Bt, the purpose of an adversarial attack on a DST is to intentionally perturb the original user utterance ut to get an adversarial example u ′ t with the two following characteristics: (i) it should mislead the DST model f to incorrectly predict B ′ t, and (ii) it should be fluent in grammar and consistent with the semantics of the original utterance ut by keeping the slot-value-related information in ut unchanged. If the adversary can achieve f(u ′ t) = B ′ t, we say the adversarial example u ′ t attacks f successfully.

heading: 3.2 Finding Adversarial Prompts
text: We begin by focusing on the first stage of PromptAttack: how to find the most effective adversarial prompts. We explore both discrete prompts (as illustrated in Figure 2a) and continuous prompts (as illustrated in Figure 2b). A discrete prompt approach is a human-designed natural language prompt that is easy to interpret. We pair this with
a treatment of continuous prompts that have more representation capacity.
Discrete Prompt Construction. To begin with, how can we design discrete prompts? For the DST task, it is time-consuming to manually design sentences containing values that are opposite to the ground truth values for each slot as adversarial prompts. Thus, we apply an intuitive template derived from belief states as an adversarial prompt template: “belief states: [s] = [v];”. First, we use the DST model to extract value vi for each slot si in ut. If vi is not empty, the corresponding slot name si is filled in [s]. Then we pick a random value v
′ i from a predefined in-domain Slot-Value
Dictionary (Li et al., 2021b) where v ′ i and vi are under the same slot si. The new random value v ′ i is used to fill the [v] in the template. Thus, the adversarial prompt becomes “belief states: si = v
′ i;”. As in Figure 2a, given ut (“I am looking for cheap food.”), the predicted Bt is {(restaurantprice range, cheap)}, then the adversarial prompt is “belief states: restaurant-price range = expensive”, where “expensive” is a random value that is different from the predicted value “cheap”.
Such a template does not have access to true slot-value pairs of the test set and only utilizes the predictions from the victim models. Since the discrete prompts are human-designed, they are more human-readable and easier to interpret. However, to obtain a prompt for each input, victim models must be queried multiple times, which may be unrealistic in some scenarios. Hence, we take the next step to search for better prompts in the embedding space of the model. Specifically, we directly optimize the continuous input embedding space through continuous prompt tuning to find the adversarial prompt vectors that are most effective.
Continuous Prompt Tuning. Continuous prompts are input-agnostic sequences of embeddings with tunable parameters that are optimized directly in the continuous embedding space of the model, as shown in Figure 2b. In our task, the length of continuous prompt patt is m, denoted as patt = p1 . . . pm where each pi ∈ Rd(1 ≤ i ≤ m) is a dense vector with the same dimension d as the DST’s input embedding (e.g., 768 for TripPy). Given the initialization of patt, we concatenate it with the representation of user utterance eu and update it by keeping all other model parameters fixed and optimize the loss of the training set. To find the adversarial prompts patt that could lead DST
models f to wrong predictions B ′ t effectively, we maximize the loss for the ground truth belief states Bt for all user utterance in the training set with the following objective:
argmax patt
Eu∼U [L (Bt, f (patt; eu))] ,
where U are user utterances and L is the loss function of the DST task. By maximizing the loss for the ground truth belief states we aim to find prompts that force the model to make the most wrong predictions by pushing far apart from the ground truth, like guessing “expensive” instead of “cheap” for ut (“I am looking for cheap food.”).
In addition, we explore an alternative tuning objective – minimizing the loss. We replace all the non-empty values in Bt to empty (e.g., (restaurant-price range, expensive) changes to (restaurant-price range, none)) and then minimize the loss:
argmin patt
Eu∼U [ L ( B ′ t, f (patt; eu) )] ,
where B ′ t is the set of target belief states. Different from our previous tuning objective, here we aim to find prompts that force the model to fail to extract the correct value for the slot from user utterances. For example, the DST will fail to extract “cheap” for slot price range in ut (“I am looking for cheap food.”) and thus the predicted belief states will become (restaurant-price range, none).

heading: 3.3 Adversarial Example Construction
text: Next, we focus on the second stage of PromptAttack: how can we use these prompts to create adversarial examples that can attack DSTs successfully while maintaining good fluency? After obtaining the adversarial prompts, we use them to generate adversarial examples via mask-and-filling (Li et al., 2021a; Yang et al., 2022; Lei et al., 2022) by pre-trained masked language models. Specifically, we tokenize user utterance ut to a list of tokens, ut = [w 1 u, w 2 u, . . . , w n u ]. Then we randomly mask tokens that are not values in Bt, slot-related words, or stopwords with a special token [MASK] and denote the masked ut as umt = [w 1 u,[MASK], . . . , w n u ]. Shown in Figure 2c, we concatenate the adversarial prompts and the masked utterance umt and use a masked language model M to predict masked text pieces and generate the perturbations based on surrounded context. As shown in Table 1, for discrete prompt pdatt, the input for M would be
the concatenation of pdatt and umt while for continuous prompt pcatt, the input would be the concatenation of pcatt and embedding of masked user utterance e1u [MASK] enu. Hence, with patt and the capability of MLM, the model M will fill in the blanks with context-consistent tokens which can keep the sentence fluency while maximizing the risk of the DST making wrong predictions, denoted as P ([MASK] = w|patt;umt ), where w is the generated perturbation. After filling [MASK] with w and removing patt, the filled user utterances are used as adversarial examples to attack victim models.

heading: 4 Experimental Setup
text: Our experiments are designed to test the effectiveness of the proposed prompt-based approach to attack DST models. We structure the experiments around four research questions: RQ1: Are adversarial examples learned by PromptAttack effective and transferable? And how do these examples compare against baseline (non-prompt) approaches? RQ2: Are the generated adversarial examples of good quality? That is, are they fluent with a low perturbation ratio? RQ3: What impact do the design choices of PromptAttack have, i.e., the ratio of perturbed tokens and prompt length? RQ4: And finally, can the generated adversarial examples be used to improve the performance of current DST models to improve their robustness?

heading: 4.1 Dataset
text: We evaluate our methods on the widely used and challenging multi-domain dialogue dataset, MultiWOZ 2.1 (Eric et al., 2020),2 which contains over 10,000 dialogues spanning seven domains. Following existing work (Li et al., 2021b; Lee et al., 2021; Yang et al., 2022), we keep five domains (train, taxi, restaurant, hotel, and attraction) with 30 domain-slot pairs and follow the standard train/validation/test split.

heading: 4.2 Evaluation Metrics
text: We evaluate the proposed methods with a standard set of metrics (Jin et al., 2020; Li et al., 2020,
2github.com/budzianowski/multiwoz, MIT License.
2021a; Simoncini and Spanakis, 2021): Joint goal accuracy (JGA): the average accuracy of predicting all (domain-slot, value) pairs in a turn correctly. Attack success rate (ASR): the proportion of generated adversarial examples that successfully mislead model predictions. Perturbation ratio (PER): the percentage of perturbed tokens in the sentence. Each replace action accounts for one token perturbed. A lower perturbation ratio indicates more semantic consistency (Li et al., 2020). Perplexity (PPL): a metric to evaluate the fluency of sentences. We calculate the perplexity of adversarial examples through GPT-2 (Radford et al., 2019). PPL is calculated across all the adversarial examples. A lower PPL score indicates higher fluency and naturalness of the adversarial examples.

heading: 4.3 Baseline Methods
text: We compare our methods with strong baselines capable of attacking a DST. TP and SD are two methods maintaining the dialogue act labels unchanged and implemented by the LAUG toolkit (Liu et al., 2021a). For a fair comparison, we do not apply slot value replacement which would modify the slot values in the original utterances. TP (Text Paraphrasing) uses SC-GPT (Peng et al., 2020) to generate a new utterance conditioned on the original dialogue acts as data augmentation. SD (Speech Disfluency) mimics the disfluency in spoken language by filling pauses (“um”), repeating the previous word, restarting by prepending a prefix “I just” before the original user utterance, and repairing by inserting “sorry, I mean” between a random slot value and the original slot value (Liu et al., 2021a). SC-EDA (Liu et al., 2021a) injects word-level perturbations by synonym replacement, random insertions, swaps, and deletions without changing the true belief states. BERT-M is introduced in this paper as another baseline method. First, we randomly mask tokens that are not slot-value related and not stopwords. Then, we use BERT (Devlin et al., 2019) to generate perturbations based on the top-K predictions via mask-and-filling, where in our experiments K = 20. We sorted the top 20 tokens based on the possibility scores and pick the one with the lowest possibility to fill the masked position. The filled user utterance is regarded as an adversarial example.

heading: 4.4 Victim Models
text: We choose the TripPy DST (Heck et al., 2020) as our base model to train our adversarial prompts
since classification-based models have better performance and are more robust than generationbased models (Liu et al., 2021a). Demonstrating the susceptibility of TripPy to our adversarial examples can reveal the limitations of current DSTs, but we further explore the transferability of the prompt-based attacks.
Transferability reflects the generalization of the attack methods, meaning that adversarial examples generated for one model can also effectively attack other models (Zhang et al., 2020). Hence, we also evaluate the prompt-based approach learned over TripPy by targeting our adversarial examples on other popular DSTs: TRADE (Wu et al., 2019), SimpleTOD (Hosseini-Asl et al., 2020), and CoCo (Li et al., 2021b), one of the state-ofthe-art models.3 Additional information about the implementations can be found in Appendix A.

heading: 5 Experimental Results
text: Given this setup, we now investigate the four experimental research questions in turn.

heading: 5.1 Attack Effectiveness (RQ1)
text: First, are the adversarial examples learned by PromptAttack effective? Table 2 summarizes the results for three versions of PromptAttack versus the baselines for the four different DSTs (TripPy, CoCo, SimpleTOD, and TRADE). We consider the discrete version of PromptAttack (denoted as PromptAttackd) and two continuous versions: one is optimized by maximizing the training loss (denoted as PromptAttackcx), while the other one is optimized by minimizing the loss (denoted as PromptAttackcn).
3These models are fine-tuned on MultiWOZ 2.1 using code from CoCo (https://github.com/salesforce/coco-dst) and follow the same post-processing strategy as CoCo. BSD 3-Clause License.
Attack Performance. First, let’s focus on the TripPy column. All versions of PromptAttack are learned over TripPy and then applied here so we can assess the susceptibility of a popular DST to adversarial examples. The four baselines lead to some degradation in terms of accuracy (JGA), with SD performing the best with a JGA of 56.5 (a 4.8 drop from the original DST).4 The three promptbased learning approaches result in strong degradation in terms of accuracy, ranging from 7.7 to 9.3 drops relative to the original. We observe that our PromptAttack models significantly outperform SC-EDA, TP, and BERT-M, the methods without introducing new slot values in the adversarial examples, in terms of JGA and ASR. Compared with the best baseline method among these three, BERT-M, PromptAttackcn decreases the JGA by 6.9 and increases ASR by 13.2, respectively. In addition, for the method introducing new slot values, SD, PromptAttackcn outperforms it by 4.5 and 8.9. Hence, these observations reveal the attack effectiveness of our proposed PromptAttack methods over these baselines no matter whether the methods introduce new slot values or not.
Transferability. To test the transferability of the generated adversarial examples, we take the examples trained over TripPy and then use them to attack other victim models CoCo, SimpleTOD, and TRADE. For CoCo and SimpleTOD, we see that PromptAttack outperforms these four baselines. Our best method PromptAttackc achieves
4We attribute this good attack performance since although this method maintains ground truth slot-value labels unchanged, it prepends new slot values before the original slot values in the user utterance. This operation is effective because it can easily confuse the model to decide which slot values are the truth slot values. In contrast, our prompt-based approaches are designed to make very few changes and to avoid introducing new slot values.
52.8 and 25.0 JGA when attacking CoCo and SimpleTOD, showing better transferability than PromptAttackd. For TRADE, PromptAttackc shows better attack performance than baselines without introducing new slot values significantly. Specifically, PromptAttackcx shows a decrease of 10.2 and an increase of 20.0 in terms of JGA and ASR, respectively. In general, our PromptAttack methods show good transferability: the adversarial examples generated for one victim model can also be used to attack another model effectively.

heading: 5.2 Adversarial Example Quality (RQ2)
text: Next, we examine whether the generated adversarial examples are of good quality. First, are they fluent with a low perturbation ratio? We automatically measure the perturbation ratio (PER) between the original input and adversarial examples, and the fluency by computing the perplexity (PPL). The lower perturbation ratio represents fewer perturbed tokens in original utterances and lower perplexity indicates better fluency. From Table 3 we observe that the PromptAttack methods achieve low perplexity and show good fluency with quite a low perturbation ratio. Specifically, our method PromptAttackcn (7.7%) achieves 169.0 PPL, showing better fluency than PromptAttackcn (28.1%) and baselines. Although SC-EDA has a lower perturbation ratio than our PromptAttackcn (28.1%), it shows less attack effectiveness (Section 5.1) and worse fluency. Thus, there are trade-offs between perturbation ratio and attack effectiveness.
Second, do the adversarial examples preserve the semantics of the un-perturbed original sentences? That is, does an utterance asking for a cheap restau-
rant lead to an adversarial example that also asks for a cheap restaurant though tricking the DST to output expensive? To answer this question, we conduct a human evaluation on semantics preservation and grammatical correctness. We first shuffled 150 examples: 50 original un-perturbed sentences, 50 adversarial examples with a 7.7% perturbation ratio, and 50 with a 28.1% perturbation ratio (following the analysis in Section 5.3.1). For the adversarial examples, each attacks the victim model successfully leading to an accuracy of 0. Following (Jin et al., 2020; Li et al., 2020), we ask three human judges to rate how well a randomly chosen sentence preserves the semantics of the original sentence (semantic), how grammatically correct the sentence is (grammar), on a scale from 1 to 5. We report the average score across the three judges in Table 3.
As we can see, the semantic score and grammar score of the adversarial examples are close to the original ones. We find that when the perturbation is reasonable (around 8%), the semantics of the original sentence are preserved quite well (scoring 4.3 for adversarial examples). Further, the grammatical quality of the sentence is also maintained well (4.8 versus 4.4). Even as the perturbation ratio increases to approximately 28%, our approach continues to uphold good semantic preservation (3.3) while retaining satisfactory grammar quality (3.8). Overall, our method consistently generates high-quality adversarial examples by effectively preserving semantics, maintaining grammatical quality and fluency, and keeping a low perturbation ratio.

heading: 5.3 Impact of PromptAttack Design (RQ3)
text: We now explore the impact of different settings on our proposed methods.

heading: 5.3.1 Ratio of Perturbed Tokens
text: First, our prompt-based approach can control how many tokens we want to change in the user utterances, which gives it flexibility. Since the perturbation ratio represents the semantic consistency between the original examples and adversarial examples and there are trade-offs between the attack effectiveness and perturbation ratio, it is important to investigate the influence of the ratio of perturbed tokens on attacking ability.
We take max(1, perturbation_ratio∗ lt) as the number of perturbed tokens, where lt denotes the length of pre-processed utterances. We set the perturbation ratio of tokens that we could perturb to
10%, 30%, 50%, 80%, and 100%, that is 7.7%, 10.2%, 15.2%, 22.6%, and 28.1% of the average length of all input examples. More data analysis can be found in Appendix B.
Table 4 shows the evaluation of attack performance and fluency of generated adversarial examples from PromptAttackcx and PromptAttackcn. We observe that for these two methods, the more tokens we perturb, the lower JGA and higher ASR we get, showing better attack ability, which is consistent with our intuition. Thus, as the ratio of perturbed tokens increases, our proposed method PromptAttack achieves better attack performance while maintaining good fluency.

heading: 5.3.2 Prompt Length
text: Next, we explore the effect of different continuous prompt lengths. Shorter prompts have fewer tunable parameters, which means under the same training setting, it would be faster to optimize and find the most effective adversarial prompts. We train continuous prompts with different length: 5 tokens, 10 tokens, and 15 tokens using PromptAttackcx. Table 5 shows that under different prompt lengths, with the increase of perturbation ratio, the model achieves better attack performance. Under the same perturbation ratios, the model with 5-token prompt achieves modest lower JGA and higher ASR. For example, when the perturbation ratio is 28.1%, PromptAttackcx with 5-token prompt gains lower JGA than PromptAttackcx with 10-token prompt
and PromptAttackcx with 15-token prompt by 0.2 and 0.8, respectively, and higher ASR by 0.4 and 1.2, indicating slightly better attack performance.

heading: 5.4 Defense against Attack (RQ4)
text: Finally, we turn to the challenge of defending a DST in the presence of such adversarial examples. We aim to answer two questions: i) can our generated adversarial examples be used to improve the performance of current DST models? and ii) can our attack method bypass such a defense method?
One of the most effective approaches to increase the robustness of a model is adversarial training, which injects adversarial examples into the training data to increase model robustness intrinsically (Bai et al., 2021). Specifically, we first apply our attack methods on the original training dataset to generate adversarial examples. Then we re-train the TripPy model on the training set augmented by the adversarial training examples and evaluate the performance on original test set. As shown in Table 6, the new defended DST model improves JGA on the original test set from 61.3 to 67.3 by 6.0, which outperforms results reported by the state-of-the-art DST model CoCo (62.6) by 4.7. This encouraging result shows that adversarial examples from our attack method can be a good source for data augmentation.
To evaluate the robustness of such an augmented DST model against our proposed attack methods, we next test how well our adversarial examples perform. From Table 6 we observe that the attack methods still show strong attack ability on the new DST model. Thus, there is an opportunity to explore stronger defense methods to strengthen DSTs against such prompt-based attacks.

heading: 6 Conclusion
text: In this paper, we present a prompt-based learning approach that can generate effective adversarial
examples for probing DST models. Through experiments over four state-of-the-art DSTs, our framework achieves the greatest reduction in accuracy with the best attack success rate. Moreover, the generated adversarial examples maintain good fluency and low perturbation ratio, evidence that they are close to legitimate non-adversarial user inputs. We also show our generated adversarial examples can bolster a DST by augmenting the original training data with adversarial examples. We find that both discrete and continuous adversarial prompts are capable of generating effective adversarial examples. Discrete prompts are more interpretable while continuous prompting allows us to search for optimal adversarial prompts more efficiently, and generates more effective adversarial examples.
Limitations
The natural idea to improve robustness is to add adversarial examples to the training set and retrain the model. However, generating adversarial examples for a large training set can be very time-consuming. Thus, it would be interesting to explore more efficient methods that implicitly involved adversarial examples in the training process, e.g., (Yang et al., 2022).
Ethics Statement
The proposed methods could also be applied to natural language generation tasks, like dialogue response generation. The misuse of such methods may generate biased or offensive responses.

heading: Acknowledgements
text: We appreciate the authors of the CoCo paper for making their code accessible to the public and for taking the time to address our inquiries.

heading: B Data Analysis
text: Figure 3 shows the distribution of length of original user utterance lo, the length of utterances after removing stop words, slot and value related tokens lt, and the difference between them, that is ∆ = lo−lt. We can see, 95.5% of user utterances have fewer than 10 tokens that could be perturbed and 59.3% of them could perturb less than 4 tokens.

heading: C Success Generation Rates of Adversarial Examples
text: To assess whether there are any masked tokens kept unchanged after the adversarial example construction process, we evaluate the success generation rates of adversarial examples generated using our methods. we observe that PromptAttackd achieves a success rate of 0.94, PromptAttackcx achieves 0.95, and PromptAttackcn achieves 0.96. These findings indicate that our methods demonstrate a high rate of successful generation. And it is important to note that unchanged utterances will not attack the models successfully.

heading: D Data Augmentation
text: In Section 5.4, TripPy trained on original training data augmented with our generated adversarial examples improves TripPy by 6.0% and outperforms CoCo by 4.7% when evaluated on the original test set following the same post-processing strategy as CoCo (Li et al., 2021b). When using TripPy’s default cleaning, the comparison results with previous methods are shown in Table 7.

heading: E Case Study
text: In Table 8, we showcase some examples of the adversarial examples generated from baseline methods and our methods.

heading: F Error Analysis
text: Although we keep the ground-truth values and slotrelated words unchanged in the adversarial examples, perturbations on surrounding words may lead to low human evaluation results. Here we show some adversarial examples that attack the victim model successfully in Table 9. For case 1, changing “night” to “hour” affects the meaning of the sentence but is consistent with ground truth “1”. However, the victim model predicts “two” for slot
“book_stay”. For cases 2 and 3, we could see by modifying words “well” and “restaurant”, which are not related to ground-truth slot-values, the victim model can not identify “six” and “entertainment” for slot “book_people” and “attraction-type”. For cases 4, 6, and 9, the adversarial examples modified words “star”, “leaves”, and “price range” that are related to slot names “stars”, “departure”, and “pricerange” and cause victim models can not identify the corresponding values. In future work, it’s important to consider more comprehensively to prevent modifying slot-related words. For case 5, the new words “office” confuse the model and make it ignore the values “center” and predict “none” for slot “area”. For cases 7, 9, and 10, “fishing”, “french english”, and “street” in adversarial examples lead victim models prediction new value “fishing boat”, “french”, and “cambridge street” for slots “attraction-type”, “restaurant-food”, and “train-departure”. For case 8, the remove of “arriving” leads victim model predict “none” for “trainarriveBy”.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethics Statement

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract, Section 1 Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4 Experimental Setup

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4 Experimental Setup

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 4 Experimental Setup

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4 Experimental Setup
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4.1 Dataset
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4.1 Dataset, Appendix B
C 3 Did you run computational experiments? Section 4 Experimental Setup

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix A
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix A

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix A

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix A
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 5.2 Adversarial Example Quality
7 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? We report the text of instructions about two metrics (Semantic and Grammar) which need to be annotated in Section 5.2 Adversarial Example Quality. There are no examples of risks in this study.
7 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? The annotators were graduate students in the computer science area and they were not compensated.
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? The annotators knew the data would be used for this paper.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: In future work, it’s important to consider more comprehensively to prevent modifying slot-related words. For case 5, the new words “office” confuse the model and make it ignore the values “center” and predict “none” for slot “area”. For cases 7, 9, and 10, “fishing”, “french english”, and “street” in adversarial examples lead victim models prediction new value “fishing boat”, “french”, and “cambridge street” for slots “attraction-type”, “restaurant-food”, and “train-departure”. For case 8, the remove of “arriving” leads victim model predict “none” for “trainarriveBy”. ACL 2023 Responsible NLP Checklist