Title: Detecting Egregious Responses in Neural Sequence-to-sequence Models
Authors: Tianxing He
James Glass
Venue: ICLR 2019
Keywords: Deep Learning
Natural Language Processing
Adversarial Attacks
Dialogue Response Generation

Abstract: In this work, we attempt to answer a critical question: whether there exists some input sequence that will cause a well-trained discrete-space neural network sequence-to-sequence (seq2seq)  model to generate egregious outputs (aggressive, malicious, attacking, etc.). And if such inputs exist, how to find them efficiently. We adopt an empirical methodology, in which we first create lists of egregious output sequences, and then design a discrete optimization algorithm to find input sequences that will cause the model to generate them. Moreover, the optimization algorithm is enhanced for large vocabulary search and constrained to search for input sequences that are likely to be input by real-world users. In our experiments, we apply this approach to  dialogue response generation models trained on three real-world dialogue data-sets: Ubuntu, Switchboard and OpenSubtitles, testing whether the model can generate malicious responses. We demonstrate that given the trigger inputs our algorithm finds, a significant number of malicious sentences are assigned large probability by the model, which reveals an undesirable consequence of standard seq2seq training. 

TL;DR: This paper aims to provide an empirical answer to the question of whether well-trained dialogue response model can output malicious responses.

Review discussion:
# Positive aspects of this submission
- This submission explores a very interesting problem that is often overlooked in sequence-to-sequence models research.
- The methodology in Sections 4 and 5 is very thorough and useful.
- Good comparison of last-h with attention representations, which gives good insight about the robustness of each architecture against adversarial attacks.
# Criticism
- In Section 3, even if the "l1 + projection" experiments seem to show that generating egregious outputs with greedy decoding is very unlikely, it doesn't definitely prove so. It could be that your discrete optimization algorithm is suboptimal, especially given that other works on adversarial attacks for seq2seq models use different methods such as gradient regularization (Cheng et al. 2018).
Similarly, the brute-force results on a simplified task in Appendix B are useful, but it's hard to tell whether the conclusions of this experiment can be extrapolated to the original dialog task.
Given that you also study "o-greedy-hit" in more detail with a different algorithm in Sections 4 and 5, I would consider removing Section 3 or moving it to the Appendix for consistency.
Thanks for the detailed review, here’s responses to the questions:
1) In Section 3, even if the "l1 + projection" experiments seem to show that generating egregious outputs with greedy decoding is very unlikely, it doesn't definitely prove so. It could be that your discrete optimization algorithm is suboptimal, especially given that other works on adversarial attacks for seq2seq models use different methods such as gradient regularization (Cheng et al. 2018).
Similarly, the brute-force results on a simplified task in Appendix B are useful, but it's hard to tell whether the conclusions of this experiment can be extrapolated to the original dialog task.
We agree that our approach is not a proof for the robustness for greedy decoding, but in this work we provide several empirical experiments from different angles (the main result, continuous relaxation and brute-force enumeration) to support that claim.
And you’re right in that our algorithm is not perfect (since the hit rate for the normal list is not 100%, there is room for improvement in the search algorithm). We are aware that the algorithm in (Cheng et al. 2018), in also applicable in our setting.
However, the main contribution of our work is not about determining which algorithm is the best.  We proposed a simple and effective gibbs-enum algorithm, and more importantly used it to demonstrate that the “egregious output” problem exists in standard seq2seq model training.
2) Given that you also study "o-greedy-hit" in more detail with a different algorithm in Sections 4 and 5, I would consider removing Section 3 or moving it to the Appendix for consistency.
The reason we put emphasis on the continuous relaxation experiment in Section 3 is that we believe this is the first natural approach researchers will try in order to find trigger inputs for some target sequence.  We felt that by demonstrating that this doesn’t work, motivated the enumeration based algorithm, such as gibbs-enum.
Thanks for the review!
Main contribution: devising and evaluating an algorithm to find inputs that trigger arbitrary "egregious" outputs ("I will kill you") in vanilla sequence-to-sequence models, as a white-box attack on NLG models.
Clarity:
The paper is overall clear. I found some of the appendices (esp. B and C) to be important for understanding the paper and believe these should be in the main paper. Moving parts of Appendix A in the main text would also add to the clarity.
Originality:
The work looks original. It is an extension of previous attacks on seq2seq models, such as the targeted-keyword-attack from (Cheng et al., 2018) in which the model is made to produce a keyword chosen by the attacker.
Significance of contribution:
The lack of control over the outputs of seq2seq is a major roadblock towards their broader adoption. The authors propose two algorithms for trying to find inputs creating given outputs, a simple one relying on continuous optimization this is shown not to work (breaking when projecting back into words), and another based relying on discrete optimization. The authors found that the task is hard when using greedy decoding, but often doable using sampled decoding (note that in this case, the model will generate a different output every time). My take-aways are that the task is hard and the results highlight that vanilla seq2seq models are pretty hard to manipulate;
however it is interesting to see that with sampling, models may sometimes be tricked into producing really bad outputs.
This white-box attack applicable to any chatbot. As the authors noted, an egregious output for one application ("go to hell" for customer service) may not be egregious for another one ("go to hell" in MT).
Overall, the authors ask an interesting question: how easy is it to craft an input for a seq2seq model that will make it produce a "very bad" output. The work is novel, several algorithms are introduced to try to solve the problem and a comprehensive analysis of the results is presented. The attack is still of limited practicality, but this paper feels like a nice step towards more natural adversarial attacks in NLG.
One last thing: the title seems a bit misleading, the work is not about "detecting" egregious outputs.
Thanks for the detailed review, here’s responses to the advice and questions:
1) I found some of the appendices (esp. B and C) to be important for understanding the paper and believe these should be in the main paper. Moving parts of Appendix A in the main text would also add to the clarity.
Thanks for reading the appendices!  We agree that it would be our preference to move them into the main body of the paper, but we were constrained by the 10 page limit.
2) The lack of control over the outputs of seq2seq is a major roadblock towards their broader adoption. The authors propose two algorithms for trying to find inputs creating given outputs, a simple one relying on continuous optimization this is shown not to work (breaking when projecting back into words), and another based relying on discrete optimization. The authors found that the task is hard when using greedy decoding, but often doable using sampled decoding (note that in this case, the model will generate a different output every time). My take-aways are that the task is hard and the results highlight that vanilla seq2seq models are pretty hard to manipulate;
however it is interesting to see that with sampling, models may sometimes be tricked into producing really bad outputs.
This white-box attack applicable to any chatbot. As the authors noted, an egregious output for one application ("go to hell" for customer service) may not be egregious for another one ("go to hell" in MT).
Overall, the authors ask an interesting question: how easy is it to craft an input for a seq2seq model that will make it produce a "very bad" output. The work is novel, several algorithms are introduced to try to solve the problem and a comprehensive analysis of the results is presented. The attack is still of limited practicality, but this paper feels like a nice step towards more natural adversarial attacks in NLG.
Your understanding about the conclusions and limitations of the this work is correct.  These are the main ideas we try to convey in the paper.
3) One last thing: the title seems a bit misleading, the work is not about "detecting" egregious outputs.
It is true, that we are looking for trigger inputs that would cause the model to output egregious targets in a given list.
Thus we agree that  “detecting” could be a bit misleading….
But we don’t have better word choice for now.  Any suggestions are welcome!
Thanks for the review!
This paper explores the task of finding discrete adversarial examples for (current) dialog models in a post hoc manner (i.e., once models are trained).
In particular, the authors propose an optimization procedure for crafting inputs (utterances) that trigger trained dialog models to respond in an egregious manner.
This line of research is interesting as it relates to real-world problems that our models face before they can be safely deployed. The paper is easy to read, nicely written, and the proposed optimization method seems reasonable. The study also seems clear and the results are fairly robust across three datasets. It was also interesting to study datasets which, a priori, seem like they would not contain much egregious content (e.g., Ubuntu "help desk" conversations).
My main question is that after reading the paper, I'm not sure that one has an answer to the question that the authors set out to answer.
In particular, are our current seq2seq models for dialogs prone to generating egregious responses? On one hand, it seems like models can assign higher-than-average probability to egregious responses. On the other, it is unclear what this means.
For example, it seems like the possibility that such a model outputs such an answer in a conversation might still be very small. Quantifying this would be worthwhile.
Further, one would imagine that a complete dialog system pipeline would contain a collection of different models including a seq2seq model but also others. In that context, is it clear that it's the role of the seq2seq model to limit egregious responses?
A related aspect is that it would have been interesting to explore a bit more the reasons that cause the generation of such egregious responses. It is unclear how representative is the example that is detailed ("I will kill you" in Section 5.3). Are other examples using words in other contexts?
Also, it seems reasonable that if one wants to avoid such answers, countermeasures (e.g., in designing the loss or in adding common sense knowledge) have to be considered.
Other comments:
- I am not sure of the value of Section 3.
In particular, it seems like the presentation of the paper would be as effective if this section was summarized in a short paragraph (and perhaps detailed in an appendix).
- Section 3.1, "continuous relaxation of the input embedding", what does that mean since the embedding already lives in continuous space?
- I understand that your study only considers (when optimizing for egregious responses)) dialogs that are 1-turn long. I wonder if you could increase hit rates by crafting multiple inputs at once.
- In Section 4.3, you fix G (size of the word search space) to 100. Have you tried different values? Do you know if larger Gs could have an impact of reported hit metrics?
- In Table 3, results from the first column (normal, o-greedy) seem interesting. Wouldn't one expect that the model can actually generate (almost) all normal responses? Your results indicate that for Ubuntu models can only generate between 65% and 82% of actual (test) responses. Do you know what in the Ubuntu corpus leads to such a result?
- In Section 5.3, you seem to say that the lack of diversity of greedy-decoded sentences is related to the low performance of the "o-greedy" metric. Could this result simply be explained because the model is unlikely to generate sentences that it has never seen before?
You could try changing the temperature of the decoding distribution, that should improve diversity and you could then check whether or not that also increases the hit rate of the o-greedy metric.
- Perhaps tailoring the mal lists to each specific dataset would make sense (I understand that there is already some differences in between the mal lists of the different datasets but perhaps building the lists with a particular dataset in mind would yield "better" results).
Thanks for the detailed review, here’s responses to the advice and questions:
1) My main question is that after reading the paper, I'm not sure that one has an answer to the question that the authors set out to answer.
In particular, are our current seq2seq models for dialogs prone to generating egregious responses? On one hand, it seems like models can assign higher-than-average probability to egregious responses. On the other, it is unclear what this means.
For example, it seems like the possibility that such a model outputs such an answer in a conversation might still be very small. Quantifying this would be worthwhile.
One clear observation that can be made from the experiments regarding greedy decoding is that the model is very robust against egregious outputs, at least those used in the experiments.
Unless one is using data-sets like Opensubtitles.  With regards to sampling, the reviewer is correct, but, since we are dealing with large vocabulary seq2seq models, the actual probability assigned to any sequence will be very small.   We believe that a very natural and desirable quality of the model is that “the probability assigned to a bad sentence should not be larger than the probability of a good(reference) sentence.”  Unfortunately, our experiments clearly show that this is not the case, which is alarming.
2) Further, one would imagine that a complete dialog system pipeline would contain a collection of different models including a seq2seq model but also others. In that context, is it clear that it's the role of the seq2seq model to limit egregious responses?
This is a good question but we believe that it is slightly out of the scope of this paper because we are examining End-to-End seq2seq models (in part because they have gained increasing popularity in recent years).  The reviewer is correct that one can have additional modules in the pipeline to prevent bad responses from by the system, but we also believe that, ideally, the seq2seq models should be robust against egregious behavior by themselves.
3) A related aspect is that it would have been interesting to explore a bit more the reasons that cause the generation of such egregious responses. It is unclear how representative is the example that is detailed ("I will kill you" in Section 5.3). Are other examples using words in other contexts?
Also, it seems reasonable that if one wants to avoid such answers, countermeasures (e.g., in designing the loss or in adding common sense knowledge) have to be considered.
To the first question, “I will kill you” is just one example, and we can do this for many alternatives. The key is that we believe the model is doing a good job of generalizing, but it does not know that some sentences are not proper to generate.
For example, people talk about “hating something”, and “you” is a noun, so the model could generalize to “I hate you”.   People also talk about “passwords”, but the model doesn’t know one should not ask “What’s your password?”
As to the second question, we believe the reviewer is suggesting future work, and we agree that these are exciting directions to pursue in the future.
4) I am not sure of the value of Section 3.
In particular, it seems like the presentation of the paper would be as effective if this section was summarized in a short paragraph (and perhaps detailed in an appendix).  Section 3.1, "continuous relaxation of the input embedding", what does that mean since the embedding already lives in continuous space?
To the first question, we agree with the suggestion that Section 3 can be shortened. The reason we put emphasis on the continuous relaxation experiment is that we believe this is the first approach researchers will try in order to find trigger inputs for some target sequence.  We thought that pointing out that this doesn’t work served as a useful motivation to turn to a enumeration based algorithm, such as gibbs-enum.
For the second (clarifying) question, it’s true that the embedding lives in continuous space, but they are constrained to be one of the columns in the embedding matrix E^{enc} in the trained model. By “continuous relaxation of the input embedding” we mean that we remove the column constraint, and allow the vector to be any continuous vector. We’ll add the explanation to the paper.
5) I understand that your study only considers (when optimizing for egregious responses)) dialogs that are 1-turn long. I wonder if you could increase hit rates by crafting multiple inputs at once.
One of the points of our work is that even if you just manipulate a 1-turn history, it is enough to trigger egregious outputs. Examining multi-turn histories will be a good subject for future work.  For us, it will involve re-implementing code and re-running experiments.  Our current expectation is that when you manipulate multi-turn history, that the hit rates will increase, but not significantly.
6) In Table 3, results from the first column (normal, o-greedy) seem interesting. Wouldn't one expect that the model can actually generate (almost) all normal responses? Your results indicate that for Ubuntu models can only generate between 65% and 82% of actual (test) responses. Do you know what in the Ubuntu corpus leads to such a result?
This is a good question.  If our algorithm is perfect, the result should be 100%. This result shows that there still remains room for (maybe big) improvements for the trigger input search algorithm.
That is also a good future research direction.  We believe that the Ubuntu result is somewhat special in that, as we tried to explain, due to the “generic response” situation we see in both the Switchboard and Opensubtitles data, we switch to sampling for the normal list.
Thus, the reason could be that a greedy-hit is a stronger constraint than sample-hit, and it is more difficult to find trigger inputs for that.
7) In Section 5.3, you seem to say that the lack of diversity of greedy-decoded sentences is related to the low performance of the "o-greedy" metric. Could this result simply be explained because the model is unlikely to generate sentences that it has never seen before?
That is a plausible explanation but we believe this problem is somewhat special due to the dialogue response setting.
When doing greedy decoding, the model tends to give very common outputs.  For other tasks like machine translation, from greedy decoding you will get very good outputs (things never seen in the data).
8) You could try changing the temperature of the decoding distribution, that should improve diversity and you could then check whether or not that also increases the hit rate of the o-greedy metric.
That is a good suggestion, and could indeed improve diversity.  It is less clear to us whether that will change the greedy decoding behavior however, because changing the temperature should not change which element is the maximum.  Do you agree?
9) Perhaps tailoring the mal lists to each specific dataset would make sense (I understand that there is already some differences in between the mal lists of the different datasets but perhaps building the lists with a particular dataset in mind would yield "better" results).
This is also good advice, and could make the “attack” more powerful.  Since that approach is more time consuming, for our initial effort we tried to create general malicious targets that should be applicable to a wide range of dialogue data.
Thanks for the review!
Thanks a lot for taking the time to reply to all of my questions/comments.
1) You write: "We believe that a very natural and desirable quality of the model is that “the probability assigned to a bad sentence should not be larger than the probability of a good(reference) sentence.”  Unfortunately, our experiments clearly show that this is not the case, which is alarming."
I still think that it would be useful to quantify this, e.g.
in terms of where does that sentence rank according to some decoding strategy. I cannot completely convince myself that above average is that bad given that the space of all sentences is large.
6) You write: " If our algorithm is perfect, the result should be 100%. This result shows that there still remains room for (maybe big) improvements for the trigger input search algorithm. "
That's interesting. I was under the impression that it was a limitation of the seq2seq model (i.e., it could not actually generate all responses). I guess I misunderstood this. Thanks for clarifying.
8) You write: "It is less clear to us whether that will change the greedy decoding behavior however, because changing the temperature should not change which element is the maximum.  Do you agree?"
Good point, I agree. You'd have to look further down the list or sample. Thanks.
Good point, we agree that studying how the egregious output rank in the beam-search list will give a better sense of how bad or not bad the situation is.
Before that, let us emphasis the reasons behind how we define sample_hit:
1. This definition is intuitive and data/sequeunce_length/vocab invariant, because it only compares the average log-likelihood the trained model assigned to the egregious outputs and reference outputs.
However, the rank in the decoding list is obviously, data/length/vocab variant.
For example, when the target length is longer or the vocabulary is larger, the target will get lower rank, but it doesn't mean the model is safe.
2. The trigger inputs for sample_hit definition is more straightforward to optimize, but for rank it will be more involved.
But we agree it remains an important question whether the sample_hit is the best definition for egregious outputs (it depends on what kind of guarantee you want your generator to have), any discussion or advice are very welcome.
Here's a study about where the egregious output rank in the beam-search list, on Ubuntu and OpenSubtitles data-sets:
1. Given the trigger input and the mal target pairs (for io_sample_min_hit) our algorithm found, we do a beam-search during decoding with a beam size of 1000. And check whether the mal target sequence is found in the 1000-best-list.
For Ubuntu, only very few mal target (2% among the hit ones) appear in the 1000-list.
Also, the experiment revealed that the list is dominated by generic responses.
However, this is not surprising. It is hard to locate a specific sequence in beam-search for a large vocabulary data, you need to use an enormous beam size, which is too costly.
For OpenSubtites, 10% among the hit targets appear in the 1000-list. This is alarming, but we believe it is due to the reason that the OpenSubtiltes data itself contains a lot of dirty sentences.
2. Since the exact seq-level rank maybe unavailable, we switch to report a word-level rank statistics. By word-level rank we mean given the right history, what is the rank of the target word in the distribution P(y_t|x,y_{1...t-1}), this statistics is very efficient to compute.
The result is that the average word-level rank for Ubuntu is 3.09, for OpenSubtitles it is 1.80 so it is on average ranked among the first 0.01% in the vocabulary.
This result shows that on word-level, the malicious target sequence has a very high rank.
Below is two examples of beam-search in the OpenSubtiltes data:
mal target: 
 that 's enough 

trigger_input: it 's one of you , you n't i 're has guess to tom <eou>
beam_id 0: 
 no . 

beam_id 1: 
 what ? 

beam_id 2: 
 no ! 

beam_id 220: 
 you 're right . 

beam_id 221: 
 you think so ? 

beam_id 222: 
 what 's the matter with you ? 

beam_id 223: 
 that 's enough 

mal target: 
 leave me alone . 

trigger input: was same in 's hi you . , you 've find ca wait ! <eou>
beam_id 0: 
 come on ! 

beam_id 1: 
 <unk> ! 

beam_id 2: 
 no ! 

beam_id 3: 
 where are you going ? 

beam_id 4: 
 stop ! 

beam_id 5: 
 hurry up ! 

beam_id 41: 
 come here ! 

beam_id 42: 
 no , no , no . 

beam_id 43: 
 leave me alone . 

We look forward to hear what you think about these results.
Yes, as stated in our paper, the "normal" list is specially designed to test the ability of the algorithm, and a perfect trigger input search algorithm should get 100% hit rate. Note that to the best of our knowledge, this "ability test" is not conducted in NLP adversarial attack literature before. Due to the difficulty of discrete-space optimization, the result that the algorithm fail to find the adversarial input, doesn't mean it doesn't exist.
Thanks for providing these. Unfortunately, I don't have useful insights about other possible metrics.
I think it would be nice to add a short paragraph about some of these results in the paper.
When you say that the "average word-level rank for Ubuntu is 3.09, for OpenSubtitles it is 1.80". Is that averaged across all words in an utterance from the mal-list?
Yes, will do. We think it is an interesting and informative investigation(thanks for the suggestion), and we will add these to the final version of the paper(if accepted).
Sorry, let us clarify: we first take all the target sentences that are "hit" w.r.t io_sample_min_hit in the mal-list(which is about 10% among all targets), then average the word-level rank during decoding for all the words in these "hit" target sentences.

Meta-review:
role: AC
data: metareview: This work examines how to craft adversarial examples that will lead trained seq2seq models to generate undesired outputs (here defined as, assigning higher-than-average probability to undesired outputs). Making a model safe for deployment is an important unsolved problem and this work is looking at it from an interesting angle, and all reviewers agree that the paper is clear, well-presented, and offering useful observations. While the paper does not provide ways to fix the problem of egregious outputs being probable, as pointed out by reviewers, it is still a valuable study of the behavior of trained models and an interesting way to "probe" them, that would likely be of high interest to many people at ICLR.

Decision: Accept (Poster)