user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Detecting Egregious Responses in Neural Sequence-to-sequence Models

Paper content:
In this work, we attempt to answer a critical question: whether there exists some input sequence that will cause a well-trained discrete-space neural network sequence-to-sequence (seq2seq)  model to generate egregious outputs (aggressive, malicious, attacking, etc.).
And if such inputs exist, how to find them efficiently.
We adopt an empirical methodology, in which we first create lists of egregious output sequences, and then design a discrete optimization algorithm to find input sequences that will cause the model to generate them.
Moreover, the optimization algorithm is enhanced for large vocabulary search and constrained to search for input sequences that are likely to be input by real-world users.
In our experiments, we apply this approach to  dialogue response generation models trained on three real-world dialogue data-sets: Ubuntu, Switchboard and OpenSubtitles, testing whether the model can generate malicious responses.
We demonstrate that given the trigger inputs our algorithm finds, a significant number of malicious sentences are assigned large probability by the model, which reveals an undesirable consequence of standard seq2seq training.
Recently, research on adversarial attacks BID5 BID19 has been gaining increasing attention: it has been found that for trained deep neural networks (DNNs), when an imperceptible perturbation is applied to the input, the output of the model can change significantly (from correct to incorrect).
This line of research has serious implications for our understanding of deep learning models and how we can apply them securely in real-world applications.
It has also motivated researchers to design new models or training procedures BID12 , to make the model more robust to those attacks.For continuous input space, like images, adversarial examples can be created by directly applying gradient information to the input.
Adversarial attacks for discrete input space (such as NLP tasks) is more challenging, because unlike the image case, directly applying gradient will make the input invalid (e.g. an originally one-hot vector will get multiple non-zero elements).
Therefore, heuristics like local search and projected gradient need to be used to keep the input valid.
Researchers have demonstrated that both text classification models BID4 or seq2seq models (e.g. machine translation or text summarization) BID2 BID0 are vulnerable to adversarial attacks.
All these efforts focus on crafting adversarial examples that carry the same semantic meaning of the original input, but cause the model to generate wrong outputs.In this work, we take a step further and consider the possibility of the following scenario: Suppose you're using an AI assistant which you know, is a deep learning model trained on large-scale highquality data, after you input a question the assistant replies: "You're so stupid, I don't want to help you."We term this kind of output (aggressive, insulting, dangerous, etc.) an egregious output.
Although it may seem sci-fi and far-fetched at first glance, when considering the black-box nature of deep learning models, and more importantly, their unpredictable behavior with adversarial examples, it is difficult to verify that the model will not output malicious things to users even if it is trained on "friendly" data.In this work, we design algorithms and experiments attempting to answer the question: "Given a well-trained 1 discrete-space neural seq2seq model, do there exist input sequence that will cause it to generate egregious outputs?"
We apply them to the dialogue response generation task.
There are two key differences between this work and previous works on adversarial attacks: first, we look for not only wrong, but egregious, totally unacceptable outputs; second, in our search, we do not require the input sequence to be close to an input sequence in the data, for example, no matter what the user inputs, a helping AI agent should not reply in an egregious manner.In this paper we'll follow the notations and conventions of seq2seq NLP tasks, but note that the framework developed in this work can be applied in general to any discrete-space seq2seq task.
In this work, we provide an empirical answer to the important question of whether well-trained seq2seq models can generate egregious outputs, we hand-craft a list of malicious sentences that should never be generated by a well-behaved dialogue response model, and then design an efficient discrete optimization algorithm to find trigger inputs for those outputs.
We demonstrate that, for models trained by popular real-world conversational data-sets, a large number of egregious outputs will be assigned a probability mass larger than "proper" outputs when some trigger input is fed into the model.
We believe this work is a significant step towards understanding neural seq2seq model's behavior, and has important implications as for applying seq2seq models into real-world applications.
First in FIG1 , we show an illustration of the forwarding process on the encoder side of the neural seq2seq model at time t, which serves as an auxiliary material for Section 2 and Section 3.1.

assistant: text: This paper aims to provide an empirical answer to the question of whether well-trained dialogue response model can output malicious responses.