言語処理学会 第30回年次大会 発表論文集（2024年3月）: Self Data Augmentation for
Out-of-Scope Detection in Dialogues
Chunpeng Ma and Takuya Makino
Megagon Labs, Tokyo, Recruit Co., LTD.
{ma.chunpeng,makino}@megagon.ai

Abstract
Detecting out-of-scope (OOS) utterances is crucial intask-oriented dialogue systems, but obtaining enough annotated OOS dialogues to train a binary classiﬁer directly is
diﬃcult in practice. Existing data augmentation methods
generate OOS dialogues automatically, but their performance usually depends on an external corpus. Herein we
propose SILVER, a self data augmentation method that
does not use external data. It improves the accuracy of
OOS detection (false positive rate: 90.5% → 47.4%).
Furthermore, SILVER successfully generates high-quality
in-domain (IND) OOS dialogues in terms of naturalness(percentage: 8% → 68%) and OOS correctness (percentage: 74% → 88%), as evaluated by human workers.

1

Introduction

Task-oriented dialogue systems [2, 3] require humanoperators to deal with intentions that are beyond their capacities, raising the issue of out-of-scope (OOS) detection.
Due to the lack of OOS annotations in open-world settings, previous research usually detects OOS samples indirectly resorting to in-scope (INS) samples. Recently, data
augmentation methods [4, 5] have made it possible to detect
OOS directly using a binary classiﬁer. One such method
is GOLD [6]. It uses simple rules to replace utterancesin known OOS dialogues with sentences selected from alarge pool. However, the dependence on external corpora
prevented the realization of GOLD s full potential.
We propose a method called Self Iterative OOS Labeling
via Ensembling Trees (SILVER), overcoming issues ofGOLD. It build pools from training data, detects OOS
This is a shortened version of the paper [1] published in the 13th
International Joint Conference on Natural Language Processing andthe 3rd Conference of the Asia-Paciﬁc Chapter of the Association
for Computational Linguistics.

Figure 1 Comparison of GOLD and SILVER. To automaticallygenerate an OOS dialogue, SILVER replaces the ﬁrst utterancewith an IND utterance, while GOLD replaces the third utterancewith an OOD utterance, making the dialogue become OOD andincomprehensible.

using an ensemble of decision trees [7], and generates
OOS dialogues iteratively. Figure 1 compares dialogues
generated by GOLD and SILVER.

2 GOLD: Generating Out-of-scope
Labels with Data Augmentation
GOLD [6] is the data augmentation method most closely
related to this work. Given a small set of annotated OOS
dialogues (1% of the size of INS), GOLD replaces utterances with sentences selected from an external pool to generate new OOS dialogues. Selected sentences should be in
the neighborhood of the original utterances. Then GOLDﬁlters the generated OOS, and combines predictions ofdiﬀerent methods via majority voting. Filtered OOS dialogues are concatenated with the original annotated OOS
dialogues and are used to train a binary classiﬁer.
GOLD has a practical appeal. Labor-intensive data col-― 1411 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

lection and annotation of OOS are unnecessary, and theselecting the black points in Figure 3 should be avoided.

data augmentation method is orthogonal to the classiﬁca-

Therefore, only appropriate utterances in the sense oftion improvements. Both advantages extend its applica-

high similarity and high divergence (i.e., green points inbility to real scenarios. However, dependence on external

Figure 3) should be selected. In practice, utterances arecorpora limits its performance.

selected from the set N(16) − N(4), where N(𝑘) is the set

3of 𝑘-nearest utterances from the original utterance.

SILVER: Methodology
Figure 2 outlines the framework of SILVER. First, we

3.2

Tree ensemble

sample a small set of dialogues from the training data.

SILVER classiﬁes OOS candidates via the gradient tree

These dialogues are known OOS. Then candidates are gen-boosting algorithm [8]. Feature sets consist of three parts.

erated by randomly choosing one utterance from seed OOS(1) Probability-based feature.

An intent classiﬁer

dialogues and swapping it with an utterance extracted fromis trained as the supporting model. Then, given a di-

INS (ref. §3.1). After generating numerous candidates, analogue 𝑑, the supporting model outputs the probability

ensemble classiﬁer is used for election (ref. §3.2). Se-

distribution over all intent labels: [ 𝑝 1 (𝑑), . . . , 𝑝 𝑙 (𝑑)],where 𝑙 is the number of possible intent labels, andlected dialogues are concatenated to seed OOS samples,increasing the number of available OOS dialogues. Iter-∀𝑖 ∈ [1, 𝑙], 0 ≤ 𝑝 𝑖 (𝑑) ≤ 1. Based on this probability

ating this process several times provides suﬃcient data to

distribution, the probability-based feature is calculated as:
𝑋 𝑝𝑟 𝑜𝑏 (𝑑) = [𝜎 −1 ( 𝑝 1 (𝑑)), . . . , 𝜎 −1 ( 𝑝 𝑙 (𝑑))],train a binary classiﬁer for OOS detection (ref. §3.3).

3.1

where 𝜎(·) is the standard logistic function.

Self candidate generation

Candidates are generated by swapping utterances in seed
OOS dialogues with those in the pool. To achieve this, two(2) Distance-based feature.

All ut-terances of INS dialogues in the training data are used tobuild the utterance pool because we aim to generate candidates without using external corpora. Furthermore, for atask-oriented dialogue system, we assume that utterancesfrom the user and system are in diﬀerent clusters. Hence,

Given a dialogue 𝑑, thedistance-based feature is calculated as below:
𝑋𝑑𝑖𝑠𝑡 (𝑑) = [Dist(ℎBERT (𝑑), ℎBERT ( D1 )),questions must be answered.
How should the utterance pool be built?

(1). . . , Dist(ℎBERT (𝑑), ℎBERT ( D𝑙 ))],

(2)

where Dist(·, ·) is the cosine distance between two vectors,
and ℎBERT (𝑑) is the representation of the last hidden layer
given input 𝑑. D𝑖 is the collection of all dialogues withintent label 𝑖, and ℎBERT ( D𝑖 ) is their average.

two pools are built: (1) one for system utterances and (2)(3) Ensemble-based feature. This is the average ofthe output probability distributions of three diﬀerent runs

one for user utterances.

by randomly dropping out diﬀerent nodes of the baseline

How should an appropriate utterance be selected?

intent classiﬁer, which is given as:
𝑋 𝑑𝑟 𝑜 𝑝 (𝑑) =][ 3
31 ∑ −1 𝑘
1 ∑ −1 𝑘𝜎 ( 𝑝 1 (𝑑)), . . . ,𝜎 ( 𝑝 𝑙 (𝑑)) ,
3 𝑘=1
3 𝑘=1

Two criteria are considered to determine appropriate utterances: (1) high similarity to the original utterance and (2)high divergence between each other. Figure 3 illustrates
their trade-oﬀ.
(1) High similarity: By selecting utterances similar tothe original one, the naturalness of the original OOS dialogue is kept. This means that the blue points in Figure
3, which were selected by GOLD, will not be selected by
SILVER.

(3)

where 𝑝 𝑖𝑘 (𝑑) is the probability of intent label 𝑖 at the 𝑘-th
run given dialogue 𝑑, after dropping out some nodes of theneural network. The dropout percentage is 10%.
The ﬁnal feature set is the concatenation of 𝑋 𝑝𝑟 𝑜𝑏 , 𝑋𝑑𝑖𝑠𝑡
and 𝑋𝑑𝑟 𝑜 𝑝 . It is trained on sampled training data. Thus,
no extra annotation is needed.

(2) High divergence: Dialogues generated by simply
modifying some words in the original utterances do not
improve the classiﬁcation performance. We hope the generated dialogues diﬀer from each other. This means that

3.3

Iterative data augmentation

SILVER generates suﬃcient data in an iterative manner.
After each iteration, newly generated dialogues are aggre-― 1412 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Figure 2
Modules

Framework of SILVER.

STAR

FLOW

ROSTD

Pool Elect Iter.

AUROC↑AUPR↑

FPR@.95↓

FPR@.90↓

AUROC↑AUPR↑

FPR@.95↓

FPR@.90↓

AUROC↑AUPR↑

FPR@.95↓

FPR@.90↓
1.28%

Ext.

Rnd.

7

0.78270.3618

90.5%

77.8%

0.66920.1503

89.1%

80.3%

0.99180.9224

1.96%

Ext.

MV

7

0.84560.4501

75.4%

59.7%

0.71110.1789

83.7%

76.4%

0.99670.9613

0.30%

0.30%

Ext.

TE

7

0.86320.4721

63.9%

48.4%

0.72870.2183

79.2%

72.1%

0.99850.9805

0.15%

0.13%

Ext.

TE

3

0.88580.4906

56.9%

38.3%

0.73730.2299

76.9%

69.3%

0.9991

0.9910

0.09%

0.09%

Int.

Rnd.

7

0.78430.2623

82.7%

74.3%

0.78250.2608

79.7%

72.4%

0.8594

0.2967

31.7%

15.0%

Int.

MV

7

0.83630.3618

71.5%

49.3%

0.80300.2995

71.3%

60.6%

0.9966

0.9744

0.25%

0.09%

Int.

TE

7

0.86430.3952

60.7%

40.4%

0.82150.3368

56.9%

45.1%

0.9971

0.9680

0.22%

0.13%

Int.

TE

3

0.89920.4212

47.4%

33.9%

0.83190.3379

55.0%

43.9%

0.9971

0.9680

0.22%

0.13%

GOLD

0.86830.4450

56.0%

40.9%

0.80220.3243

60.6%

49.5%

0.99900.9933

0.17%

0.09%

Table 1 Results of OOS detection. Column “Pool” means whether external (Ext.) data (PersonaChat [9]) or internal (Int.) data (i.e.,
training data) is used to generate utterance pool. Column “Elect” gives diﬀerent election methods: random selection (Rnd.), majority
voting (MV), or tree ensemble (TE). Column “Iter.” indicates whether dialogues are generated iteratively (3) or not (7). Therefore,
Int. + TE + 3means all components of SILVER are applied. Best and runner-up of diﬀerent conﬁgurations are denoted by bold andunderlined texts, respectively. Last line is copied from [6].

dialogues increases rapidly, reaching the target number in
only a few iterations.

Figure 3

Trade-oﬀ between similarity and divergence whenselecting appropriate utterances from an utterance pool.

4

Experiments

4.1

Datasets and conﬁgurations

We conducted experiments on STAR [10], FLOW [11]and ROSTD [12] data.
The supporting model was a classiﬁer ﬁnetuned on thegated and considered known OOS dialogues. Then these
are used to generate more dialogues in the next iteration.
This iterative process generates high-quality dialogues
with high eﬃciency.
High quality.

The candidate list is kept small, and con-tains only appropriate (i.e., high similarity & divergence)
dialogues, which are rarely INS dialogues. When combined with a powerful ensemble classiﬁer, the generated
dialogues have a satisfactory quality.
High eﬃciency.

Because INS dialogues rarely exist

in the candidate list, many generated dialogues remain after election. Consequently, the number of available OOS

task of intent classiﬁcation, consisting of a pretrained
BERT model1）, with two feed-forward layers above. The
model inputs were the ﬁrst 256 words of the dialogues. The
model was optimized using the Adam algorithm [13].
We forced the size of seed OOS dialogues to be 1% ofINS, and the target number of generated OOS dialogues
was 24 times the seed size.
Experiment results were evaluated using the following
metrics: (1) AUROC, area under receiver operating characteristic curve, (2) AUPR, area under precision-recall curve,and (3) FPR@𝜃, false positive rate with threshold 𝜃.
1） https://huggingface.co/bert-base-uncased― 1413 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Method

OOS

NaturalnessGOLD
SILVER

74%
88%

8%
68%

Table 2

Human evaluation results of 50 dialogues generated
by GOLD and SILVER. Numbers are the percentages of real
OOS/natural dialogues.

# Unique utterancesin generated data

#Unique utterances
in utterance poolGOLD4, 15393, 472

SILVER5, 28932, 320

Table 3

4.2

Numbers of unique utterances.

Results on OOS detection

Table 1 shows the key experiment results of SILVER for
OOS detection. To verify the eﬀectiveness of the three key

Figure 4

Visualization of the original and generated OOS dia-logues.

components of SILVER corresponding to §3.1 to §3.3, we
modiﬁed or removed some of these components. We canin the generated data and the utterance pools. Although

see that except for special cases (e.g., length-1 dialogues

SILVER utilized a much smaller pool (built on training

in ROSTD), combining all three components of SILVER

data), the generated data contains more unique utterances,

achieves the best performance for OOS detection.

indicating a larger diversity.

4.3

SILVER generated IND OOS data.

Evaluation of data quality

An advantage of

SILVER is its ability to generate INS OOS dialogues. We

Next, we analyzed the quality of the generated data. It

calculated the representations of the original OOS dia-should be emphasized that the quality of generated dia-logues and the dialogues generated by GOLD or SILVER

logues is evaluated intrinsically not extrinsically. Speciﬁ-

using vanilla RoBERTa [14]. Figure 4 shows the 2-dim t-cally, we focus on evaluating (1) the quality of the gen-SNE visualization [15] of these representations along witherated dialogue itself and (2) the generated data as awhole. Herein the gain of the classiﬁcation performancethe average Mahalanobis distances between clusters. The
OOS dialogues generated by GOLD diﬀered from the orig-

contributed by generated data is not considered. Although

inal ones, indicating that these dialogues are OOD. In con-intrinsic high-quality does not necessarily contribute totrast, the overlap between the original OOS and SILVER-

extrinsic tasks directly, it is indispensable in practice.

generated dialogues is large, implying that SILVER gener-

To evaluate the quality of generated dialogue itself, we
evaluate whether the generated dialogues are (1) OOS andates IND data.

5

Conclusion(2) natural. The second row of Table 2 shows the human evaluation results of dialogues generated by SILVER.

We proposed SILVER to generate OOS dialogues with-

Compared with the ﬁrst row, SILVER outperformed GOLDon both evaluation metrics.

out using external data. The components in SILVER are

To evaluate the quality of generated data as a whole, we

state-of-the-art augmentation methods. Using only train-compared the generated data of GOLD and SILVER withing data, SILVER successfully generated high-quality IND

the original data, resulted in the following observations.

OOS dialogues, which not only contributed to the improved

designed to overcome issues and realize the full potential ofis possible that one utterance was selected twice duringperformance of extrinsic tasks such as OOS detection, but
are also natural enough intrinsically, indicating the poten-

generation. This reduces the diversity of the generated

tial for future applications.

SILVER-generated data has a larger diversity.

Itdata. Table 3 shows the numbers of unique utterances― 1414 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

References[1] Chunpeng Ma and Takuya Makino. Silver: Self data augmentation for out-of-scope detection in dialogues. In Proceedings of the 13th International Joint Conference
on Natural Language Processing and the 3rd Conference of the Asia-Paciﬁc Chapter of the Association
for Computational Linguistics, pp. 26–38, Nusa Dua,Bali, November 2023. Association for Computational Linguistics.
[2] Pawe l Budzianowski, Tsung-Hsien Wen, Bo-Hsiang
Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan,and Milica Gašić. MultiWOZ - a large-scale multidomain Wizard-of-Oz dataset for task-oriented dialogue
modelling. In Proceedings of the 2018 Conference on
Empirical Methods in Natural Language Processing,pp. 5016–5026, Brussels, Belgium, October-November
2018. Association for Computational Linguistics.
[3] Ssu Chiu, Maolin Li, Yen-Ting Lin, and Yun-Nung Chen.
SalesBot: Transitioning from chit-chat to task-oriented dialogues. In Proceedings of the 60th Annual Meeting of
the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6143–6158, Dublin, Ireland,

May 2022. Association for Computational Linguistics.
[4] Nathan Ng, Kyunghyun Cho, and Marzyeh Ghassemi.
SSMBA: Self-supervised manifold based data augmentation for improving out-of-domain robustness. In Proceedings of the 2020 Conference on Empirical Methods
in Natural Language Processing (EMNLP), pp. 1268–

1283, Online, November 2020. Association for Computational Linguistics.
[5] Evgeniia Razumovskaia, Ivan Vulić, and Anna Korhonen.
Data augmentation and learned layer aggregation for improved multilingual language understanding in dialogue. In
Findings of the Association for Computational Linguistics: ACL 2022, pp. 2017–2033, Dublin, Ireland,[10] Johannes EM Mosig, Shikib Mehri, and Thomas Kober.
Star: A schema-guided dialog dataset for transfer learning.
arXiv preprint arXiv:2010.11853, 2020.
[11] Jacob Andreas, John Bufe, David Burkett, Charles Chen,
Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang, Alan Guo,
David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy
Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy,
Theo Lanman, Percy Liang, Christopher H. Lin, Ilya Lintsbakh, Andy McGovern, Aleksandr Nisnevich, Adam Pauls,
Dmitrij Petters, Brent Read, Dan Roth, Subhro Roy, Jesse
Rusak, Beth Short, Div Slomin, Ben Snyder, Stephon
Striplin, Yu Su, Zachary Tellman, Sam Thomson, Andrei Vorobev, Izabela Witoszko, Jason Wolfe, Abby Wray,
Yuchen Zhang, and Alexander Zotov. Task-Oriented Dialogue as Dataﬂow Synthesis. Transactions of the Association for Computational Linguistics, Vol. 8, pp.
556–571, 09 2020.
[12] Varun Gangal, Abhinav Arora, Arash Einolghozati, andSonal Gupta. Likelihood ratios and generative classiﬁers
for unsupervised out-of-domain detection in task oriented
dialog. Proceedings of the AAAI Conference on Artiﬁcial Intelligence, Vol. 34, No. 05, pp. 7764–7771, Apr.
2020.
[13] Diederik P Kingma and Jimmy Ba. Adam: A method forstochastic optimization. arXiv preprint arXiv:1412.6980,
2014.
[14] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke
Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint
arXiv:1907.11692, 2019.
[15] Laurens Van der Maaten and Geoﬀrey Hinton. Visualizing data using t-sne. Journal of machine learning
research, Vol. 9, No. 11, 2008.

May 2022. Association for Computational Linguistics.
[6] Derek Chen and Zhou Yu. GOLD: Improving out-of-scope
detection in dialogues using data augmentation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 429–442,

Online and Punta Cana, Dominican Republic, November
2021. Association for Computational Linguistics.
[7] Llew Mason, Jonathan Baxter, Peter Bartlett, and Marcus Frean. Boosting algorithms as gradient descent. Advances in neural information processing systems,
Vol. 12, , 1999.
[8] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable
tree boosting system. In Proceedings of the 22nd acm
sigkdd international conference on knowledge discovery and data mining, pp. 785–794, 2016.

[9] Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur
Szlam, Douwe Kiela, and Jason Weston. Personalizing
dialogue agents: I have a dog, do you have pets too?
In Proceedings of the 56th Annual Meeting of the
Association for Computational Linguistics (Volume
1: Long Papers), pp. 2204–2213, Melbourne, Australia,

July 2018. Association for Computational Linguistics.

― 1415 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

ASTAR dataset, it takes about 10 minutes for each epoch.

Details of Datasets

We stop training after 13 epochs, and select the model withThroughout this paper, we conducted experiments on
three datasets: STAR [10], FLOW [11] and ROSTD [12].

the largest AUROC on the development data as the ﬁnal
model.

Statistics of each dataset is shown in Table 4. All theseFor evaluation,

we resort to scikit-learn li-

datasets consist of task-oriented dialogues. Each dialogue

brary.6）

consists of one or several utterances between human andaverage precision score functions to calculate AU-

chatbot/human. All utterances are in English.

ROC and AUPR, respectively. Calculation of FPR@𝜃

split

STAR

FLOW

ROSTD

train
dev
test

22,051/1,248
2,751/178
2,708/168

60,119/4,499
3,239/228
3,227/239

30,521/3,200
4,181/453
8,621/937

Table 4

Speciﬁcally,

we use roc auc score andis implemented by ourselves, by simply combining
roc curve function with binary search.

C

Numbers of INS/OOS dialogues in each dataset.

Details of Human Annotation
To evaluate the quality of automatically generated dia-

All these datasets follow the MIT license. Copyrights

logues, we randomly sampled 50 dialogues and ask humanbelong to their creators. Our use of these datasets wasworkers to check them manually. Human evaluation results
have been reported in Table 2. In this section, we reportconsistent with their intended use, i.e., for the research on
dialogues of natural languages. All datasets are suﬃciently
anonymized to make identiﬁcation of individuals impossible. We randomly sampled 100 dialogues, and asked

more details of human annotation.
For OOS correctness, we gave annotators the following
instruction.

human workers to check these dialogues. We found that

Is this dialogue really out-of-scope? For example, thethese dialogues do not contain any information that names

chatbot can only deal with hotel reservation, but theor uniquely identiﬁes individual people, and do not containcustomer asks today’s weather. Another example isoﬀensive content.

that the customer becomes angry because the chatbot

Bcannot understand his/her intention.

Details of Experiments

For naturalness, we gave annotators the following in-

We have reported key conﬁgurations in Section 4.1. In

struction.

this section, we report more details of experiments to re-Does the replaced utterance make the whole dialogue

produce reported results.

strange? Speciﬁcally, if the dialogue remains to beform.2） The instance used for experiments contains one

natural after replacing by the new utterance, then label this dialogue as “natural,” otherwise, label thisGPU (Nvidia T4).

dialogue as “unnatural.”All experiments were conducted on Google Cloud Plat-

For data augmentation, we implement the tree ensemble

All annotators are full-time employees aﬃliated in themodule using XGBoost library.3）Grid search is used forsame team as the authors. They all have high levels of

searching the best hyperparameters for tree ensemble. For

English proﬁciency, and are able to annotate dialogues

STAR dataset, if we use all three types of features, it takes

correctly. Annotation was done in an in-house environ-about 70 minutes for tree ensemble.

ment, and all dialogues are used only for the purpose of

After data augmentation, we train a binary classiﬁer to detect OOS dialogues.

research. After annotation, no ethics issues were reported.

The classiﬁer consistsof a bert-base-uncased model (109 million parameters), and two feed-forward layers (231 thousand parameters). We resort to libraries (e.g., pytorchlightning,4）transformers,5） etc.) to simplify implementation. For2） https://console.cloud.google.com/
3） https://xgboost.ai4） https://www.pytorchlightning.ai5） https://github.com/huggingface/transformers6） https://scikit-learn.org/― 1416 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).