Zero-Shot-BERT-Adapters: a Zero-Shot Pipeline for Unknown Intent Detection
Authors: Daniele Comi
Dimitrios Christofidellis
Pier Francesco Piazza
Matteo Manica
Venue: EMNLP 2023 Findings (2023)
Keywords: Zero Shot
Intent Detection
Emerging Intents
BERT
Adapters
NLP
Multiligual
TL;DR: We propose Zero-Shot-BERT-Adapters, a two-stage method relying on a Transformer architecture fine-tuned with Adapters, for multilingual unknown intent discovery and classification in a zero-shot setting.

Abstract: Intent discovery is a crucial task in natural language processing, and it is increasingly relevant for various of industrial applications.
Identifying novel, unseen intents from user inputs remains one of the biggest challenges in this field.
Herein, we propose Zero-Shot-BERT-Adapters, a two-stage method for multilingual intent discovery relying on a Transformer architecture,
fine-tuned with Adapters. We train the model for Natural Language Inference (NLI) and later perform unknown intent classification in a zero-shot setting for multiple languages. In our evaluation, we first analyze the quality of the model after adaptive fine-tuning on known classes. Secondly, we evaluate its performance in casting intent classification as an NLI task. Lastly, we test the zero-shot performance of the model on unseen classes, showing how Zero-Shot-BERT-Adapters can effectively perform intent discovery by generating semantically similar intents, if not equal, to the ground-truth ones. Our experiments show how Zero-Shot-BERT-Adapters outperforms various baselines in two zero-shot settings: known intent classification and unseen intent discovery. The proposed pipeline holds the potential for broad application in customer care. It enables automated dynamic triage using a lightweight model that can be easily deployed and scaled in various business scenarios, unlike large language models. Zero-Shot-BERT-Adapters represents an innovative multi-language approach for intent discovery, enabling the online generation of novel intents.
A Python package implementing the pipeline and the new datasets we compiled are available at the following link: https://github.com/GT4SD/zero-shot-bert-adapters.

Zero-Sho-BBERT--Aappee:: a Zero-Shot Pipeline for Unknown Intent
Detection
Daniele Comi
Dimitrios Christofidellis
IBM Consulting Italy daniele. comi @ibm. com
IBM Research Europe dicêzurich ibm. com
Pier Francesco Piazza
Matteo Manica
IBM Consulting Italy
IBM Research Europe tte@zurich ibm. com
pier. www.ffranccccocm piazza@it ibm. com
Abstract
In dialogue systems, understanding the actual in- tention behind a conversation initiated by user is fundamental, as well as identifying the user's intent underlying each dialogue utterance (Ni et al., 2021). Recently, there has been a growing interest in such applications, evidenced by the plethora of datasets and competitions established to tackle this problem. (Science, 2022; Casanueva al., 2020; Zhang al., 2021). Unfortunately, most of the available most of the available production systems leverage models trained on a finite set of possi- ble intents, with limited or no possibility to gen- eralize to novel unseen intents (Qi et al., 2020). Such limitations constitute a significant blocker for most applications that do not rely on finite set of predetermined intents. They, instead, require constant re-evaluation based users' inputs and feedback. Additionally, most of the datasets for intent classification are mainly available for the English language, significantly limiting the devel- opment of dialogue systems in other languages. Expanding the set of available intents online is im- perative to build better dialogue systems closely fitting users' needs. Besides better-matching user expectations, this also enables the definition of new common intents that slowly emerge from pool of different users. This is key aspect of building systems that are more resilient over time and can follow trends appropriately. Current super- vised techniques fall short of tackling the challenge above since they cannot usually discover novel in- tents. These models are trained by considering a finite set of classes and cannot generalize to real- world applications (Larson al., 2019). Here, we propose multi-language system that combines de- pendency parsing (Honnibal and Johnson, 2015; Nivre and Nilsson, 2005: ExplosionAI, 2015) extract potential intents from a single utterance and a zero-shot classification approach. We rely on BERT Transformer backbone (Vaswani et al., 2017: Devlin et al., 2018) fine-tuned for NLI (Nat-
Intent discovery is a crucial task in natu- ral language processing, and it is increas- ingly relevant for various of industrial appli- cations. Identifying novel, unseen intents from user inputs remains one of biggest challenges in this field. Herein, pro- pose Prro-Shot-BERT-Adaaaa method for multilingual intent discovery rely- ing on Transformer architecture fine-tuned with Adapters. We train the model for Nat- ural Language Inference (NLI) and later per- form unknown intent classification in zero- shot setting for multiple languages. evaluation, first analyze the quality of the model after adaptive fine-tuning on known classes. Secondly, we evaluate its perfor- mance in casting intent classification an NLI task. Lastly, we test the zero-shot perfor- mance of the model on unseen classes, show- ing how Paro-Shot-BERTS BERT--aa-ss can effec- tively perform intent discovery by generat- ing semantically similar intents, if not equal, to the ground-trude truth ones. Our experiments show how Teee-Shot-BER-Adaapaee forms various baselines in zero-shot tings: known intent classification and unseen intent discovery. The proposed pipeline holds the potential for broad application in customer care. It enables automated dynamic triage using lightweight model that can be easily deployed and scaled in various business sce- narios, unlike large language models. Zero- Mhot-BERT-Adapter represents innovative multi-language approach for intent discovery, enabling the online generation novel intents. A Python package implementing the pipeline and the new datasets we compiled are avail- able the following link: https: //githbb. com/GT4SD/ http://roo--hae wwwrrro-shoo-be---aaaaap
two-stage
our
outper-
1 Introduction
Language Models (LM) and, in general, Natural Language Processing (NLP) methodologies have a pivotal role in modern communication systems.

---

QUERY
INTENT
would like to withdraw money
make- withdraatl
Transter money pre- pre-paid paid
Intent Detection Model
translermmoney
conclude my last payment?
Z-BERT-A pipeline
unknown low-conlidence intent
fee-peyment
Can withdraw money
Figure I: mentoyment scenario, where the current intent detection (classification) model is initially used to examine user input and based the classification outcome problematic cases forwarded --e---hooot---liiee for intent discovery.
ural Language Inference) (Xia al., 2018; Yin et al., 2019) to select the intent that is best fit- ting the utterance in a zero-shot setting. the NLI fine-tuning, we leverage Adapters (Houlsby et al., 2019; Pfeiffer et al., 2020) to reduce mem- ory and time requirements significantly, keeping base model parameters frozen. We evaluated our approach, Beero-ShotBBERT-Aaapee BERT-Adapters, for English, Italian, and German, demonstrating the possibility of its application in both high and low-resource language settings. The design focuses on pro- duction setting where automatic intent discovery is fundamental, c.g., customer care chatbots, to en- sure smooth user interaction. Figure depicts how the
each cluster a mean ACTION-OBJECK pair repre- senting the common emerging intent for that par- ticular cluster. Recently, Large Language Models (LLMs) achieved remarkable zero-shot generaliza- tion (Chowdhery et al., 2022; Sanh et al., 2021; Parikh et al., 2023) capabilities. (Parikh et al., 2023) report benchmarks intent classification using GPT-3 (Chowdhery et al., 2022) and analyze performance both zero and few-shot settings, While these solutions are compelling, they are only an ideal fit for some real-world use cases. Models of these sizes can be expensive to operationalize, especially in on- premise settings with strict hard- ware constraints. While there is no literature on the novel intent discovery and zero-shot classification in a multilingual setting, there has been recent attempt to perform zero-shot intent detection (Liu et al., 2019). The authors study the problem of intent detection using attention- baased models and cover low-resource languages such as Italian. Other approaches leveraging multilingual solutions, such as Nicosia et al., focus on a zero-shot multilingual semantic parsing task. Here, the authors present novel method to produce training data for multi- lingual semantic parser, where machine translation applied to produce new data for a slot-filling task in zero-shot.
pipeline fits in an intent detection system. From now on, for simplic- ity, we will refer Sero-Shot-BERT-Adpppee as Z-BERT-A in Tables and Figures.
2 Related Literature
There have been various efforts aiming at find- ing novel and unseen intents. The most popular approach is two-step method where, initially, binary classifier determines whether utter- ance fits the existing set of intents, and, finally, zero-shot techniques are applied to determine the new intent (Xia et al., 2018: Siddique et al., 2021; Yan et al., 2020). (Liu et al., 2022) proposed an approach leveraging Transformers-basee architec- tures, while most relied on RNN architectures like LSTMs (Xia et al., 2018). For our problem of interest, two notable efforts have attempted ad- dress the issue of unseen intent detection (Vedula al., 2019; Liu al., 2021). (Liu et al., 2021) han- dled novel intent discovery as a clustering problem, proposing an adaptation of K-means. Here, the au- thors leverage dependency parsing to extract from
Background
In intent discovery, we aim at extracting from a sin- gle utterance a set of potentially novel intents and automatically determine the best fitting one for the considered utterance. We can cast intent discov- ery as Natural Language Inference (NLI) prob- lem. We can rely on language model ((,,7())) to predict the entailment between the utterance (x) and set of hypotheses based on the candidate

---

Input e.g. 'Where can i to get some money by making withdrawal?
we leverage a dependency parser (Honnibal and Johnson, 2015; Nivre and Nilsson, 2005) ex- tract set of potential intents by exploiting spe- cific are dependencies between the words in the utterance. In the second stage, we leverage the set of potential intents as candidate classes for the utterance intent classification problem using zero-shot approach (Xian et al., 2018) based on NLI relying on BERT-based model (Devlin al., 2018), as depicted in Figure 2. The model is tuned with Adapters (Houlsby et al., 2019) for the NLI task (BERT-A) and is then prompted with premise- hypothesis pairs for zero-shot classification on the candidate intents, completing the Zero-Shot-BERT- Adapters pipeline. The entire pipeline's 's implemen- tation follows the Hugging Face pipeline API from the transformers library (Wolf al., 2019).
Shot Detection model
Spacy Unknown Intent set gencration
Emboodinn layer
Bull Attention
x12
Transtome--Acppp
Boaey translarmur levell detect con &
FRNN Pooling
Zoro Shot Clazzrler
( y(t)
e.g. "make withdrraa"
architecture. The two stages are shown: first stage on the right where the intent generation algorithm 1 is run ob- tain language-dependdnt intent candidates ((r) which are then used as labels for the pero-shot classification in the second stage.
Figure 2:
Intent generation Defining a set of potential in- tents from an input utterance is crucial for effective intent discovery, especially when dealing with a multilingual context. To provide this set of po- tentially unseen candidates, we choose to exploit the dependency parser from spaCy (Honnibal and Johnson, 2015; Honnibal et al., 2020), considering: for English the model on_core_web_tee wwf.ereebeeeeeeeeee for Ital- ian the model it_ meccorre_nww___ee and, for German the model de _ee_ddp_eews_____ trf (all implemented in spaCy- transffrooss (ExplosionAl, 2019)). We ex- tract pair of words from an input sentence through the dependency parser. We define each pair by searching for specific Arc-Relations (AR) in the dependency tree of the parsed sentence. Unfor- tunately, this approach is language-dependen different languages can require a specific pattern- matching for the ARs.
intents (7(z)) where y is a function used to ex- tract hypotheses from the set of potential intents yr- As previously shown by Xian et al., using NLI models for zero-shot classification represents a practical approach in problems where the set of candidate intents is known. In practice, the classifi- cation problem becomes an inference task where a combination of a premise and a hypothesis are associated with three possible classes: entailment, neutral and contradiction. Yin et al. have shown how this approach allows considering an input hy- pothesis based on an unseen class. It then generates a probability distribution describing the entailment from the input wremise-hypother score correlating the input text to the novel class. While this technique is highly flexible and, in prin- ciple, can handle any association between utterance and candidate intent, determining good candidates based on the analyzed utterance remains a signifi- cant challenge.
pair, hence a
Since intents are usually composed by an action- object pair (Vedula et al., 2019; Liu et al., 2021), exploit this pattern for all languages when look- ing for candidate are relationships. We search for DOBJ, compound, and AMOD are relations in En- glish. We must adjust this in Italian and search for AUX, ADVMOD, and compound are relations. We further adapt the tags to RE, SB, and MO in German. We perform a four-level detection, which means finding the four primary relations that can generate base intent using these relations. Once these relations, or a subset of them, are found, we add for all the languages the pairs composed by (VERB, NOUN) and, for English only, (ADJ, PRONM) with the most out/in going arcs. We re- fer to Appendix for the Tables 11, 12 and 13
4 Approach
Herein, we focus on building a pipeline to han- dle unseen classes at inference time. In this con- text, we need both to generate a set of candidate intents from the considered utterance and clas- sify the provided input against the new set of can- didate intents. We tackle the problem by imple- menting a pipeline in two stages. In the first stage,

---

for a complete definition of the AR and Part-of- Speech (POS) tags considered. We lemmatize the extracted potential intents using NLTK (Loper and Bird, 2002). Lemmatization is applied to verbs and nouns independently. The lemmatized intents rep- resent the labels that our model uses for zero-shot classification. Algorithm details in pseudocode the pipeline for the intent generation with (n) as time complexity, where is the number of depen- dency ares in an utterance.
and the ID-OOS queries are built up based on 27 held-out in-scope intents.
We also explore the effects of pretraining lever- aging NLI adaptation of Banking77 (Yin et al., 2019). To investigate the impact of pretraining on similar data, extended the Banking77 dataset by casting the intent classification task as NLL We consider the input utterance the premise and ex- tract the most relevant word associated with us- ing KeyBERT (Sharma and Li, 2019). The word is then used to generate an entailed hypothesis based on the corresponding synset definition from WordNet via NLTK (Loper and Bird, 2002; Miller, 1995). Exemplar samples of synthetic Banking77 NLI data are reported Table 1. For the hypothe- ses that are not considered entailed, we repeat the procedure for randomly sampled unrelated words. This process enabled us to consider the training split of Banking77-OOS for adaptive fine-tuming of the NLI model component. We call this generated dataset Banking77-00S-NLL.
Zero-shot classification The generated potential intents are the labels for the zero-shot BERT-bbasd classifier implemented using NLI that scores the entailment between the utterance used as a premise and the hypothesis based on the intent. Given an input utterance, we select the intent related to the pair with the highest score. We use sentence em- bedding vectors to define the scores (Reimers and Gurevych, 2019).
5 Datasets
To analyze the validity of our approach in multi-lingual setup with lower data availability, we compiled and made available both an Italian and German version the Banking77 dataset. We followed a two-step process involving translation using the Google Translation API (Google, 2023) and manual annotation and correction by human agents to maintain data quality following the proto- col described in (Bellomaria et al., 2019).
We consider two datasets in our analysis: SNLI (Bowman et al., 2015) and Banking77- 00 (Casanueva et al., 2020; Zhang et al., 2021):
The SNLI corpus (Bowman et al., 2015) is collection of 570k human-written English sentence pairs manually labeled as entailment, contradiction, and neutral. It is used for nat- ural language inference (NLD), also known as recognizing textual entailment (RTE). The dataset comes with split: 550152 samples for training. 10000 samples for validation, and 10000 samples for testing. Each sample is composed of a premise, a hypothesis, and a corresponding label indicating whether the premise and the hypothesis represent an en- tailment. The label can be set to one of the following: entailment, contradiction, or neu- tral.
6 Training
We fine-tune two versions of BERT-A (BERT- based transformer with Adapters). The first ver- sion is trained for NLI on the SNLI dataset using the original split for training, validation, and testing (Bowman al., 2015). The second version also considers the previously introduced Banking77-00S-NNL which keeps the split of in- scope and the out-of-scope ratio of 50 27 out of 77 total intents. Depending on the language considered we use different BERT pretrained mod- els: for English we use bert-base-uncased (Devlin et al., 2018); for Italian we use pert-base-italian- xxl-uncased (dbmdz, 2020); for German we use bert-base-gee caase-german-casee (Deepset, 2019). During training, we keep the BERT backbone frozen and only update the added Adapter layers minimize training time and memory footprint. By freezing all the original layers and letting the model be trained only on the adaptive layers, we end up with
Sanking77-OOO (Casanueva et al., 2020; Zhang et al., 2021) is an intent classification dataset composed of online banking queries annotated with their corresponding intents. It provides very fine-grained set of in- tents in the banking domain. comprises 13,083 customer service queries labeled with 77 intents. It focuses on fine- -raiied single- domain intent detection. Of these 77 intents, Banking77-OOSS includes 50 in-scope intents,

---

Table 1: Sample transformations of Banking77 dataset instances from intent classification to NLI. SD is the Synset definition extracted from WordNet, which use generate an augmented NLI dataset starting point for any Text Classification dataset. A similar set of samples for the Italian version is available in the Appendix.
Utterance
Extracted key-phrase
Wordnet Synset definition (SD)
Generated hypothesis
the activity of providing for or maintaining by supplying with necessities the act of delivering distributing something a sum of money paid
where do you support?
support
this text is about SD
card delivery?
delivery
this text about SD this text about SD
last payment?
payment
claim discharged
396'066 trainable parameters. The final model then has almost 111 million parameters (110 million pa- rameters from the BERT-base-modd and 896' '66 parameters added through the adapters, all with a default size of 32 bit floating point). All training runs relied on the AdamW (Loshchilov and Hutter, 2017) optimizer with a learning rate set to 2- 10 and warm-up scheduler. The models have been fine-tuned for a total of 6 epochs using early stop- ping. Our computing infrastructure for training the models run on Nvidía Tesla T4 GPUs using Py- Torch and Python 3.7. this configuration, each training epoch took approximately one minute to run. During the testing phase each run took around 36 seconds, including the overhead due to metrics computation.
for the intent generation. As a hypothesis for the NL.--aaed classification, we used the phrase: "This example is < label >". Recently, large LMs have demonstrated remarkable zero-shot capabilities in a plethora of tasks. Thus, we included four LLMs as baseline models, namely TO (Sanh et al., 2021), GPT-J (Wang and Komatsuzaki, 2021), Dolly (lab, 2023) and GPT-3.5 (Brown al., 2020) using text- davinci-003 via OpenAl API (base model behind InstrucIGPT and GPT-3.5 (Ouyang al., 2022; OpenAI, 2022)). In these models, the template prompt defines the task interest. We examined whether such L.MSs can serve the end-to-end in- tent extraction (intent generation and classification). We investigated them both in a completely unsu- pervised, i.c., zero-shot settingr and using intents generated with our dependency parsing method. In the former case, the given input is just the utterance of interest, while in the latter case, the provided in- put includes the utterance and the possible intents. In both cases, the generated output is considered as the extracted intent. Table 7 (in Appendix) reports all prompts in different languages used to query the LLMs. We use prompts where the models generate the intent without providing set of possible op- tions (prompts 1 and 2). Prompts that contain the candidate intents extracted using the first stage of our pipeline (prompts 3 and 4). Regarding GPT-3.5, we used two different prompts (prompts 5 and 6) reported in the results table. These are the two best performing prompts and they were built following the suggested examples from OpenAL. In this set- ting, the intents generated can't 't match perfectly the held-out ones. For this reason, chose mea- sure the performance using semantic similarity metric, based on the cosine-similarity between the sentence embeddings of the ground-truth intents and the generated ones (Vedula et al., 2019). To set a decision boundary, we relied on a threshold based on the distributional properties of the com- puted similarities. The threshold / is defined in
7 Evaluation
First, analyzed the performance of the BERT-A component and evaluated its results on NLI task using accuracy, precision, and recall, Afterward, we compared its result on the zero-shot classifica- tion task with other available models on the same Banking77 split using accuracy. In this initial eval- uation, the intents were known. The baselines considered in this setting were BARTO (Lin et al., 2022), a multitask model with 406 million parame- ters based on Bart-large (Lewis al., 2019) based on prompt training; and two flavors of Zero-Shot DDN (ZS- (ZS-DNN) (Kumar et al., 2017) with both encoders Universal Sentence Encoder (USE) (Cer et al., 2018) and SBERT (Reimers and Gurevych, 2019).
In the unknown intent case, we compared the pipeline against set of zero-shot baselines based various pre-trained transformers. As basclines, we included bart-large-mnli (Yin al., 2019) it achieves good performance in zero-shot sequence classification. We used this model as an alterna- tive to our classification method, but in this case, we maintained our dependency parsing strategy

---

Table 2: Accuracy of the BERT-A module on each corresponding test set for the NLI task. It shows the perfor- mance differences when using SNLI, Banking77 augmented datasets in three different languages. These the results of five different evaluation runs, on the described conditions in the training section, together with validation performances. We are indicating with an -DE suffix results for Italian and German respectively. In those cases BERT-A indicates the Adapters fine-tuned version of the respective base models.
Model
Dataset
Test Accuracy
Precision
Recall
Fl-score
Validation Accuracy
BERT-A
Banking77-OOS --IIIT
0.841
0,853
0.852
0,847
0,863
BERT-A
Banking77-00s-NLAN
0.832
0.844
0.841
0.838
0.851
BERT-A
SNLI
0.866
0.871
0.869
0,870
0.902
BERT-A
Banking77-0OS-NLI
0.882
0.894
0.894
0.890
0.894
Table 3: Accuracy of the BERT-A module for the zero- shot intent classification task, on Banking77-OOS com- pared with the considered baselines. This test is to provide a view of how 7ero-Shoo-BEERT-Adaale
and recall achieved confirm the quality of the pre- trained model which achieves a 1.6% in accuracy and 2% in Fl-score then BERT-A fine-tuned with SNLL We also report the results the Italian and German fine-tuned models to highlight the impact of the different languages.
and the other baseline models are performing on the task of zero-shot classification task the test set, without hav- ing trained the model on the classes present in the test set.
Table 3 shows how the Z-BERT-A component improves results over most baselines in terms of accuracy in the known intent scenario where we present all available classes to the model as options in a zero-shot setting. Remarkably, the BERT-A version fine-tuned on Banking77-0OS--L. outper- forms all the considered baselines. Also, when considering at the German and Italian version of BERT-A we obseve comparable performance with the English counterpart, showing again how our approach can adapt different scenarios.
Model BARTO
Accuracy
0.147
ZS-DNN-USE
0.156
BERT-A (SNLI)
0.204
7S-DNN-SBERT
0.390
BERT-A (Banking7-00SS-NLLLDE))
0.396
BERT-A (Banking77-00--NLIITT]
0.404
BERT-A (Banking77-0OS-NLI)
0.407
Equation 1.
Finally, report Zero-Shot Mellot-BER--AA MEee--EERTT--eete Adapters the unknown intent discovery task. Tables 4 and 5 show the performance of BERT-A fine-tuned on SNLI and Banking77-00SS-NI in comparison with a selection of zero-shot baselines for intent discovery. For the baselines, we show the perfor- (1b) mance on zero-shot classification given potential intents from our approach (prompts 3, 4, and 6) and on new intents discovery solely using the in- put utterance (prompts 1, 2, and 5). Table 7 in Appendix A reports all prompts considered. Both flavors of Zero-Shot- Sero--hho-BERRT--A BEET--ERRRT--T- -dRT-Adapters outperform the considered baselines by a consistent margin, by 2.7%, against the best baseline of TO. In the multi- lingual setting, Table 5 and 6, we did not consider GPT-J and Dolly as a baselines because they par- tially or completely lack support for Italian and Ger- man. Our pipeline applied for the Italian language achieves remarkable results. It outperforms the best baseline, TO, by 13% in cosine similarity score. also shows how L.MMs performance exhibit a bias towards low-resource languages. For German, we have observed again better perfomance com-
0.5
if p S 0.5
1
(1a)
otherwise
n
T; 3/1
/1
| ||aill
1=1
0²
n
|
Fi
Yi
||vell
14
(1c)
where is an arbitrary parameter to control the variance impact, which we set to 0.5 in our study.
The evaluation of the pipeline was repeated five times to evaluate the stability of the results.
8 Results
Firstly, we report the performance of BERT-A on the NLI task, see Table Here, BERT-A obtained the best results when being fine-tuned on Banking77-00S-NL. The accuracy, precision,

---

Table 4: Cosine similarity between ground-truth and generated intents for the full Zero-Shot-BEil Schot-BERT-AR Bdda--dtpteesrss pipeline and the other baseline models the ID-OOS test set of Banking77-OOS. We hereby include the final desired results also range in which the Cosine similarity is found on the multiple runs.
Model
Prompt
Cosine similarity
f
GPT-J
en-1
0.04
0.5
GPT-J
en-2
0.04
0.5
GPT-J
en-3
0.117
0.5
GPT-J
en-4
0.098
0.5
TO
en-1
0.148
0.5
en-2
0.189
0.5
TO
en-3
0.465
0.5
TO
en-4
0.446
0.5
Dolly
en-1
0.316
0.5
Dolly
cn-2
0.351
0.5
Dolly
en-3
0.384
0.5
Dolly
en-4
0.333
0.5
GPT-3.5
en-5
0.326
0.5
GPT-3.5
en-6
0.421
0.5
bart-large-mnli
0.436
0.5
Z-BERT-A (SNLI)
0.478 0.003
0.5
Z-BERT-A (Banking77-00S-NNL
0.492 + 0.004
0.546 + 0.011
Table 5: Cosine similarity between ground- truth and generated intents for the full Saro-Shoo-BERT-Adappee pipeline baseline models the ID-OOS set of Italian Banking77-OOS. We hereby include on the final
Table 6: Cosine similarity between ground- truth and generated intents for the full Zero-Shoo-BERT-Adapprr pipeline and other baseline models on the ID-OOS set of German Banking77-OOS.. We hereby include on the final desired results also range in which the Cosine similarity is found on the multiple runs.
the other
desired results also a range in which the Cosine similarity is found on the multiple runs.
Model
Prompt
Cosine similarity
t
Model
Prompt
Cosine similarity
it-l
0.127
0.5
de-1
0.158
TO
it-2
0.145
0.5
TO
de-2
0.192
0.5
TO
it-3
0.337
0.5
TO
de-3
0.321
0.5
TO
it-4
0.302
0.5
TO
de-4
0.318
0.5
GPT-3.5
it-5
0.24
0.5
GPT-3.5
de-5
0.05
0.5
GPT-3.5
it-6
0.266
0.5
GPT-3.5
de-6
0.17
Z-BERT-A
0.467 0.002
0.5
Z-BERT-A
0.36 + 0.007
0.5
pared to the baselines, even though, in this com- parison, TO is competitive with method. It is interesting observe how approaches relying smaller models like outperform LLMs. The results show that combin- ing smaller models with focused simpler classi- cal NLP approaches can still bring better results in intent discovery and detection. The baselines consisting of LLMs, while achieving acceptable results, are not able to get similarity score high Zero-Shot GERT-Adapters This indicates that lightweight approaches that can be easily deployed and scaled are a viable sollution for task-focused scenarios.
tween the generated intents for each of the ground- truth intents in the multilingual setting. It combines the results between the three different language settings. We can see how for various correspond- ing intents, See--SSot-BEET-AARE proportionally similar results in all the languages, As expected, for some classes the results on the English language settings are superior.
can
achieving
9 Conclusions and Future Work
We proposed pipeline for zero-shot prediction of unseen intents from utterances. We performed a two-fold evaluation. First, we showed how our BERT-based model fine-tuned with Adapters on
Figure reports the average cosine similarity be-

---

Multi Lingual Cosine Similarity Score Z-BERT-A
c-nceegg 1as carzs: Baaiovsta
caid pat sorkin cit non fadiinars Surse cane fassprmnnt cie
044
sappling
liiket pata tecrato -in kinges antooe
you by topolee morer
dre up
viual risst Ladagretee)
10cadd acppame coottudens Katinskknpiec
shotaal car
ystepo ng un ore marraren Milaatt wchees
Io sppout at cod accuss Kinte 1ot
bacene salkeutondeeeiin www.eeeeeeeeeereeee.eo.. VErROOu
1opascoes
10cardin inat ceat ingieiee Fapia vancvuch
secert suree huse wet are in tuma dn lool Dkepptitee Getiped
incerto Ubbrestuung Kints
(kaaitm contumn storets Mirggee aiw lrhnk acttutre
24 diocsiie roal laid onden und carta i cm gatts beis rao din vieer Ciaagia candehene 2overt went turia dewnts sirabl:
duit Aermstore Karn
Figure 3: Bar plot showing the average cosine- soain--imillarty for each unseen intent class in the multilingual settings. The corresponding numerical classes for English, Italian and German are also displayed as legend on the right.
NLI can outperform a selection of baselines on the prediction of known intents in zero-shot setting. Secondly, evaluated the full pipeline capabil- ities comparing its performance with the results obtained by prompting LLMs in an unknown in- tent set considering multiple languages. These results prove that our solution represents an ef- fective option to extend intent classification sys- tems to handle unseen intents, key aspect of modern dialogue systems for triage. Moreover, using relatively lightweight base model and re- lying on adaptive fine-tuning, the proposed solu- tion can be deployed in the context of limited re- sources scenarios, e.g., on-premise solutions or small cloud instances. In this way, we don't com- promise on model performance, as we showed that smaller models like the ones powering our pipeline can outperform LLMs in intent discovery. An interesting avenue to explore in the future con- sists in relying on zero-shot learning approaches in the intent generation phase (Liu et al., 2021) without compromising on model size and infer- ence requirements. Zero-Shot-BERT-Adapppe the Italian and German datasets are available at the following link: https: //gi thub. com Com/GTTSD//// zero- wwwtthhtttt-tteeee
pipelines can help mitigate bias by dynamically adapting intents based on the user audience inter- acting with the system. Relying on the new emerg- ing intents from Berr--Sho--EET-AAappte can easily compile a dataset fine-tune an existing intent detection model that carries an inherent bias coming from its training dataset. Moreover, our approach aims to minimize the usage of computing resources by relying on relatively small model that requires minimal resources for training and in- ference. This parameter efficiency is relevant when considering the carbon footprint associated with training and serving of recent LLMs.
one
11 Limitations
The main limitation of this pipeline currently lies in the new intent generation stage where we are using classic dependency parsing to generate po- tential new intents. This is limitation for two main reasons. Firstly, because we are bound by the input utterance when producing an intent, risking to lack in terms of generalization power. Secondly, when multiple languages are involved the result of the dependency parser has to be interpreted in a language-dependent way. As shown in Tables 11 and 12, different languages call for different arc- relationships. This limitation is something that can be addressed by replacing the dependency parsing with a model- zero-shot approach that would be the natural extension of the presented method.
and
10 Ethical considerations
Dialog systems are helpful tool that companies and organizations leverage to respond to inquiries resolve issues of their customers promptly. Un- fortunately, an uncountable number of users inter- act with such tools on various occasions, which inherently poses bias concerns. We believe that us- ing Zero-Shot-B Serro-Shoo-BERT-ACCA BERT-Adapters in intent detection
References
Valentina Bellomaria, Giuseppe Castellucci, Andrea Favalli, and Raniero Romagnoli. 2019. Almawave- slu: A new dataset for slu in italian.

---

Bowman, Samuel R., Angeli, Gabor, Potts, Christo- pher, Manning, and Christopher D. 2015. A large annotated corpus for learning natural language in- ference. In Proceedings of the 2015 Conference on Empirical Methods Natural Language Processing (EMNLP). Association for Computational Linguis- tics.
Anjishnu Kumar, Pavankumar Reddy Muddireddy,
Markus Dreyer, and Bjöm Hoffmeiiter. 2017. Zero- shot learning across heterogeneous overlapping do- mains. In INTERSPEECH, pages 2914-2918.
DataBricks lab. 2023. Free dolly: Introducing the world's first truly open instruction-tume Ilm. patpss//ibne release blog.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, et al. 2020. Language models are few-shot learners. CoRR, abs/2005.14165
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of-scope prediction.
Tñigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic. 2020. Efficient intent detection with dual sentence encoders. CoRR. ahs/2003.04807.
Mike Lewis, Yinhan Liu, Naman Goyal, Mar- jan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019, Bart: Denoising eqquenceto-sequenee pre-training for natural language generation, translation, and comprehension.
Daniel Cer. Yinfei Yang. Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespeddss Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, al. 2022. Palm: Scaling language modeling with pathways.
Bill Yuchen Lin. Kangmin Tan, Chris Miller, Beiwen Tian, and Xiang Ren. 2022. Unsupervised cross- task generalization via retrieval augmentation.
Bavarian State Library dbmdz. 2020. Italian xxl electra. nttps://ge thgth com/dbmdz/berrt..
Han Liu, Siyang Zhao, Xiaotong Zhang. Feng Zhang. Junjie Sun, Hong Yu, and Xianchao Zhang. 2022. A simple meta-learning paradigm for zero-shot in- tent classification with mixture attention mechanism. arXiv preprint 22X:::2006001799
Deepset. 2019. Bert-base-gerran--caeed. https:// nuggingface..oobbrtt-boee
cw/berttbasseeeeeeeeee baase--grman-coee
Jacob Devlin, Ming-Wei Chang. Kenton Lec, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint 1rXiv:1810.04805,
Pengfei Liu, Youzhang Ning. King Keung Wu, Kun Li, and Helen Meng. 2021. Open in- tent discovery through unsupervised semantic clus- tering and dependency parsing. arXiv preprint arXiv:2104.1211 121.4.
ExplosionAl. 2015, spacy. https://github.cr com/ explosion/spaCy.
Zihan Liu, Genta Indra Winata, Zhaojiang Lin, Peng Xu, and Pascale Fung. 2019. Attention-infommed mixed- laxeu--anguage training for zero-shot cross-lingual task lask-oriented dialogue systems.
ExplosionAL. 2019. tracy-transformers. https://
github. www/ coon//epllsion//ppee wtapai///spccyttassfrreeeeee transformers.
Google. 2023. Google translate api.
Edward Loper and Steven Bird. 2002. Nltk: The natu- ral language toolkit.
Matthew Honnibal and Mark Johnson. 2015. An im- proved non-monotonic transition system for depen- deney parsing. In Proceedings of the 2015 Con- ference Empirical Methods in Natural Language Processing, pages 1373-1378, Lisbon, Portugal. As- sociation for Computational Linguistics.
Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization.
George A Miller. 1995. Wordnet: lexical database for english. Communications of the ACM, 38(11):39- 41.
Matthew Honnibal, Ines Montani, Sofie Van Lan- deghem, and Adriane Boyd. 2020. spaCy: Industrral-strennth Natural Language Processing in Python.
Jinjie Ni, Tom Young, Vlad Pandelea, Fuzhao Xue, Vinay Adiga, and Erik Cambria. 2021. Re- cent advances in deep learning based dialogue systems: A systematic survey. arXiv preprint 145447710..04887.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for ulp. International Conference on Machine Learning, pages 2790-2799. PMLR.
Massimo Nicosia, Zhongdi Qu, and Yasemin Altun. 2021. Translate & fill: Improving zero-shot multi- lingual semantic parsing with synthetic data.

---

Joakim Nivre and Jens Nilsson. 2005. Pseudo- projective dependency parsing. Proceedings of the 43rd Annual Meeting of the Association for Com- putational Linguisties (ACL 005), pages 99-106, Ann Arbor, Michigan. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information process- ing systems, 30.
Nikhita Vedula, Nedim Lipka, Pranav Maneriker, and Srinivasan Parthasarathy. 2019. Towards open intent discovery for conversational text.
OpenAT. 2022. Introducing chaigpt. https://openall honn/blogchhatga
Ben Wang and Aran Komatsuzaki. 2021. GPT-J- 6B: A 6 Billion Parameter Autoregressive Lan- guage Model. https://githuu. woa.ub..com/kingflllllz mesh- transformer-
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Pe- ter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow in- structions with human feedback.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Fun- towicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv: 1eXxv:9100007771
Soham Parikh, Quaizar Vohra, Prashil Tumbade, and Mitul Tiwari. 2023. Exploring zero and few-shot techniques for intent classification. arXiv preprint arXiv:230..07157.
Congying Xia, Chenwei Zhang, Xiaohui Yan. Yi Chang, and Philip Yu. 2018. Zero-shot user intent detection via capsule neural networks. arXiv preprini urXiv:I809.000
Jonas Pfeiffer, Andreas Rücklé Clifton Poth, Aish- warya Kamath, Ivan Vulié, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020. Adapterhub: framework for adapting transform- ers. In Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing: System Demonstrations, pages 46-54.
Yonggin Xian, Christoph H Lampert, Bernt Schiele, and 7eynep Akata. 2018. Zero-shot learning- comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, 41(9):225122265
Guangfeng Yan, Lu Fan, Qimai Li, Han Liu, Xiaotong Zhang, Xiao-Ming Wu, and Albert YS Lam. 2020. Unknown intent detection using gaussian mixture model with an application to zero-shot intent classifi- cation. In P Proceedings of the 58th annual meeting of the association for computational linguistics, pages 1050-1060.
Haode Qi, Lin Pan, Atin Sood, Abhishek Shah, Ladislav Kunc, Mo Yu, and Saloni Potdar. 2020. Benchmarking commercial intent detection services with practice- -neive evaluations.
Nils Reimers and Tryna Gurevych. 2019. Sentence- bert: Sentence embeddings using siamese bert- networks. arXiv preprini urXiv: 10084.
Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. arXiv preprint 19X.:::990900111
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Amaad Stiegler, Teven Le Seao, Arun Raja, et al. 2021. Multitask prompted training en- ables zero-shot task generalization. arXiv preprint arXiv:2110.08207
Jianguo Zhang. Kazuma Hashimoto, Yao Wan, Ye Liu, Caiming Xiong, and Philip S. Yu. 2021. Are pre- trained transformers robust intent classification? A missing ingredient in evaluation of oft-of-scope intent defection. CoRR, abs/2106.046 06.06464.
Amazon Science. 2022. Intent induction from conversations for task-oriented dialogue. https://githhbb com/ comm/amazon-esseee researcch 1etel1-traakk--inee---inn
Prafull Sharma and Yingho Li. 2019. Self-supervised contextual keyword and keyphrase retrieval with self-labelling.
AB Siddique. Fuad Jamour, Luxun Xu, and Vagelis Hristidis. 2021. Generalized zero-shot intent detec- tion via commonsense knowledge. In Proceedings of the 44th International ACM SIGIR Conference Research and Development in Information Retrieval, pages 1925-1929.

---

A Intent generation and results
Table 11: Are-Relations (AR) and Part-of-Speech (POS) tags used in the intent generation phase for the English language.
Algorithm Intent generation algorithm
Input: utterance z, language I
AR/POS tag
Description
verh, covering all verhs except for auxiliary verb
Output: set of potential intents I
VERB
noun, corresponds to all cases of singular or plaral nouns adjective, covering also relative and superiative adjectives pronoun. kinds of pronouns direct object, noun phrase which is (accusative) object verb adjectival modifier serves modify meaning of the noun phrase noun compounds
NOUN
1: let deps sereedeney_parseree
ADJ
2: let ARs ('en': I'DOBJ', 'AMOD', 'com-
PRONM
pound'l. 'it': r'AUX,, 'ADVMOD', 'com-
DOBJ
pound"]]
3: for are in deps do
AMOD
4: if arcc"label") in ARs[[] then
compound
5: Let start = beginning word of the are rela-
tionship
Table 12: Are-Relations (AR) and Part-of-Speech (POS) tags used in the intent generation phase for the Italian language.
6: Let end = ending word of the are relation-
ship
7: Lappend (start" word'] + end[" and(wword"))
8: end if
AR/POS tag
Description
verb, covering all verbs except for auxiliary verb noun, correspondss to all cases of singular or plural nouns auxiliary verbs
9: end for
10: Let best_w = set of words with most
NOUN
in/outgoing arcs
AUX
11: for (word_1, word_2) in best_w as (VERB, NOUN) or (ADJ, PRONM) do 12: Lappend(word + word_2)
adjectival medifier that serves to modify the meaning the noun phruse
ADVMOD
compound
noun compounds
13: end for
14: I = (lemmatize(i) for i in }
Table 13: Are-Relations (AR) and Part-of-Speech (POS) tags used the intent generation phase the German language.
return
Tables 11, 12 and 13 report the POS tags for the arc-relationship chosen for the different language settings. These tags are used for choosing the best zero-shot class candidates through the dependency parser obtained from the input utterance.
fag
Description
verb, covering all except for auxiliary verb corresponds to all cases of singular or plaral nouns auxiliary verbs
VERB
NOUN
AUX
PROPN
A proper noun used in sentence for a object. noun compounds
Algorithm describes in detail our intent genera- tion algorithms. It relies on the Spacy Transformer Dependency Parser on which we apply the previ- ous explained AR and POS language-depende relations. The algorithm takes in account informa- tion regarding this AR language-dependen rela- tions. It is then able find various potential intents from an input utterance. We also show the testing prompts table 7, where each prompt used for testing is shown. For both Italian and English the prompts are the same, they are just correctly translated and iinuaage-ddapted For the various baselines we are using four different prompts, two with potential intents in input, obtained through the pipeline of Berr-SSoo-BEET--Adappe
compound
As well as for the original Banking77 dataset we present in Table 9 some Italian exemplar samples and in Table 10 some German examplar samples. These are synthetic samples for the italian and german Banking77 NLI adapted dataset. Concluding, to appreciate the quality of the generated intents, in Table 14 we report some examples of unseen ground-truth intents and the corresponding Zero-Shot SSott-EER--AAA BEe--BERT--aaaa Adaterss predictions. It is interesting to see how intents obtained through the pipeline can be sometimes equal other than just semantically equal. The italian and German intents are also listed. It's interesting to see how changing the language can lead to intents, while semantically similar, different from other languages.
and two with only the utterance. For ChatGPT's GPT-3.5 model we used different specific prompt obtained from OpenAI suggested prompt examples.

---

Table 7: Prompts used for evaluating LLMs.
Prompt nome
Prompt text
Censidering this [ulterance]. What
that deseriles
Considering this utterance: lutterance]. What is the intent that best descrihes it expressed as phreee of one
20:2
words?
Given the utterance: [utterance] What is the best fitting intent, if any.
<0-3
the following: [potential intentsjp
01-4
[uttorincc]] Choose the most snitable intent bised the above ufterance. Opticess: [potential intents)
09-5
Extract the intest based on this utterance: (utterance)
following intents: [potental intents]. Choose best fiting. I'ayy. for otterance: [utterance].
16-1
Considerndo quesia frase: [ultcrance) Quale C l'ttenno che la descrive meglis?
Considerando questa [utterance]. Qualc I'intesto che descrive meglio
R-2
expresso in una dus parole?
frae: [utierancc] Quale migliore
corrispomenti.
8-3
se ce ne è uno, tra seguenti: pposential insents
8-4
[umeranee] Socgli il miglioe intenis comispondentc bastto sull frase precedentc. Optioni: [potential intents]
8:5
Estrane l'intento in base a quesia frasc: [utterance].
8-6
"Daii segaenti intenti: [potential inteuts]. Sceyliere più aiblio, preseste,
frase: [ntlerance)
do-1
Retrachsen wir diesen Sate: [stternne]] Welche Abwicht beschreilt es bosken?
Betrachten wir diesen Sate: [uticraned]] Welche Absicht beschreili sic am bosten,
4e-2
ausgedrlickt
zwel Wonen?
Angesichts der Phrnse [unerance]. Weiche der foigendes Absichten passt am besten zu den folgenden: (potential intents]?
de-3
de4
[utteranne] Walhka Sie die am beston pussende Absich tusierend auf dam vorherigen Satz. Optionen: [potonial intents] Extrahicren
de-
Absichi anhand dieses "Angesichts der folgenden Absichten: potenniill intenss). Wülhlen Sie, falls vorhundenn den um besten geeigneten Saiz flir diesen Satz aus: [unerance]]"
[uticrane:]]
de-6
Table 8: A few examples of the incoming utterances and the respective predicted intents as predicted by the presented method and the the two main baselines. This is to facilitate the manual comparison of the generated intents of our method versus the baselines.
peili ge
The
3:3m
meoenc dre Eopfennc on la MincIn
niciiia cara
Cons par orfone cuss
abod aariioo pincs
perna phaa
cila
Liara & quota Jolioia & mric aact le
rean 2' borking be
14 lae Wicher
Winken l6 de lile.
18 kickim catt
nakeve, Kark Kare lide
De Alralo clnas Satin borkl Lork Eaiin de
Table 9: Exemplar transformations of Italian Banking77 dataset instances from intent classification to NLL. SD is the Synset definition extracted from WordNet which will he used to generate an augmented dataset from NIII having as a starting point any Text Classification dataset.
Utterance
Extracted key-phrase
Wordnet Synset definition (SD)
Generated hypothesis
l'attività di mantenere fornire
dove c'è supporto?
supporto
questo testo riguarda SD
attraverso la fornitura di necessità
I' l'atto di consegnare distribuire qualcosa
consegna della carta?
consegna
questo testo riguarda SD
una somma di denaro pagato o una pretesa assolta
ultimo pagamento?
pagamento
questo testo riguarda SD
Table 10: Exemplar transformations of German Banking77 dataset instances from intent classification to NLI. SD is the Synset definition extracted from WordNet which will be used generate an augmented dataset from NLI having as a starting point any Text Classification dataset.
Utterance
Extracted key-phrase
Wordnel Synsel definition (SD)
Generated hypothesis
Tätigkeit des Unterhaltens oder Bereitstellens durch Versorgung mit dem Nöligsten der Akt der Lieferung oder Verteilung von etwas
Wo gibt es Unterstiizung?
Untersttizzug
In diesem Text geht es um SD
Kartenversand?
Lieferung
In diesem Text geht es um SD
ein gezahlter Geldbetrag oder eine belriedigte Forderung
letzie Zahlung?
Zahlung
In diesem Text geht es um SD

---

Table 14: Five samples new intent discovery through Zcto-Sho--EERT-Adaptta pipeline for English, Italian and German languages, full list available http://hhhhhccccomm////////oooo0 /tterrttaaaatteeroooo/maaaaaaaia resul trupttss.ddscc csv
Ground-truth intent
Z-BERT-A-EN
Z-BERT-A-IT
Z-BERT-A-DE
exchange-rate card-delivery-estimad
exchange-rate
cambio-soldi
geld wechsee
delivery-time
consegna-data
pünktticc-lieferm
lost-or-stolen-care lost-card
carta-sparita
verlorene-karte
getting-virtualcardd virtual-card
carta-virtuale sbloccare-pin
virtuelle-karte
pin-blocked
pin-block
entsperr-pin
B Use-case for new emerging intents monitoring on real data
Here we show practical use-case in which Zero- Shot-BERT-Adapter is successfully applied. From series of users' telco assistance utterances in which its intent has been recognized, we apply Sethot-BRTTAdaptee and we extract over period of time the various emerging intents. These intents are then displayed this dashboard we developed by applying also Semantic Similarity versus existing intents' categories in order to get better glimpse of actual new emerging intents. We also compute various useful statistical information together with graph showing Hierarchical clus- tering using the Agglogmerative Clustering algo- rithm of the intents based on the semantic similarity score as clustering distance. The approach used in order to get the various intents grouped together is to first cluster similar extracted intents based on the cosine similarity of their embeddings, similarly to what we presented in the described work. Then assign each cluster intent name based on the most common occurrences of action and intent sep- arately, taken from each cluster of intents. This helps normalize the same intent expressed thorugh a a set the various possibile variants from differ- ent utterances, identifying a common intent among the ones with similar semantic meaning. This approach is also cited in (Liu et al., 2021).
In the following Figure glimpse of the EID (Emerging Intent Dash- board) and its practical usage, backed by the Zero-Shot- BERT-Adapters pipeline.
show

---

teerinng merer sere
400
FLATORS
chargeoffe
104
+06
Parguder
400
Energing Hare
406
seg.gggaaaa whunsap
Imer sin letty entring one
Inten duarting grech
momere meroodtt rocken
recp reteriav
Detorry
Figure 4: Emerging Intent Dashboard showing emerging intent obtained from Scro-Shot-BERT-Adeee


Reviews:
[{"review_id": "POnuqpw0mh", "reviewer": "Reviewer_cqfr", "summary": "", "questions": "1. How does this method compare with other existing language model adapters.\n2. What are the tunable parameters and what are the frozen parameters in the model?\n3. What is the size of the trainable parameters in the proposed method?\n4. The model has been used in English and Italian, can experiments be added to  one more language to better prove the multilingual ability? \n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the comments and the suggestions to improve our paper. \n\nThe primary focus of this study is centered around the proposed two-step pipeline. The first step involves identifying potential intents within a given utterance, leveraging syntax dependencies. Subsequently, in the second step, an evaluation is conducted to determine which of these suggestions accurately encapsulates the true intent of the utterance. This straightforward yet highly effective pipeline holds dual significance:\n\n(i) It can be readily employed as-is, obviating the need for predefined intent categories. This capability facilitates the extraction of intents from incoming utterances.\n\n(ii) It can be seamlessly integrated into other intent detection pipelines, providing support in identifying novel and unfamiliar intents that emerge from incoming utterances.\n\nThe incorporation of adapters (Q1) serves primarily to reduce the memory footprint of our pipeline and to expedite the training time of the corresponding model. In our research, we have utilized one of the most widely recognized adapter architectures (Pfeiffer  et al., 2020). However, it is important to acknowledge that further refinements in this aspect could be possible.\n\nConsequently, our experimental endeavors have been concentrated exclusively on the intent detection task. This approach has been motivated by our main objective of proposing an unknown intent identification methodology.\n\nRegarding the Q2 and Q3 and as we highlight in the paper we kept the whole BERT model frozen and only update the added Adapter layers to minimize training time and memory footprint. This means that our final model has almost 111 million parameters (110 million parameters BERT + 896'066 parameters adapters) from which only 896066 are updated during training. We will revise the respective part of the paper to illustrate further these details.\n\nTo prove the multilingual ability as it is suggested in Q4, we performed further experiments using a translated version of Banking dataset in German. Aligned, with performance in English and Italian our model outperforms (cosine similarity scores - ZBERTA: 0.36, GPT-3.5 without intents candidates: 0.05, GPT-3.5 with intent candidates: 0.17). We are going to update the respective section of the paper to include these further evaluations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper designed a method as a BERT adapter to handle the zero-shot intent discovery task. The model has been evaluated on two datasets and achieved state-of-the-art performance.", "reasons_to_reject": "The contribution of the paper is not very clear, how does this method compare with other existing language model adapters.\nMore ablation study could be done to prove the effectiveness of components in the model architecture.\n", "questions_to_authors": "1. How does this method compare with other existing language model adapters.\n2. What are the tunable parameters and what are the frozen parameters in the model?\n3. What is the size of the trainable parameters in the proposed method?\n4. The model has been used in English and Italian, can experiments be added to  one more language to better prove the multilingual ability? \n", "paper_topic_and_main_contributions": "This paper proposed a method to do zero-shot intent classification, it can be applied to BERT-based transformer models. The method contains two stages, where for stage-1, the dependency parser is used to get potential intents and in stage-2 the zero-shot classification is performed for final output. Experiments are done on public datasets to verify the effectiveness of the proposed method.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Here are some minor notes that the author may consider:\n1. In line-184, building a pipeline [that is] able to handle unseen classes.\n2. As the proposed method is a two-stage pipeline, it could be better if stage-1 and stage-2 can be clearly illustrated in the pipeline figure.\n"}}, {"review_id": "LzJ6Vg1zuU", "reviewer": "Reviewer_ihi6", "summary": "", "questions": "1. What is the frequency of examples in the dataset where the intent is explicitly mentioned in the sentence? If this is almost all of the cases, then my first reason to reject is not important. If there are a lot of examples without the intent mentioned, this method is fundamentally limited compared to LLMs which can generalize better than this approach (generate an intent without the intent being mentioned explicitly).\n\n2. Are there any baselines that you could compare to for zero-shot intent classification? If so, why didn't you include them in Table 4?\n\n3. What is the test set for Table 4?", "limitations": "", "rating": 2, "confidence": 2, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer. We appreciate the comments and the suggestions to improve our paper.\n\nRegarding Q1, it is indeed worth noting that in all instances within the dataset we employed, the intended intent is explicitly articulated within the provided utterances. Notably, since the dataset comprises real-world utterances concerning bank transactions, it becomes evident that this pattern holds true for standard dialogues of this nature. This inherent characteristic significantly influenced the development of our pipeline. Furthermore, based on the results indeed this fact helps us to perform better than much larger LLMs. We will try to emphasize further on this aspect in the updated version of the paper.\n\nRegarding Q2, we can consider all the prompts that have been used for LLMs which include the possible intents in them as further zero-shot baselines for this task. These prompts are prompt en-3, en-4 and en-6 present in the Table VIII of the Appendix. Based on Table VI and Table VII, our method outperforms all these baselines in terms of cosine similarity for this task. Furthermore, in the newly added from us German translated version of the dataset (see review cqfr) our model is able to outperform these baselines also for the German language. In the updated version, we are going to consolidate all these results to highlight better this comparison. Furthermore, as reviewer raised an interesting point regarding other ways of evaluating the performance of our work. We attach below a few examples of the incoming utterances and the respective predicted intents as predicted by our method and the rest baselines. A completed set of such instances will be added in the appendix of the paper to facilitate the manual comparison of the generated intents of our method versus the baselines. Based on this manual investigation, we observe that our model is generally slightly better in English, while for the other language its performance is significantly better than the baselines. This helps to have concise intent answers while being able to generalize over completely unseen intent classes for each different request.\n\nHere some examples:\n\nEnglish:\n\nUtterance: can you tell me where you get your exchange rates?, Groundtruth: exchange-rates, Zberta: exchange-rate, Openai: The intent of this utterance is to request information., T0: exchange rates? get rates. idioms and expressions\n\nUtterance: can i choose a date for delivery?, Groundtruth: card_delivery_estimate, Zberta: delivery-date Openai: The intent of this utterance is to inquire about the possibility of selecting a delivery date. T0: choose date?..none of these choices\n\nItalian:\n\nUtterance: aiuto! non riesco a trovare la mia carta, Groundtruth: carta_persa_o_rubata, zberta: trovare-carta, Openai: L’intento di questa richiesta è di trovare una carta, T0: non riesco. i’m looking for my card.\n\nUtterance: com’è il tasso di cambio su questa app?, Groundtruth: tassi_di_cambio, Zberta: tasso-cambio openai: L’intento di questas richieste è di richiedere informazioni. T0: è app. tasso di cambio\n\nGerman:\n\nUtterance: Ich muss Ihre Wechselkurse kennen, Groundtruth: Wechselkurse, Zberta: kennen wechselkurse, Openai: Ich verstehe, dass Sie meine Wechselkurse kennen müssen.\n\nUtterance: Wann kann ich mit der Lieferung meiner neuen Karte rechnen? Groundtruth: Kartenlieferung, Zberta: Karte erhalten, Openai: Es hängt von der Liefergeschwindigkeit ab, aber normalerweise dauert es 1\n\nThe test set for Table 4 and all the rest evaluations and tables presented in this work is the provided standard test set from BANKING-77 and SNLI. We are going to mention this clearly in the revised version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Creativity: The authors present a creative pipeline that combines several components to predict new intents in a zero-shot setting.\n\nExperiments: For many experiments, the authors show results for several different methods, comparing to a variety of LLMs.", "reasons_to_reject": "Simplistic approach: The method presented in Algorithm 1 just extracts words from the sentence. If the intent word is not explicitly expressed in the sentence, this method will be incapable of generating the correct intent.\n\nLack of baseline in Table 4: The authors only present various settings for their model. I'm not familiar with this research area, so I have no idea if there are approaches in previously published work that outperform this method that the authors have left out.\n\nMarginal improvement in Table 4: The difference in results for each approach are very small, so the benefit of the proposed method does not seem large.\n\nInterpretability of remaining results: It's hard to compare the performance to the LLMs because they only use cosine distance. It's clear the model outperforms in semantic similarity (according to the semantic encoder models used), but for more trustworthy results, a small sample of human evaluations should be used as well to be sure that this method outperforms the LLMs in the zero-shot setting. Another option would be to modify the LLM experiment such that label F1 scores could be produced (use a verbalizer to map LLM output to intent classes).", "questions_to_authors": "1. What is the frequency of examples in the dataset where the intent is explicitly mentioned in the sentence? If this is almost all of the cases, then my first reason to reject is not important. If there are a lot of examples without the intent mentioned, this method is fundamentally limited compared to LLMs which can generalize better than this approach (generate an intent without the intent being mentioned explicitly).\n\n2. Are there any baselines that you could compare to for zero-shot intent classification? If so, why didn't you include them in Table 4?\n\n3. What is the test set for Table 4?", "paper_topic_and_main_contributions": "The paper presents a technique for zero-shot intent classification. The authors make use of a BERT model finetuned on the NLI task and a dependency parser to discover new intents not seen before.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jYwtGQBcSp", "reviewer": "Reviewer_wrS3", "summary": "", "questions": "In section 4, the intent generation process generates the candidates novel class for intent classification, but I wonder for a set of unseen classes, how to normalize same intention with different utterances, and how to determine the total number of new intent classes?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the comments and for the provided review. Indeed, the use of NLI for zero-shot tasks is quite common, yet we propose to incorporate it in a simple yet effective pipeline which as highlighted also from the reviewer let us outperform much larger LLMs.  \n\nThe questions raised by reviewer regarding the normalization of identical intentions expressed through different utterance, is really interesting. The approach that we followed in the use case that we present in the Appendix is to first cluster similar extracted intents based on the cosine similarity of their embeddings. Then assign to each cluster an intent name based on the most common occurrences of action and intent separately. This is an interesting approach also cited (Liu et al., 2021) in one of the similar works of our paper. We are going to describe this step in more detail in the updated version of the paper even if this is not part of our contribution.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper focus on important tasks of both known intents zero-shot classification and unseen intent discovery, and can leverages dependency parsers to enhance the intent generation process.\n2. Experimental results show the proposed methods are effective in zero-shot intent detection.\n", "reasons_to_reject": "1. This work is better suited as a demo track paper, rather than a regular long paper.\n2. The idea of using NLI to handle zero-shot learning tasks are quite common.\n", "questions_to_authors": "In section 4, the intent generation process generates the candidates novel class for intent classification, but I wonder for a set of unseen classes, how to normalize same intention with different utterances, and how to determine the total number of new intent classes?", "paper_topic_and_main_contributions": "This paper tackles the zero-shot intent detection tasks and proposes a two-stage zero-shot bert adapters (Z-BERT-A), which first leverages a dependency parser to extract a set of potential intents, then uses NLI methods relying on Bert models to classify on the candidate classes. Experimental results show this method can outperform a wide variety of baselines in both known intents zero-shot classification and unseen intent discovery.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a method for zero-shot intent discovery using a BERT adapter, which is evaluated on two datasets and achieves improved performance over compared baselines. However, one reviewer complains that the contribution of the paper is not clearly stated and its effectiveness compared to other language model adapters is unclear. Another reviewer thought more baselines and comparison should be made, while the differences in results between the proposed method and other approaches are marginal. The authors are asked to provide more information on how their method compares to other existing language model adapters, the tunable and frozen parameters in the model, the size of the trainable parameters, and whether there are any baselines that could be compared to for zero-shot intent classification. The work was also considered more suitable for a demo track paper instead of a regular long paper by one reviewer.
Decision: Accept-Findings