Exploring Description-Augmented Dataless Intent Classification (2024)

Abstract: In this work, we introduce several schemes to leverage description-augmented embedding similarity for dataless intent classification using current state-of-the-art (SOTA) text embedding models. We report results of our methods on four commonly used intent classification datasets and compare against previous works of a similar nature. Our work shows promising results for dataless classification scaling to a large number of unseen intents. We show competitive results and significant improvements (+6.12% Avg.) over strong zero-shot baselines, all without training on labelled or task-specific data. Furthermore, we provide qualitative error analysis of the shortfalls of this methodology to help guide future research in this area.

heading: 1 Introduction
text: Task-oriented dialogue systems (TODS) by design, aid the user in accomplishing tasks within specific domains, and can have a wide range of applications from shopping (Yan et al., 2017) to healthcare (Wei et al., 2018; Valizadeh and Parde, 2022). Modular TODS (Wen et al., 2017) will typically contain an intent classification component (Louvan and Magnini, 2020; Chen et al., 2019; Su et al., 2022) used by a dialogue manager to determine the appropriate task the user intends to complete. In recent years, neural-based models using supervised training have reached state-of-the-art on many natural language processing tasks, including intent classification. However, supervised learning methods require human-labelled data for a predefined set of intents, which may be time-consuming and labour-intensive to acquire (Xia et al., 2018), and may have poor scalability if new intents are added, or task definition changed. An early approach to tackle this problem is dataless intent classification (Chang et al., 2008; Song and Roth, 2014) which aimed to leverage the pairwise similarities between
∗Corresponding author
semantic representations of utterances and intent classes to perform classification without reliance on human-labelled data. However, this approach relies heavily on the quality of semantic representations (Chang et al., 2008). In recent years, successful zero-shot intent classification approaches (Liu et al., 2019; Yan et al., 2020; Yin et al., 2019) have received greater attention, whereby learning conducted using labelled examples of a subset of seen intent labels is transferred to unseen intents. However, these methods still require human-labelled data, and tend to bias towards seen intents, with the number of unseen intents also generally much lower than seen intents (Liu et al., 2022; Zhang et al., 2022).
In this work, with the significant recent advancements in the quality of text embedding models (Muennighoff et al., 2023), we explore the potential for dataless intent classification methods using a number of recent state-of-the-art text embedding models. We introduce several approaches for generating intermediate textual representations for intents, most notably using intent label descriptions, and formalise our methodology. We perform extensive evaluation of our methods, including scenarios with large numbers of intents from different domains, using three commonly used intent classification datasets. We summarise our contributions as follows:
• We introduce a new scheme for generating intent descriptions with an aim to minimise reliance on human expert input. • We show that our intent descriptions yield significant improvements over label tokenization through extensive evaluation. • We introduce an approach utilising utterance paraphrasing and masking which yields further improvements and show this is consistent across a range of models. • We aggregate and explore the potential of a multitude of current SOTA text embedding
13
models for dataless classification. • We extensively evaluate our methodology
on four commonly used intent classification datasets and report on the results. • We provide qualitative error analysis aimed at guiding future work.

heading: 2 Related Works

heading: 2.1 Generalized Zero-Shot Learning
text: Zero-shot learning (ZSL) (Yin et al., 2019) aims to leverage learning previously performed on labelled examples from seen tasks to unseen tasks, of which there are no labelled examples available for supervised training. ZSL has seen increasing popularity in the domain of intent classification (Liu et al., 2019; Yan et al., 2020) in recent years, whereby models are trained on a subset of intent labels and evaluated on another disjoint subset of intent labels. In more recent years, the concept of generalized zero-shot learning (GZSL) has seen an increase in prominence in the domain, in which the performance on both seen and unseen classes are considered in tandem (Zhang et al., 2022; Lamanov et al., 2022). Several GZSL approaches learn a label prototype space during training, which is transferred to unseen classes through methods such as interclass relationship modelling (Zhang et al., 2021) and prototype adaptation (Zhang et al., 2022). Approaches such as (Lamanov et al., 2022) encode the utterance and labels in a sentence-pair setup, with template-based lexicalisation of labels used as class prototypes. Other approaches exist that use label prototypes as centroids in Gaussian mixture models trained on seen class utterances (Yan et al., 2020; Liu et al., 2022). An issue that can occur with GZSL is biased towards seen classes (Zhang et al., 2022), which can lead to significantly lower performance on unseen classes. It is also difficult to see the efficacy of transfer to a large number of diverse unseen classes, as the number of unseen classes in evaluation is also typically much smaller than the number of seen classes.

heading: 2.2 Dataless Classification
text: Dataless text classification (Chang et al., 2008) is defined as tackling text classification without prior training on any labelled data. Generally regarded as a precursor to zero-shot text classification, this approach typically leverages sentence representations without any training on labelled data, by comparing the semantic representations between a sentence
and that of the intent classes (Song and Roth, 2014). (Zha and Li, 2019) utilises “seed" words associated with each intent class to further contextualise the intent class representation, as a single word may often be insufficient to encapsulate the meaning of the class (Chen et al., 2015). Some approaches further leverage class hierarchy to augment classification performance (Li et al., 2016; Popov et al., 2019).

heading: 3 Methodology

heading: 3.1 Problem Definition
text: Let C be a set of intents supported by a taskoriented dialogue system, U = ⋃{Uc}c∈C defines the set of all user utterances, Uc = {ui}1≤i≤nc is the set of utterances belonging to intent class c. The model undergoes no task-specific training and is tasked with making an intent prediction ŷi for a previously unseen utterance ui at inference time. We follow the paradigm set by previous works in dataless text classification (Chang et al., 2008; Song and Roth, 2014) to conduct nearest-neighbour classification over the sentence embedding space. For a given utterance ui, an encoder h(·) and a set of class label representations {lc}c∈C , we make a prediction ŷi as follows:
ŷi = argmax c s(h(ui),h(lc)) (1)
where s(u,v) = u · v/||u||2||v||2 is the cosine similarity between two vectors.
In order to conduct nearest-neighbour classification using intent labels, we require an intermediate representation, or prototype, which encapsulates to some degree the meaning of a class (Zha and Li, 2019), from which we can obtain a suitable embedding. A commonly used approach in dataless classification is to use the labels (Chang et al., 2008).

heading: 3.2 Label Tokenization
text: A class prototype is obtained by tokenizing intent labels directly, inserting spaces and replacing character separators, i.e.
AddToPlaylist → Add To Playlist oil_change_how→ Oil Change How
However, this approach depends on the descriptiveness of the original intent labels, which can vary significantly between datasets and tasks. As such, we propose an additional step to produce
intent label descriptions which we hypothesise can (1) better align the semantic representation with the characteristics of the class and (2) provide more consistent performance across datasets or approaches without requiring in-task data, which previous works (Lamanov et al., 2022) have shown could improve performance over purely using tokenized labels.

heading: 3.3 Our Approach

heading: 3.3.1 Intent Description
text: Our objective is to produce a brief description of the intent expressed by the user in a given utterance, while ensuring the process requires minimal expert human effort so as to remain scalable for large numbers of intent classes. Rather than producing a general description of the intent (Gao et al., 2023), we formalise our template for producing intent descriptions with the two following constraints:
Label Preservation The resulting intent description must contain tokens from the original intent label i.e. car_rental → User wants to rent a car, or replace with an appropriate word (lexical cognates, synonyms etc.).
Format Consistency Descriptions should be written in the declarative form, beginning with either "User is [asking|saying]", or "User wants [to]", and aim to introduce minimal extraneous tokens in a similar manner to abstractive summarization (De Raedt et al., 2023). Our approach differs from the template-based approach in (Lamanov et al., 2022) in that we use exclusively the declarative form in writing our descriptions to maintain consistency across intent classes and datasets. Example descriptions can be seen in Table 1, more examples can be found in Appendix I. We examine the robustness of our approach in Section 6.
In our experimentation (Section 4), our intent descriptions added on average 6.6 tokens to the tokenized intent labels (1.9 → 8.5), with 98.3% of descriptions containing at least one of the label tokens in exact form, and 82.7% of all label tokens preserved.

heading: 3.3.2 Utterance Paraphrasing
text: The diversity of user utterances for any given intent can pose a challenge as intents may not be obvious (Mueller et al., 2022). We hypothesise that a format consistency constraint over the user utterance can benefit dataless intent classification
performance. Previous works primarily focused on utterance paraphrasing as a means of data augmentation (Kumar et al., 2019; Jolly et al., 2020; Sahu et al., 2022) or to reduce overfitting (Dopierre et al., 2021). Our approach leverages inference-time paraphrasing to enforce a weaker degree of our intent descriptions’ format consistency constraint on user utterances. Given a paraphraser model p(·) we compute a sentence embedding of the paraphrased utterance p(ui):
Pui = h(p(ui)) (2)
We leverage a 1.6B StableLM model1 (Bellagente et al., 2024) to generate a single paraphrase for each utterance. Our selection was based on said model being the top-performing model under 2B parameters on the Open LLM Leaderboard (Beeching et al., 2023) as of the time of writing. We additionally experimented with 1.6B Zephyr (Tunstall et al., 2023) and 1.3B Phi-1.5 (Li et al., 2023a) models but found no significant difference on our task. Example templates and further details are shown in Appendix A. The mean cosine similarity between the paraphrases and the original utterances across 4 intent classification tasks and 12 embedding models is 0.89± 0.06.

heading: 3.3.3 Label Entity Overlap & Masking
text: We note that sentence embeddings tended to capture the topic and entity information rather than the associated action, which can lead to misclassifications in the event that two or more intent classes share entities (i.e. AddToPlaylist and PlayMusic can both refer to songs as their objects). To tackle this, we introduce a masking
1https://huggingface.co/stabilityai/ stablelm-2-1_6b-chat
Algorithm 1 Utterance Masking Procedure 1: Given user utterance ui = {ui,1, . . . ui,t} 2: Ti← DependencyParser(ui) 3: procedure MASKTREE(T ) 4: n← root(T ) 5: if relation(n) is obj then 6: n← [MASK] 7: DROP children(n) 8: else 9: for ui,j in children(n) do
10: MASKTREE(ui,j) 11: end for 12: end if 13: end procedure
component which given user utterance ui masks spans containing the object of said utterance, identified through dependency parsing2 (de Marneffe and Manning, 2008; Schuster and Manning, 2016), to produce mi. mi is then weighted to form the masking component:
Mui = h(mi)×Overlaps(ui, k)×1masked (3)
where Overlaps(u, k) denotes whether there is likely entity overlap in the top k candidate intents by similarity and 1masked is whether there exists a masked version of the original sentence. We did not find significant differences in performance for k > 3, and thus we use k = 3 for all our experiments.
Masking Algorithm 1 illustrates the masking procedure which identifies and masks object spans in the utterance. We define such object spans as subtrees within the dependency tree in which a parent node has any of {dobj, pobj, ccomp} relations. We note that object relations are not always present in the dependency tree, in such cases masked representations are not used. From our experiments, some degree of masking was performed for 97.29% of utterances from the ATIS dataset, 98.04% of SNIPS, 90.88% of CLINC and 84.24% of MASSIVE. We show an example of this procedure in Appendix B.
Entity Overlap For each intent, we predict a set of entities ec = {ec1 , . . . , eck} from the intent description that may describe the object of said
2We leverage an off-the-shelf dependency parser, en_core_web_trf from Spacy url: https://spacy. io/models/en
class. As such, entities are defined at problem definition and can be modified alongside intent descriptions when they are added/removed. We precompute an overlap matrix Overlap where
Overlap[i, j] = { 1 ei ∩ ej ̸= ∅ 0 otherwise
(4)
At inference time, we compute overlaps for classes with top k embedding similarities for an utterance ui. Given a similarity vector si = {si,1, . . . , si,c}c=|C| of embedding similarities between utterance embedding h(ui) and intent description embeddings h(lc)c∈C , we compute Topk(ui) as the top k classes with similarity scores sorted in descending order. We then compute pairwise overlap for all pairs in Topk(ui) as follows:
Overlaps(ui, k) = ⋃
m,n ∈ Topk(ui),m ̸= n Overlap[m,n] (5)
We note that future work could explore expansion of the definition of relevant entities to each intent class, as the current solution relies on the quality of intent descriptions and only covers the most likely entities across an entire class, a more dynamic inference-time solution that determines overlap based on candidate classes would be desirable.

heading: 3.4 Combined Sentence Representation
text: We formulate the final representation of the user utterance within the embedding space as the sum of the original utterance embedding with the paraphrasing and masking components:
hi = h(ui) + Pui +Mui (6)
ŷi = argmax c s(hi,h(lc)) (7)

heading: 4 Experiments

heading: 4.1 Datasets
text: We evaluate our methods on four commonly used English task-oriented dialogue (TOD) system intent classification datasets, covering a diverse range of number of intents (7-150) and domains (up to 18). (1) ATIS (Hemphill et al., 1990) is an English air-travel information system dataset containing 18 intent classes. For comparison, we follow previous works (Zhang et al., 2022) in filtering out intent classes containing fewer than 5 examples.
(2) SNIPS-NLU (Coucke et al., 2018) contains 7 intent classes, totalling 14,484 utterances. (3) CLINC (Larson et al., 2019) is a dataset for outof-scope intent classification, with 150 intents and 22,500 utterances spanning 10 domains. (4) MASSIVE (FitzGerald et al., 2023) is a multilingual spoken language understanding dataset containing 60 intents across 18 domains, we select the 16,521 instances from the en-US split of the dataset for our experiments. As our method does not involve fine-tuning on task-specific data, we consider entire datasets to consist of unseen data for evaluation3.

heading: 4.2 Models
text: We select 11 models from the Massive Text Embedding Benchmark (MTEB) (Muennighoff et al., 2023) that are in the top 20 at the time of writing4. Our selections are based on the following criteria: (1) the model weights must be released (2) documentation of training methods and experimentation details must be readily available. Additionally, owing to computational limits5, we only consider models up to 3GB in size. Our final selection of 11 models can be largely grouped into 4 families of models: InstructOR (Su et al., 2023), E5 (Wang et al., 2022), GTE (Li et al., 2023b) and BGE (Xiao et al., 2023). More details on selected models are provided in Appendix C.
We report results in Section 5 for all E5, GTE and BGE models using averaged token embeddings as sentence representations. We additionally compare model performances against a commonly used embedding model in OpenAI’s text-embedding-ada-002 (Neelakantan et al., 2022) which we refer to in our tables as ‘Ada-002’. We also investigated the generation of synthetic examples as intent prototypes (Appendix I) but did not find significant improvements over our approach using intent descriptions (Appendix J).
3We make our code and datasets publicly available and can be found at https://github.com/ruoyunlp/ dataless-intent-classification
4November-December 2023. We note our top-performing selected models are still competitive with current topperforming models from MTEB fitting our criteria as of May 2024
5All experiments conducted using a single 9GB GPU

heading: 5 Results

heading: 5.1 Baselines and Terminology
text: We compare the performance of our methods against several unknown intent classification methods previously detailed in Section 2. Here we clarify the terminology used henceforth to refer to these methods in our results. We refer to scores on unseen intent labels reported by (Zhang et al., 2021) as ICR, (Yan et al., 2020) as SEG, (Liu et al., 2022) as ML-SEG, dataless approach trained using original data from (Lamanov et al., 2022) as TIROrig and likewise TIRSyn for training on synthetic data. We refer to the results of the adapted method of (Gidaris and Komodakis, 2018) reported in (Zhang et al., 2022) as CosT and the reported main results as LTA. We refer to the best-performing model of a similar size to our selection from (Gretz et al., 2023) as TTCD.

heading: 5.2 Metrics
text: Following from previous works (Zhang et al., 2022; Lamanov et al., 2022), we report Accuracy and Macro-F1 scores for intent classification on each of the datasets, in addition, we also compute the average of Accuracy and F1 score for direct comparison between our methods similar to (Gritta et al., 2022). We show macro-F1 only for MASSIVE in Table 2 for comparison’s sake as the previous work (Gretz et al., 2023) did not report Accuracy scores. Full results for each of our approaches including Accuracy scores are shown in Table 9.

heading: 5.3 Methods using Tokenized Labels
text: Despite a lack of task-specific fine-tuning, models using tokenized intent labels generally performed comparably to most of the baselines on unseen intents. The best-performing model (BGELarge) outperforms baseline scores for ICR (+9.13 Mean), SEG (+10.21 Mean) and ML-SEG (+3.14 Mean), TIRSyn (+13.60 Mean), TIROrig (+4.55 Mean) and TTCD (+0.31 F1). BGELarge outperforms CosT on all datasets; however, it also significantly underperforms LTA on all 3 datasets (-16.38 ATIS, -7.49 SNIPS-NLU, -1.21 CLINC). We note that this approach appears quite sensitive to the model as indicated by the comparatively high standard deviation (σOvr = 5.65) across models.

heading: 5.4 Methods using Intent Descriptions
text: Our method using intent label descriptions yields a significant improvement over using tokenized la-
bels (Tables 2 and 3), with an average increase per model of +11.24 overall. This supports our hypothesis (1) (Section 3.2) in that the additional contextualisation added through describing the label via a declarative sentence better encapsulates the semantic information represented by a label. We also note from Table 3 that the standard deviation in performance across models is significantly lower when using descriptions (σOvr = 1.98), supporting our hypothesis (2) that descriptions can improve consistency across models and approaches. Our overall best-performing model (BGELarge) also considerably outperforms the strongest baseline on SNIPS-NLU (+2.14 Mean), CLINC (+4.62 Mean) and MASSIVE (+8.66 F1). We do note that all of our approaches in this setup underperform on the ATIS dataset compared to the baseline, with our overall best-performing approach yielding 60.51 vs 60.55; we provide further insight into possible reasons in Section 7 to help guide future research.

heading: 5.5 Methods with Additional Paraphrasing and Masking
text: Our addition of paraphrase and masked utterance embeddings yields further overall score improvements on average of +3.16 over label descriptions and is consistent across different models (Table 3). Our best-performing model (BGELarge) significantly outperforms previous approaches on all 4 datasets (+0.49 ATIS, +5.42 SNIPS-NLU, +7.06 CLINC, +11.54 MASSIVE). Additionally, our approach outperforms previous work on 9 out of 12 selected models.

heading: 6 Ablations
text: Addition of paraphrasing and masking Table 3 illustrates the mean performance across SNIPS, CLINC and MASSIVE datasets for each model different class prototypes. We note the consistent improvement in performance between tokenized intent labels and our approach using declarative intent descriptions (+7.86 Mean), and the further improvements with added paraphrasing and masking (+10.56 Mean). We omit ATIS from this table as it is significantly unbalanced, the impact of which we explore in Section 7, and its results are already included in Table 2.
Combination of techniques Table 4 demonstrates the performance (mean of accuracy and macro-f1) between different combinations of our techniques using a bge-large-en-v1.5 model. We observe that the addition of paraphrasing increases performance by an average of +2.06% compared to methods without, supporting our hypothesis (3) that inference-time paraphrasing can benefit dataless intent classification. We observe that masking increases performance by an average of +1.80% and the addition of masked embedding only when entity overlaps are predicted increases performance by +0.32% on average. We perform further ablations over combinations of techniques using other models in Appendix E and note similar behaviour across different models.
Choice of Descriptions To investigate whether our proposed method is sensitive to the choice of
intent descriptions, we generate paraphrases of our manually produced descriptions with increasing temperature values and sampled 200 combinations of descriptions for each dataset. Table 5 contains the mean and standard deviations of the macrof1 scores for each dataset, we report macro-f1 for this ablations experiment due to the severely unbalanced nature of the ATIS dataset towards a single class flight (accounting for ∼ 74% of the dataset). Further details on description paraphrase generation and sampling along with examples are provided in Appendix F. Methods using only tokenized intent labels are outperformed by our methods using label descriptions (+4.51%), with further improvements from the addition of paraphrasing and masking (+8.00%). The overall scores per dataset are slightly affected by the choice of intent descriptions, with standard deviations between 1- 2% with the exception of the ATIS dataset. Future work could focus on the combination of multiple intent descriptions (via paraphrasing) or description refinement with unsupervised training (Chu et al., 2021; Müller et al., 2022) to further improve robustness to the choice of descriptions.

heading: 7 Analysis and Future Work
text: In-Domain Saturation We visualise the embeddings generated by our best-performing model (BGELarge) on the 4 evaluation datasets using tSNE (van der Maaten and Hinton, 2008), along with the embedding for the intent label description to gain insight into the source of errors in our approach. Figure 1 shows the distribution of embeddings on the ATIS and SNIPS datasets. In the interest of space, visualisations of CLINC and MASSIVE are shown in Appendix G . We observe a poor alignment on the ATIS dataset between the intent label descriptions (Figure 1a) and utterance embeddings corresponding to each class, possibly explaining the poor performance in general on this dataset across models. We note the single-domain nature of the ATIS dataset, with all utterances relating to air-travel/flight; additionally, we note the significantly imbalanced nature of the ATIS dataset (Nan et al., 2021), with ∼ 74% of utterances belonging to the flight class, which is a label that overlaps the domain of the dataset. We hypothesise this may lead to the intent label descriptions being much worse at capturing semantic information distinct to each class. This is supported by analysis of the pairwise embedding similarities of utterances
belonging to the same class vs utterances belonging to different classes (Table 13) where models’ embeddings on the ATIS dataset consistently had lower percentage-difference in embedding similarity between in-class and out-class, implying more difficulty in distinguishing the utterances using solely embeddings. This issue is mitigated to some degree with our addition of paraphrasing and masking, as the number of misclassifications where there are entity overlaps between classes is reduced on average by 19.19%. We see this visually in Figure 1d as the cluster for each class is more distinct compared to 1b. Errors from classes with overlapping entities in SNIPS are reduced by 29.31%.
Error Analysis We perform qualitative analysis of the remaining errors and identify two categories of commonly occurring errors. (1) Description Scope: Our approach utilises a single description for each intent and can work well when an intent
concerns a limited number of topics; however, intents such as meta and small_talk from the CLINC dataset, and qa from the MASSIVE dataset can encompass a significantly broader range of topics than other intents within the same dataset. The impact of topical granularity per intent class and the potential for a hierarchical approach to intent classes in a dataless setting can be the focus of future work in this area. (2) Action Overlap: Our approach mitigates some errors arising from shared entities across intents through masking. Whilst this has shown success in reducing errors of this nature (i.e. between PlayMusic and AddToPlaylist from the SNIPS dataset), it is less successful in events where an action is shared across classes, such as play from the MASSIVE dataset, and SearchCreativeWork and SearchScreeningEvent from the SNIPSNLU dataset. Future work could investigate the potential to decouple the desired action and object
(topical information) in utterance embeddings.
Label Candidate Analysis We observed from our results (Table 2) that our approach, despite outperforming strong baselines on ATIS and MASSIVE datasets, still consistently underperforms compared to the same setup on SNIPS-NLU and CLINC. We therefore investigate the position of the correct label when ranking embedding similarities. Table 6 shows the percentage of examples where the correct label is ranked within the top-k by embedding similarity for k = 1, 3, 5, 10. We note for erroneous predictions, the correct label is within the Top-3 in 67.11% of cases, 81.89% in Top-5 and 90.94% in Top-10. This implies that our approach can be used to identify candidate intents from a larger set of intents, with a high success rate even for small values of k (i.e. 91.01% Top-3).
Analysis Summary Our proposed approach performs well overall against the strong baseline methods in unseen intent classification; however, it struggles in certain instances with overlaps in intents within the same domain. We identified potential areas for future work to pursue in tackling said issues. The results of our experiments have shown intent label descriptions can perform well as intent prototypes in this problem setting, and that the addition of paraphrasing and masking can further improve performance.
Limitations This approach contains a number of limitations: We have identified issues with the descriptiveness of individual labels earlier in this section, and textual labels may not be readily available for certain datasets, though summarisation methods may be effectively applied to a few user utterances to produce such labels. Our evaluation compares against previous works using scores as reported in their respective papers, further work can be done to replicate their experiments to mitigate any potential risk arising from differences in experimental settings. Future work may also investigate
the application of descriptions to tasks outside of intent classification, such as emotion recognition (Rashkin et al., 2019).

heading: 8 Conclusion
text: Dataless classification allows for scaling to a large number of unseen classes without requiring training on labelled, task-specific data. The benefits of such an approach can enhance development of task-oriented dialogue systems in application to data-poor or compute-limited scenarios where supported intents may also change as the system is developed. In this paper, we have explored the potential of current SOTA text embedding models in dataless intent classification settings using three different approaches for representing intent classes and compared our results against strong zero-shot learning baselines. We proposed a method for standardising the generation of intent label descriptions with an aim to minimise the amount of human annotation required to further support scaling to high numbers of intent classes. Our results have shown that description-augmented dataless classification methods can achieve comparable, and sometimes superior performance to zero-shot methods on the task of intent classification.

heading: 9 Acknowledgements
text: We would like to thank Anandha Gopalan for his helpful comments on the paper. Student Ruoyu Hu was funded by UKRI CDT in AI4Health - grant number EP/S023283/1.

heading: A Utterance Paraphrasal
text: Table 7 contains an example template used to generate paraphrases for utterances from the CLINC dataset. Examples used in the template do not appear in the dataset and do not make explicit mentions of classes. We use length_penalty=-1 to encourage shorter outputs, repetition_penalty=1.2 and num_beams=3, we use default values for all other generation parameters.
We perform an additional ablation study over the choice of examples in the paraphrase generation template using 9 different examples across 3 configurations for each of SNIPS and MASSIVE datasets. We select these datasets specifically as we believe they differ sufficiently in number of intents and domains. Across 3 ablation configurations and the original paraphrasing setup, we obtain an overall score (mean of accuracy and macro-f1) of 92.66 ± 0.19% for SNIPS and 65.48 ± 0.18% for MASSIVE. As the standard deviation is low in both instances, we conclude that the choice of examples in the paraphrase generation prompt has little impact on the final performance through our setup.

heading: B Example Masking Procedure
text: Given an user utterance “i want to watch animated movies at Showcase Cinemas”, we first perform dependency parsing to identify utterance objects that can be masked. Figure 2 shows an illustration of the resulting parsed dependency relations. Following the approach outlined in Section 3.3.3, we mask out nodes with any of {dobj, pobj, ccomp} relations, namely “animated movies” and “Showcase Cinemas” to produce the resulting masked representation “i want to watch [MASK] at [MASK]”.

heading: C Details of selected models
text: Basic model specifications are shown in Table 8.
InstructOR (Su et al., 2023) embeds the utterance with a task description, allowing for taskspecific conditioning at inference time, with good performance on unseen domains. Trained on 330 datasets using a contrastive learning objective (Ni et al., 2022). This family of models is initialised from GTR (Ni et al., 2022) models, which are inturn initialised from T5 (Raffel et al., 2020) models.
E5 (Wang et al., 2022) performs unsupervised pretraining on the model on ∼270M text pairs using an InfoNCE (van den Oord et al., 2019) objective with other utterances within the batch acting as negative examples, followed by supervised fine-tuning on 3 datasets. We select the Base and Large variants, initialised from bert-base-uncased and bert-large-uncased-whole-word-masking respectively.
GTE (Li et al., 2023b) pretrains the model on ∼800M text pairs and fine-tunes using 33 datasets. The contrastive learning objective used in this work considers, for each query-document pair (qi, di) in a batch, the pairwise relation to the remaining examples {(qj , dj)}j ̸=i. The embedding similarities s(qi, dj), s(qi, qj), s(di, dj) are added to the partition function, where s(q, d) is the cosine similarity between two embeddings.
BGE The work (Xiao et al., 2023) initialised from BERT (Devlin et al., 2019) models and trained using RetroMAE (Xiao et al., 2022) whereby both the input sentence and sentence embeddings in an autoencoder setup are randomly masked during MLM training. The authors use [CLS] token embeddings as the sentence representation. Our experimentation showed a slight improvement when using averaged token embeddings (Mean performance +0.82% Tokenized-labels, +1.06% Classdescription).

heading: D Full Results
text: See Table 9 for individual accuracy and macro-f1 scores by task and model.

heading: E Further Ablations
text: We conduct further ablation studies using bge-small-en-v1.5 (Table 10) and gte-large (Table 11) models to verify the findings of our main ablation study conducted on bge-large-en-v1.5 (Table 4). We note that similar trends are observed with the different models, in that our proposed setup utilising a combination of the original utterance embedding with paraphrase embedding and masked utterance embedding using entity overlaps produced consistently higher scores.

heading: F Description Paraphrasing
text: To produce paraphrases of intent descriptions, we leverage a stablelm-2-1_6b-chat model in a similar setup to our inference-time utterance paraphrasal. We increase temperature value from 0.5 to 4.1 in increments of 0.2, producing a paraphrase for each value. We then filter the generated set of descriptions for duplicates and enforce our Label Preservation and Format Consistency constraints, resulting in an average of 3.94 paraphrases per intent in addition to the original manually produced intent description. Each paraphrase has an average Levenshtein distance of 4.61 to the manual intent description. We replace half of all intent descriptions for each dataset with randomly sampled paraphrases, we produce 200 such combinations and repeat our experiments. Table 12 shows examples of paraphrased intent deescriptions for each dataset.

heading: G t-SNE Visualisation
text: Due to the challenge to readability posed by the large number of intents in the CLINC dataset, instead sample the 15 top-performing (100% accuracy) and lowest-performing (24.47% accuracy) intent classes for illustration, with the results shown in Figures 1c and 1d respectively.

heading: H Embedding Similarities Analysis
text: We perform additional analysis on the mean embedding similarity of sentences within the same intent class (in-class) and of different intents (outclass). For a set of intent classes C and utterances U , we calculate the mean in-class similarity sin and out-class similarity sout as
sin = 1 |C| ∑
c∈C
∑
ui∈Uc
∑
uj∈Uc\{ui}
s(h(ui),h(uj))
nc(nc − 1)
sout = 1 |C| ∑
c∈C
∑
ui∈Uc
∑
uj∈Uc′
s(h(ui),h(uj))
ncnc′
where Uc and Uc′ denotes the set of utterances belonging to class c and all classes other than c′ respectively, nc is the number of utterances in set Uc. The mean in-class and out-class similarity scores are shown per dataset (Table 13). From a basic correlation analysis of the mean embedding similarity against a number of metrics, we note for model
performance on the MTEB benchmark there exists a strong positive correlation to the difference ∆s between in-class and out-class examples (Pearson r = 0.72, p < 0.01) as well as %∆s (Pearson r = 0.73, p < 0.01), and there exists a strong negative correlation to the mean out-class similarity µsout (Pearson r = −0.72, p < 0.01).

heading: I Synthetic Examples
text: We compare additionally against synthetic utterance generated for each intent class. We leverage gpt-3.5-turbo (OpenAI, 2023) for this purpose, by including the tokenized intent labels and label description within the prompt to generate a set S of questions or commands fitting said intent i.e. “Given a category tokenized_intent and the description description, Please generate n different example sentences of users asking questions or making commands that fit the given category.". At inference time, we sample k synthetic
examples for c classes and make prediction ŷi as follows:
ŷi = argmax c
∑k m s(h(ui),h(s c m))
k
where scm denotes the m th example utterance belonging to intent class c ∈ C. Examples of synthetic utterances can be found in Appendix I. We report on the results separately in Section I.1 and the full results can be seen in Appendix J. We also consider synthetic examples generated using gpt-4 but found the average performance to be lower on our task (Appendix K).
I.1 Results: Methods using Synthetic Data
We evaluate the efficacy of methods using synthetic examples by generating a set of n = 20 synthetic examples, from which we sample k to act as class prototypes, we repeat this procedure 20 times and compute the average performance
across all samples. Table 14 shows averaged model performance across all 12 selected models and samples for k = [1, 3, 5, 10, 15]. For full results see Table 18 in Appendix J. We conducted additional experimentation with k > 15 but found further increasing k did not yield significant improvements in performance. We note our method using k = 15 synthetic examples outperforms tokenized labels on SNIPS (80.06 vs 76.30) and CLINC (69.99 vs 67.18) datasets, but underperforms slightly on the ATIS dataset (31.12 vs 31.70). Synthetic examples underperforms description-based methods by a considerable margin on all datasets, suggesting single intent label descriptions can be more powerful as class prototypes than synthetic instances. We note also the higher standard deviation σ in performance compared to the description-augmented method but lower compared to methods using tokenized labels.
I.2 Table of intents, descriptions and sampled synthetic examples generated using gpt-3.5-turbo
See Table 15 (ATIS), Table 16 (SNIPS) and Table 17 (CLINC).

heading: J Full table of results for approach using synthetic examples generated using gpt-3.5-turbo
text: See Table 18.

heading: K Table of averaged mean and standard deviation statistics for examples generated using gpt-4
text: See Table 19.

heading: Future Work
text: The impact of topical granularity per intent class and the potential for a hierarchical approach to intent classes in a dataless setting can be the focus of future work in this area. (2) Action Overlap: Our approach mitigates some errors arising from shared entities across intents through masking. Whilst this has shown success in reducing errors of this nature (i. e. between PlayMusic and AddToPlaylist from the SNIPS dataset), it is less successful in events where an action is shared across classes, such as play from the MASSIVE dataset, and SearchCreativeWork and SearchScreeningEvent from the SNIPSNLU dataset. Future work could investigate the potential to decouple the desired action and object
(topical information) in utterance embeddings. Label Candidate Analysis We observed from our results (Table 2) that our approach, despite outperforming strong baselines on ATIS and MASSIVE datasets, still consistently underperforms compared to the same setup on SNIPS-NLU and CLINC. We therefore investigate the position of the correct label when ranking embedding similarities. Table 6 shows the percentage of examples where the correct label is ranked within the top-k by embedding similarity for k = 1, 3, 5, 10. We note for erroneous predictions, the correct label is within the Top-3 in 67. 11% of cases, 81. 89% in Top-5 and 90. 94% in Top-10. This implies that our approach can be used to identify candidate intents from a larger set of intents, with a high success rate even for small values of k (i. e. 91. 01% Top-3). Analysis Summary Our proposed approach performs well overall against the strong baseline methods in unseen intent classification; however, it struggles in certain instances with overlaps in intents within the same domain. We identified potential areas for future work to pursue in tackling said issues. The results of our experiments have shown intent label descriptions can perform well as intent prototypes in this problem setting, and that the addition of paraphrasing and masking can further improve performance. Limitations This approach contains a number of limitations: We have identified issues with the descriptiveness of individual labels earlier in this section, and textual labels may not be readily available for certain datasets, though summarisation methods may be effectively applied to a few user utterances to produce such labels. Our evaluation compares against previous works using scores as reported in their respective papers, further work can be done to replicate their experiments to mitigate any potential risk arising from differences in experimental settings. Future work may also investigate
the application of descriptions to tasks outside of intent classification, such as emotion recognition (Rashkin et al. , 2019)