Can Your Model Tell a Negation from an Implicature? Unravelling Challenges With Intent Encoders (2024)

Abstract: Conversational systems often rely on embedding models for intent classification and intent clustering tasks. The advent of Large Language Models (LLMs), which enable instructional embeddings allowing one to adjust semantics over the embedding space using prompts, are being viewed as a panacea for these downstream conversational tasks. However, traditional evaluation benchmarks rely solely on task metrics that don’t particularly measure gaps related to semantic understanding. Thus, we propose an intent semantic toolkit that gives a more holistic view of intent embedding models by considering three tasks– (1) intent classification, (2) intent clustering, and (3) a novel triplet task. The triplet task gauges the model’s understanding of two semantic concepts paramount in real-world conversational systems– negation and implicature. We observe that current embedding models fare poorly in semantic understanding of these concepts. To address this, we propose a pretraining approach to improve the embedding model by leveraging augmentation with data generated by an auto-regressive model and a contrastive loss term. Our approach improves the semantic understanding of the intent embedding model on the aforementioned linguistic dimensions while slightly effecting their performance on downstream task metrics.

heading: 1 Introduction
text: Conversational systems use intent embedding models to encode input utterances into vectors that are used to understand intent semantics for few-shot intent classification and/or intent discovery (Zhang et al., 2021; Ma et al., 2022; Sung et al., 2023). The intent classification task leverages a predefined distance metric to find the nearest intent class/instances to the test utterance in the embedding space (Vinyals et al., 2016; Snell et al., 2017;
∗Work conducted as in an intern/employee of Amazon.
Dopierre et al., 2021), while intent discovery considers a clustering algorithm on top of multiple utterance embeddings to detects novel intent clusters. While these applications have been studied separately, a common underlying assumption is the embedding space encodes semantics (and a distance metric) between utterances and intents. To achieve this, various approaches have been proposed such as supervised pre-training on labeled utterance data belonging to a wide range of intents (Zhang et al., 2021, 2022a) or by combining pseudo intent names (Sung et al., 2023; Mueller et al., 2022).
However, the evaluation of these embedding models only consider existing conversational benchmarks (and task metrics) that lack dedicated test data necessary to evaluate gaps in semantic understanding. In this paper, we consider a complementary evaluation approach that tries to understand how well these embedding models capture the semantics of two common linguistic phenomenon seen in real-world conversational systems: negation – negation semantics alongside an explicitly mentioned intent utterance (e.g. No, I don’t want you to play music!) and implicature – utterances indirectly hinting at an intent that might require some reasoning steps (e.g. I feel like dancing =⇒ play some music). For this purpose, we propose an Intent Semantics Toolkit that includes challenging test splits for existing classification & clustering tasks and a novel triplet task. The triplet task consider an utterance triplet ⟨original, implicature, negation⟩ and evaluates if the implicature utterance is closer to the original utterance in the embedding space as compared to negation. The implicature and negation utterances for the test data are generated using two novel prompt designs for ChatGPT followed by human-in-the-loop quality control mechanisms.
The recent popularity of embeddings derived from Large Language Models and the possibility
552
of prompt-based encoding give an impression of semantic understanding, making them seem like an ideal candidate for the aforementioned intent identification tasks. Our proposed Intent Semantics Toolkit indicates that current representation of the negation and implicature utterances are far from perfect (see Figure 2).
To improve an embedding model’s semantic understanding on the aforementioned linguistic phenomena, we consider a fine-tuning approach that leverages LLM-generated positive (related to an intent) and negative (unrelated to an intent) utterances for augmentation alongside a contrastive loss objective. Our ablations highlight the need for both kinds of utterances for augmentation and our best model consistently outperforms the original embedding models on the triplet task. Observing different magnitudes of improvement on the different tasks, we dive deep to understand correlation between the various tasks and highlight some negative interactions. This highlights that approaches to improve embedding models on semantic understanding and downstream task might need to consider trade-offs; this is in line with several previous works (Marelli et al., 2014; Jeretic et al., 2020; Sengupta et al., 2021; Cooper Stickland et al., 2023). To summarize, our contributions are three-fold: (1) We iden-
tify two linguistic challenges that are commonly observed in real-world intent detection systems, but are overlooked in the literature: negation and implicature.1 (2) We devise Intent Semantics Toolkit that includes a novel triplet task and exposes the shortcomings of intent embedding models on semantic understanding of negation and implicature. We propose prompting strategies to generate evaluation data with ChatGPT and consider humanin-the-loop quality control. (3) We explore finetuning approaches with automatically generated utterances for data augmentation and interpret the semantic dimensions of implicature and negation as positive and negative examples in the contrastive learning loss. The results show that generated utterances can help improve the performance on the triplet task.
The code to reproduce Intent Semantic Toolkit and the finetuned models will be available at https://github.com/amazon-science/ intent-semantics-toolkit.

heading: 2 Related Work
text: Sentence embedding evaluation. Sentence embeddings are used in various downstream appli-
1We note that these phenomenon also have equivalence in the context of reasoning.
cations, especially where efficiency is important (Reimers and Gurevych, 2019; Su et al., 2023). For instance, semantic search of relevant document from a large vector database based on a similarity metric (Nguyen et al., 2016; Thakur et al., 2021), clustering a set of ‘unorganized’ documents based on the semantics (MacQueen et al., 1967; Ester et al., 1996), and/or classification of an input at test time to the semantically closest class seen during training (Vinyals et al., 2016; Snell et al., 2017; Conneau and Kiela, 2018). Recent works have also proposed a benchmark that unifies the evaluation of sentence embeddings across the various tasks (Muennighoff et al., 2023). Along these lines, Liu et al. (2023) proposes a triplet evaluation task that examines similarity of triplets sampled from existing class labels. Peng et al. (2024) explores triplet task for evaluating instruction-following embedders. In our work, we also propose a triplet evaluation task, but consider examining triplets based on dimensions related to language semantics such as implication and negation. Conversational implicature. Conversational implicature is often conveyed with an utterance to imply something without literally mentioning it (Grice, 1975). It is the non-conventional meaning of the utterance and is the assumption made in a certain context (Recanati, 1989), which can be used to efficiently communicate, maintain social relations or mislead without lying (Zalta et al., 1995). For instance, the utterance “I have not had breakfast today” can imply that the person is hungry even though it is not literally being said. In the domain of natural language processing, it is often implicitly mentioned in QA settings when a question needs to be clarified (Aliannejadi et al., 2021;
Sun et al., 2022; Zhou et al., 2022). There are also works claiming that current NLI language models or even LLM do not have enough understanding of implicature (Jeretic et al., 2020; Ruis et al., 2022). In this work, we brought implicature to intent detection scenarios and we are especially interested in the capabilities of embedding models.

heading: 3 Preliminary
text: A model (f ) for embedding encodes input sentences (ui) onto a continuous vector space (f(ui)). For a good embedding model, a distance metric D over embeddings (say f(ui) and f(uj)) should be able to capture some notion of semantic similarity that in turn empowers downstream applications such as clustering and classification. In this paper, we will consider D to be the widely popular cosine distance, i.e. D(f(ui), f(uj)) = 1− f(ui) T f(uj)
||f(ui)||||f(uj)|| . While these embedding models are common in several applications, we will concentrate on intent embedding models, where input sentences are user utterances (e.g. I would like to order food) and intents (e.g. order food, cancel order, etc.) comprise of semantic clusters represented by a set of similar user utterances. In such contexts, embedding models have been used to perform few/zero-shot classification of test-time utterances to a set-of predefined intents (Snell et al., 2017; Dopierre et al., 2021; Sung et al., 2023) or to discover novel intent classes given a set of unlabelled user utterances (Zhang et al., 2022b; Gung et al., 2023).
Despite its success, we observe that these intent embedding models often fail to capture nuanced language semantics that are common in realworld conversational agents. While previous work has shown such failures are common in the context of real-world noise (Sengupta et al., 2021; Cooper Stickland et al., 2023), we focus on two semantic aspects– (1) Negation: utterances that explicitly express no interest towards a particular intent (e.g. I don’t want to order a drink.), and (2) Implicature: utterances that do not explicitly convey an intent but imply it (e.g. I am hungry → order_food). Figure 2 shows that intent embedding models embed negations (that clearly disregards an intent) closer to intent utterances than implicatures (that implies an intent). In the real-world, we observe negation often occurs when a system incorrectly directs a customer towards an intent (and the customer has to explicitly mention they
didn’t intend it), while implicature is common due to a customer’s incomplete knowledge about the potential functionality supported by a chatbot.

heading: 4 Intent Semantics Toolkit
text: In this section, we first introduce the four evaluation tasks in our toolkit that analyze the semantic understanding capabilities of intent embedding models (§4.1). Then, we consider the challenge of obtaining data related to negation and implicature (§3) and quality assurance procedures to ensure a high-quality test set (§4.3). Eventually, we show evaluation of SOTA models on our dataset (§4.4).

heading: 4.1 Evaluation Tasks
text: Triplet Task A triplet task is composed of three utterances {ui, upi , uni }, where ui (I want to order a pizza) is an utterance belonging to intent i (order_food), uni is the negation of ui (I don’t want to order pizza), and upi is either another utterance of the same intent (I need pizza) or an implicature one (I am really hungry). For each triplet, we expect the negated utterance uni to be embedded further away from ui than u p i , i.e. D(f(ui), f(u p i )) < D(f(ui), f(u n i )). As we show later, this is often difficult for embedding models that may focus on other surface-form aspects of utterances rather than nuanced semantic understanding of intents. We calculate success among a set of NT triplets as,
1
NT
NT∑
i=1
I(D(f(ui), f(upi )) < D(f(ui), f(u n i ))) (1)
In principle, we can also define another success rate by interchanging ui and u p i as follows,
1
NT
NT∑
i=1
I(D(f(upi ), f(ui)) < D(f(u p i ), f(u n i ))) (2)
Given upi and u n i are not direct negations of one another, the value of D(f(upi ), f(u n i )) in (2) is expected to be higher than D(f(ui), f(uni ))) in (1) above. Hence, (2) is a more relaxed success criterion. Intuitively, as shown in Figure 3, it is harder for the model to discriminate “I don’t want to order pizza” from ”I want to order a pizza” due to the high vocabulary overlap between them. Thus, we denote the (1) as Thard and (2) as Teasy hereafter. We will also show two cases “Ori-Ori” or “Ori-Imp” where upi is from either the original test set or the implicature set.
Binary Classification In the binary classification task, an utterance ui needs to be classified into either original intent i or a negated intent class¬i. Thus, success implies D(f(ui), f(i)) < D(f(ui), f(¬i)). And the success rate is calculated among a set of NB utterances,
1
NB
NB∑
i=1
I(D(f(ui), f(i)) < D(f(ui), f(¬i))) (3)
We compute success rates for three different sets of utterances– original and implicature (closer to f(i)) and negation (closer to f(¬i)). We report accuracy for binary classification.
Clustering The input to the task is a set of unlabeled utterances {ui}. A clustering algorithm C then takes the embeddings {f(ui)} as inputs and outputs clustering indices {qi} where number of clusters k is first specified. Since the permutation of {qi} might be different from labels, we measure normalized mutual information (NMI) (Estévez et al., 2009) between them as our metric. We show results on both original and implicature sets with two clustering algorithms: k-means (MacQueen et al., 1967) and agglomerative clustering (Nielsen and Nielsen, 2016).
Multi-class Classification We adopt ProtoNet (Snell et al., 2017) as the classifier for few-/zero- shot classification aligning with previous works (Dopierre et al., 2021; Sung et al., 2023). It takes as inputs a training set {ũj , ỹj} and label names {lc}. A class prototype is first calculated for each class by averaging the embeddings of utterances of that intent class and the label name,
pc = 1 Nc + 1 {[ ∑
ỹj=c
f(ũj)] + f(lc)} (4)
During test time, we simply find the prototype that is closest to the test utterance argmincD(pc, f(ui)). Different from previous works, we discard episodic training and always use full categories within the dataset instead of random M ways. We argue that such a setting is
more realistic and show results on both original and implicature sets with 0/10-shots. For evaluation metric, we report accuracy.

heading: 4.2 Data Generation
text: In this section, we show data generation workflows with ChatGPT2 models (in order to achieve high quality) on test splits of 3 intent-classification benchmarks: BANKING77 (Casanueva et al., 2020), HWU64 (Liu et al., 2021) and CLINC150 (Larson et al., 2019). Across all datasets, we needed to make the intent names more informative (e.g. “mpg” to “check car mpg”) and focused on four dimensions when optimizing prompts for the LLM-generation: (R1) faithful: The generated utterances correctly conveys/negates the target intent, (R2) realistic: The generated utterances can be said by a real-world customer, (R3) diverse: The generated utterances are diverse enough for evaluation on a large scale, and (R4) reproducible: The temperature was set to 0 to guarantee reproducibility. Negation. Directly negating an utterance without a target intent has lower guarantee on generation faithfulness. For instance, consider the utterance “are there restrictions for carry-ons on Delta” that belongs to the intent “carry on”. The generated negation utterance is “are there no restrictions for carry-ons on delta” with gpt-4-0613 and temperature set to 0, which still belongs to the same intent. In order to reduce the task complexity for LLMs, we first manually write a few negated intents for each original intent. And then we instruct the LLM to directly modify original utterances according to a randomly sampled negated intent corresponding to the original one. To further increase the generation quality, we provide 6 in-context examples in the prompt manually written by humans. We choose gpt-4-0613 for negation. Implicature. Implicature utterances are completely generated by LLMs without original utterances provided. In order to guarantee diverse and realistic utterances, we first ask an LLM to brain-storm 10 scenarios that a customer may encounter scenarios for a certain intent, where the scenario may contain various user roles and situations, such as “Honeymooner” and “The customer wants to receive travel alerts for their honeymoon destination to ensure a smooth and safe trip.” for intent “request travel alert”. We choose
2Both gpt-4-0613 and gpt-3.5-turbo-0613 are used.
gpt-3.5-turbo-0613 for this part since the preliminary results show sufficient quality. The purpose of this step is to (a) help the model better understand the intent and generate high-quality utterances and (b) diversify the utterances by directly controlling the scenarios. Then, we use another LLM to generate 3 utterances for each scenario. We choose gpt-4-0613 for this step to ensure data quality. And we additionally provide the definition of implicature and 3 manually written in-context examples for higher quality.

heading: 4.3 Data Quality Control
text: We first employ automatic metrics to understand the generated data. To achieve this, we use training set as reference and calculate BLEU (Papineni et al., 2002), ROUGE-L (Lin, 2004), METEOR (Banerjee and Lavie, 2005) and BertScore (Zhang et al., 2020). We show the measures on three sets separately in Figure 4. As expected, original set has the largest vocabulary overlap with training set, probably because the original utterances are generated in the same style with lack of variability. Negation set contains fewer vocabulary overlap but still significantly higher than implicature set, which demonstrates they are similar to original ones in surface form.
However, merely measuring vocabulary overlap does not necessarily guarantee the faithfulness of generated data. Hence, we conduct human evaluation centered around two questions: (1) “Can the utterance imply the intent?” (2) “If yes, is it conveyed explicitly?” We first sample 180 utterances containing 60 from original, 60 from negation and 60 from implicature. 8 human annotators from our internal team (with fluent English skills) provided binary answers for these two questions in a sequen-
tial manner3. We show the annotation guidelines in Figure 7. For each utterance, 3 annotations are gathered, and those without 3 annotations are filtered out. We take the majority as the final decision. In Table 1, we show the results for all three sets. Both negation and implicature generations are indeed faithful. Furthermore, 88.46% of original set are annotated as explicit while 62.50% of implicature set are annotated as not explicit. Finally, the inter-annotator agreement (i.e. the proportion of queries with 3 consistent answers and with those that have less than 3 answers filtered) for Q1 is 79.10% and Q2 is 62.11%, which shows consensus in Q1 while some conflicts in Q2 probably due to its subjectivity.

heading: 4.4 Evaluation Results
text: We report results on 3 intent encoders. (1) paraphrase is a vanilla Sentence-BERT (Reimers and Gurevych, 2019) model4 which shows strong performance on intent detection (used as an initialization in Sung et al. (2023)). (2) IAE (Sung et al., 2023) first employs an intent-role labeler to extract pseudo intent names and then optimizes contrastive objective leveraging utterances, pseudo intent names and golden intent names. (3) Instructor (Su et al., 2023) pre-trains on multi-task dataset with instructions (“Represent the purpose for retrieval: ”) pretended on each input text. We experiment with both the base and large versions.
The evaluation results in Table 2 shows (1) success rates from Thard are consistently low (< 25%) indicating positive utterances (i.e. utterances with the same intent) are further away that negations. In contrast, success rates for Teasy are much higher, highlighting implicature and negation utterances
3We also allow them to annotate “unsure”. 4https://huggingface.co/sentence-transformers/
paraphrase-mpnet-base-v2
are far away from one another; (2) for binary classification, performances on the implicature split are lower than those on the original set showcasing the former is set of utterances are further awar for the expected intent than the latter; (3) for clustering and classification, performances on implicature sets are consistently lower than the original sets, however, this might be due to the multi-labeled nature of the implicature utterances. In Appendix A, we verify the upper bound with gpt-4-0613 on 10-shot classification task and demonstrates there is still a large room for improvement; (4) instructor-large consistently improves upon its base model, and instructor-base that possesses a similar parameter size with IAE seems to outperform the latter on most tasks. Given this, we use instructor-large as the baseline embedding model for improvement.

heading: 5 Model Improvement
text: We seek to improve the semantic understanding of embeddings models to negation and implicature. To answer this, we first introduce a data curation procedure where new utterances are generated based on an unlabeled dialogue corpus (§5.1) and then investigate continued fine-tuning approaches with a contrastive learning loss objective adapted for the triplet task (§5.2). The lower part of Figure 1 given a diagrammatic overview.

heading: 5.1 Fine-tune Data Curation
text: Similar to (Sung et al., 2023), we first collect a set 252, 744 unique and unlabelled utterances from general domain dialogue datasets: MultiWOZ (Zang et al., 2020), SGD (Rastogi et al., 2020), TOP (Gupta et al., 2018) and TOPv2 (Chen et al., 2020). We then use a performant LLM, namely falcon-40b-instruct (Almazrouei et al., 2023) for the sub-modules in Figure 5.5 Another rationale for these design choices is that they ensure a separation between training and evaluation data, thereby demonstrating the model’s generative capabilities.
As highlighted in Figure 5, for each unlabelled utterance, we first use the LLM to generate a user goal using the prompt what does a customer want by saying ‘ui’?. Then, we use a dependency parser6 to extract the action (ROOT) and the object (dobj) tokens from the predicted user goal (e.g. “know
5We opt out of using OpenAI models due to their restrictions on distilling data for model training, see §7 for details.
6https://spacy.io/
the distance between their home and the town of Warren.” in Figure 5). We empirically found that in some cases, the object is not found, so we further prompt the LLM to extract an object token from the user goal. Otherwise, the final extracted actionobject tuple (e.g. “know distance” in Figure 5) is then used for the finetuning utterance generation.
To generate hard positive/negative utterances, we propose to utilize a “zoo” of prompts written by experts that asks the LLM to imagine itself as the customer and generate user utterances. Specifically, for each candidate utterance, we sample two prompts, one for generating positive (that aligns with the original intent) and one for generating negative (that diverge from original intent). Table 9 in Appendix gives an overview of all the prompts we used for fine-tuning in this work. The generated utterances will be further diversified before training using heuristics as presented in Appendix B. Apart from LLM-generated utterances, we also combine with positive and negative utterances from the same training dataset. Specifically, for each unlabelled utterance, we retrieve one positive utterance that has the same action-object and one negative utterance that has different action-object but close in cosine distance7. These retrieved utterances will be used to facilitate contrastive learning as well as
7We always use the same embedding model for utterance retrieving and fine-tuning.
finetuning an ablation model in Table 5 that does not utilize LLM-generated utterances. Refer to Appendix B for more details.

heading: 5.2 Fine-tuning Objective
text: We adopt fine-tuning objective from Su et al. (2023); Zhang et al. (2023) where our input is a batch B of triplets {u, up, un} generated in the previous section (§5.1) and placed within the same instruction template used in the triplet task evaluation (in §4.4). More precisely, if we let i and j denote indices of B, the objective function pulls together the original utterance ui and it corresponding positive example upi while pushing away all in-batch negative unj ,∀j ∈ B. Precisely, we consider the following loss function:
li = exp(s(f(ui), f(u p i ))/γ)∑
j∈B exp(s(f(ui), f(u n j ))/γ)
+ exp(s(f(upi ), f(ui))/γ)∑
j∈B exp(s(f(u p i ), f(u n j ))/γ)
(5)
where γ is a temperature parameter. The second term swaps the positive and anchor.

heading: 6 Experimental Results
text: In this section, we first evaluate our fine-tuned models in §5 on our proposed Intent Semantics Toolkit. We then select the best prompt combination for training models on the validation set, and then evaluate on the test set. Finally, we shed lights on the
correlations between tasks in §6.3 and the effectiveness of LLM-augmented utterances when combining intent-aware encoder training (Sung et al., 2023) in §6.4.

heading: 6.1 Experimental Setup
text: We follow fine-tuning parameters in Su et al. (2023), except that we choose smaller learning rate 4× 10−6, batch size 8, maximum sequence length 128 and train with one epoch. These parameters are consistent across all experiments in this paper. Each finetuning experiment can be run on a single Amazon EC2 G5 instance.

heading: 6.2 Main Results
text: The fine-tuned models are evaluated on our proposed Intent Semantics Toolkit in §4. We split the evaluation data into 50% for dev and 50% for test to select proper prompt combinations. We report results based on the test set using instructor-large as the baseline in Table 3 and highlight some ablations study with various prompts in Table 5 (see Appendix for experiments with other prompts and instructor-base). As It is difficult to exhaust all possible combinations of prompts, we consider a subset of experiments in our ablations and rank the various trained models using Colombo et al. (2022). We observe that (1) disabling all LLM-generated data achieves lower performance than Baseline, (2) disabling LLMgenerated hard positive degrades performances on clustering and classification tasks, while disabling hard negative degrades performances on triplet tasks and binary classification tasks, and (3) the best performance is achieved by “−P 4 − N1,3”, which outperforms the Baseline on most tasks. As
T ha rd (o
ri)
T ea sy (o
ri)
T ha rd (im
p)
T ea sy (im
p) BC (o ri) BC (im p) BC (n eg) KM (o ri) KM (im p) Agg (o ri) Agg (im p) 0s (or i) 10s (o ri) 0s (im p) 10s (im p)
Thard (ori) Teasy (ori) Thard (imp) Teasy (imp)
BC (ori) BC (imp) BC (neg) KM (ori) KM (imp) Agg (ori)
Agg (imp) 0s (ori)
10s (ori) 0s (imp) 10s (imp)
1 0.9 1 0.8 0.7 -0.4 -0.4 -0.1 -0.2 0.4 0.3 0.2 -0.3 0.3 0.3 0.9 1 0.8 0.9 0.9 -0.7 -0.7 -0.1 -0.1 0.1 0.1 -0 -0.2 0.1 0.1 1 0.8 1 0.8 0.6 -0.4 -0.4 -0.1 -0.1 0.3 0.3 0.2 -0.2 0.2 0.3 0.8 0.9 0.8 1 0.8 -0.8 -0.8 -0.1 -0 0 -0 -0.1 -0.1 -0 0 0.7 0.9 0.6 0.8 1 -0.8 -0.8 -0.2 -0.2 -0 -0.1 -0.1 -0.2 -0 -0 -0.4 -0.7 -0.4 -0.8 -0.8 1 1 0.3 0.2 0.4 0.4 0.5 0.2 0.4 0.4 -0.4 -0.7 -0.4 -0.8 -0.8 1 1 0.3 0.2 0.4 0.4 0.5 0.2 0.4 0.4 -0.1 -0.1 -0.1 -0.1 -0.2 0.3 0.3 1 1 0.7 0.7 0.8 0.9 0.7 0.8 -0.2 -0.1 -0.1 -0 -0.2 0.2 0.2 1 1 0.6 0.6 0.8 1 0.7 0.7 0.4 0.1 0.3 0 -0 0.4 0.4 0.7 0.6 1 0.9 0.9 0.5 0.9 0.9 0.3 0.1 0.3 -0 -0.1 0.4 0.4 0.7 0.6 0.9 1 0.9 0.6 0.9 0.9 0.2 -0 0.2 -0.1 -0.1 0.5 0.5 0.8 0.8 0.9 0.9 1 0.7 0.9 0.9 -0.3 -0.2 -0.2 -0.1 -0.2 0.2 0.2 0.9 1 0.5 0.6 0.7 1 0.5 0.6 0.3 0.1 0.2 -0 -0 0.4 0.4 0.7 0.7 0.9 0.9 0.9 0.5 1 0.9 0.3 0.1 0.3 0 -0 0.4 0.4 0.8 0.7 0.9 0.9 0.9 0.6 0.9 1
Figure 6: Pearson correlations between tasks using performances on dev set as a feature vector for each task.
seen in Table 3, while we notice consistent improvement with our model across original and our proposed tasks, the magnitude of improvements is not uniform (eg. a ↑ 27.7 on Thard vs. ↓ 0.3 on the binary task with implicatures). To understand this better, we consider the correlation between tasks.

heading: 6.3 Correlation between tasks
text: We plot the pearson correlation matrix between pairs of tasks across the results from instructor-large (Table 5) and instructor-base (Table 8) dev set performances. Figure 6 clearly highlights the negative correlation between the first two tasks (triplet tasks and binary classification) that demand the discrimination of negation and the last two (clustering and classification). Thus, in order to develop robust conversational system, we encourage future works to consider trade-offs in improving an embedding model across all tasks.
Original Intent Semantic Toolkit Model Rank Clustering Multi-class Triplet (Ori-Ori) Triplet (Ori-Imp) Binary Classification Clustering Multi-class
green represents increased score compared with “Baseline” (vanilla instructor-large) and red vice versa.
Original Intent Semantic Toolkit Model Clustering Multi-class Triplet (Ori-Ori) Triplet (Ori-Imp) Binary Classification Clustering Multi-class

heading: 6.4 Augment Intent-Aware-Encoder
text: We further show the performances by adding LLMgenerated data into Intent-Aware-Encoder (IAE) pre-training data in order to properly compare performances. “IAE-pos” replaces part of the IAE loss function that uses pseudo label names with LLM generated data augmentation (for positive labels), while “IAE-pos-neg” uses both positive and negative examples (either retrieved or LLM-generated) in the contrastive loss term. From Table 6, we still observe performance improvements on triplet task and binary classification when using “IAEpos-neg” and comparable performances for clustering and multi-class classification tasks. These results further confirm the effectiveness of utilizing LLM-generated utterances to improve embedding models.

heading: 7 Conclusion and Future Works
text: In this paper, we propose a new evaluation toolkit for intent embedding models that measure their semantic understanding on two lingusitic phenomenon commonly observed in conversational systems– negation and implicature. For this we propose Intent Semantic Toolkit that combines a novel triplet task, a binary classification task and the test splits that evaluate the model’s semantic understanding on downstream intent recognition (classification and clustering) tasks. Our study shows that current intent embedding models do not have sufficient understanding of these two real world
phenomenon, i.e. negation and implicature. We then propose to integrate hard positives and negatives generated from an LLM with a “zoo” of prompts to fine-tune the model. The fine-tuning is conducted on a set of unlabeled utterances from general domain and is evaluated on our proposed toolkit. Our best model demonstrates improvements across most tasks over baseline models. We further combine LLM-generated data with IntentAware-Encoder training and show performance improvements across all original test datasets and most newly introduced evaluation datasets in the Intent Semantic Toolkit. Finally, the correlation analysis between pairs of tasks indicates that a better balance between negation-related tasks and regular benchmark tasks need to be achieved. Our work also inspires future works to develop and evaluate an instruction-following embedding model that can improve performance via prompting without the need for further fine-tuning of the model.
The negation utterances discussed in this paper have significant relevance with out-of-scope utterances – utterances that can not be covered by existing intents/services. On the other hand, negation utterances should also be rejected by current intent classifiers. However, negation utterances may be harder to be distinguished due to the high vocabulary overlap with positive utterances. Moreover, customers may demand the conversational agents to respond differently for negation and out-of-scope utterances. Thus it is necessary to study them separately.
Limitations
Usage of LLMs. In this paper, we used ChatGPT for evaluation and a smaller LLM falcon-40b-instruct for training. This is mainly due to the legal concern. Our use of OpenAI for this publication, is to the best our knowledge, in compliance with applicable OpenAI’s terms and conditions as of March 14, 2023. Our results show that Falcon-based data augmentation improves performance significantly. However, we argue that using a larger and more capable LLM can potentially improve the quality of data generated and thus further improve the model. Multi-intent Issue of Implicature Utterance. The dataset we introduced in this paper is collected via prompting LLMs. We designed strategies in §4.2 to generate utterances for negation and implicature and then validated the quality of generated data in §4.3. However, as mentioned in §4.4, we observe multi-intent utterances in our generated utterances, which might affect the understanding of results from clustering and multi-class classification. For instance, the utterance “set the mood please” can correspond to both “change light hue” and “play music”. We thus validate the classification performance with GPT-4 as presented in Appendix A, and show that performance of the intent embeddings are still much lower than it. We also want to point out that such a multi-intent utterance might naturally occur in real-world applications and thus they are also valuable for understanding model performance.
Ethical Considerations
This paper uses various open-source datasets and models for evaluation and training, which are reproducible. Our evaluation toolkit uses OpenAI models which might impact the reproducibility. However, we set the temperature to 0 in order to reduce the variances.

heading: Acknowledgements
text: We would like to thank the AWS AI Team in general and James Gung, Hossein Aboutalebi, and Dennis Ulmer in particular for insightful discussions and their contribution to our pilot study. In addition, we also would like to thank the reviewers in two iterations of ARR, for the February 2024 and the April 2024 cycle, whose comments and suggestions have helped us improve the work.

heading: A Classification Upper-Bound with GPT-4
text: In Section 4, we discussed the multi-label nature of implicature utterances because of the inherent ambiguity of these utterances. For instance, “set the mood please” could mean both “INTENT:change_light_hue” and “INTENT:play_music”. Such a phenomena could be one of the reasons that the evaluated embedding models show degraded performances compared with original set. In order to decompose the effects of multi-label, we present a classification upper-bound on 10-shot multi-class classification with gpt-4-0613. We present gpt-4-0613 with the utterance and the top-5 predictions from instructor-large plus one ground truth label. We then ask the model to classify it into one of the class. The calculated accuracy is comparable with 10-shot classification performance in Table 2. The obtained results from gpt-4-0613 is 55.2% which is significantly higher than 34.3%. This further illustrates that we still have a lot of rooms for improvement for embedding models.

heading: B More Details on Fine-tuning: Retrieving and Diversifying
text: Retrieving Apart from LLM-generated data, we also use the retrieved utterances which is considered as a standard data collecting procedure (and thus a baseline method) in contrastive learning. We retrieve one positive and one negative data for each of the utterances in the unlabeled corpus. For positive one, we simply uniformly choose one utterance from the same-intent utterance group. Notice that although there is no guarantee a positive utterance can be retrieved, in practice we found most of utterances in the training corpus can find positive pairs, i.e. 95.39% (241, 080 utterances). And we simply filter out the rest of utterances that can not retrieve at least one positive. For negative one, we first encode all the negative utterances and the original utterance into the embedding space with the same embedding model that is going to be trained. And then we calculate cosine distance between original embedding and negative embeddings, and sort them from smaller to larger. We then acquire the one in the middle of the list as we empirically found that these utterances are both similar to the original
utterances in terms of surface forms and possess different intents. Diversifying We empirically observe that the generated utterances usually contain similar surface forms, e.g. most utterances from P 4 starts with “i do not want to”. This is due to the question forms being used in the specific prompts, and may potentially harm the diversity of training data. In order to diversify them, we manually create more patterns such “i try not to”, “i prefer not to” for P 4, and then modify the original utterance with it. In addition, we also remove those responses that reject to produce answers by identifying keywords “ai language model”.

heading: C Qualitative Analysis on Errors
text: Note that we have some qualitative examples in Figure 2 that are sampled from our dataset to demonstrate the characteristics of implicature and negation. In order to better showcase some examples, we provide more qualitative analysis here on CLINC150, especially failure cases during binary classification tasks (classify to one of original label or negated label) using instructor-large.
Implicature errors:
i’ve recently retired and i’m trying to cut back on my spending (credit_limit_change) the passenger will be stopping in london, is the time difference going to be a problem? (timezone) i think my bags missed the connection (lost_luggage) i’m having a hard time understanding the app because it’s not in my native language (change_language) i can’t decide what to have for dinner (flip_coin)
Negation errors:
i didn’t mean to ask for the spanish word for pasta (no_need_to_request_translate) please don’t change my name, it was a mistake (no_need_to_change_user_name) my visa was damaged but it has been replaced (card_not_damaged) i like this song, don’t skip it (stay_at_current_song) i mistakenly set an alarm (no_need_to_set_alarm)
Furthermore, we performed clustering on these failure cases with instructor-large embeddings and then perform k-means clustering. We then run Tf-idf with each cluster as a document to acquire feature vectors for them, where the high value entries indicate keywords for the cluster. We show some example keywords for both negation and implicature here:
Implicature: ’my’, ’to’, ’the’, ’you’, ’your’, ’can’, ’not’ ’my’, ’card’, ’credit’, ’to’, ’the’, ’wallet’, ’bank’ ’to’, ’the’, ’for’, ’recipe’, ’salad’, ’lunch’, ’dinner’ ’to’, ’the’, ’my’, ’need’, ’meeting’, ’have’, ’for’ ’my’, ’to’, ’meter’, ’the’, ’utility’, ’reading’, ’bill’ ’the’, ’song’, ’to’, ’my’, ’this’, ’quiet’, ’is’ ’to’, ’the’, ’car’, ’my’, ’need’, ’get’, ’long’ ’to’, ’in’, ’thinking’, ’the’, ’for’, ’trip’, ’my’ ’language’, ’in’, ’my’, ’the’, ’to’, ’app’, ’english’ ’my’, ’luggage’, ’the’, ’suitcase’, ’bags’, ’bag’, ’plane’
Negation: ’reservations’, ’list’, ’to’, ’for’, ’the’ ’phone’, ’to’, ’know’, ’my’, ’you’ ’account’, ’my’, ’bank’, ’401k’, ’paid’ ’happened’, ’song’, ’what’, ’the’, ’is’ ’whisper’, ’mode’, ’settings’, ’changed’, ’unexpectedly’ ’card’, ’credit’, ’my’, ’the’, ’limit’ ’meeting’, ’booked’, ’my’, ’flight’, ’to’ ’thanks’, ’for’, ’not’, ’nothing’, ’thank’ ’my’, ’engine’,
’light’, ’to’, ’check’ ’jump’, ’car’, ’start’, ’my’, ’to’ The results did not indicate strong bias towards a specific topic, and thus showing that such failures are ubiquitous across domains and intent classes.
Original Intent Semantic Toolkit Model Rank Clustering Multi-class Triplet (Ori-Ori) Triplet (Ori-Imp) Binary Classification Clustering Multi-class
Original Intent Semantic Toolkit Model Rank Clustering Multi-class Triplet (Ori-Ori) Triplet (Ori-Imp) Binary Classification Clustering Multi-class

heading: Future Work
text: Our work also inspires future works to develop and evaluate an instruction-following embedding model that can improve performance via prompting without the need for further fine-tuning of the model. The negation utterances discussed in this paper have significant relevance with out-of-scope utterances – utterances that can not be covered by existing intents/services. On the other hand, negation utterances should also be rejected by current intent classifiers. However, negation utterances may be harder to be distinguished due to the high vocabulary overlap with positive utterances. Moreover, customers may demand the conversational agents to respond differently for negation and out-of-scope utterances. Thus it is necessary to study them separately. Limitations
Usage of LLMs. In this paper, we used ChatGPT for evaluation and a smaller LLM falcon-40b-instruct for training. This is mainly due to the legal concern. Our use of OpenAI for this publication, is to the best our knowledge, in compliance with applicable OpenAI’s terms and conditions as of March 14, 2023. Our results show that Falcon-based data augmentation improves performance significantly. However, we argue that using a larger and more capable LLM can potentially improve the quality of data generated and thus further improve the model. Multi-intent Issue of Implicature Utterance. The dataset we introduced in this paper is collected via prompting LLMs. We designed strategies in §4. 2 to generate utterances for negation and implicature and then validated the quality of generated data in §4. 3. However, as mentioned in §4. 4, we observe multi-intent utterances in our generated utterances, which might affect the understanding of results from clustering and multi-class classification. For instance, the utterance “set the mood please” can correspond to both “change light hue” and “play music”. We thus validate the classification performance with GPT-4 as presented in Appendix A, and show that performance of the intent embeddings are still much lower than it. We also want to point out that such a multi-intent utterance might naturally occur in real-world applications and thus they are also valuable for understanding model performance. Ethical Considerations
This paper uses various open-source datasets and models for evaluation and training, which are reproducible. Our evaluation toolkit uses OpenAI models which might impact the reproducibility. However, we set the temperature to 0 in order to reduce the variances