Intent Discovery with Frame-guided Semantic Regularization and Augmentation
Year: 2023

Abstract:
Most intent discovery methods leverage representation learning and clustering to transfer the prior knowledge of known intents to unknown ones. The learned representations are limited to the syntactic forms of sentences, therefore, fall short of recognizing adequate variations under the same meaning of unknown intents. This paper proposes an approach utilizing frame knowledge as conceptual semantic guidance to bridge the gap between known intents representation learning and unknown intents clustering. Specifically, we employ semantic regularization to minimize the bidirectional KL divergence between model predictions for framebased and sentence-based samples. Moreover, we construct a frame-guided data augmenter to capture intent-friendly semantic information and implement contrastive clustering learning for unsupervised sentence embedding. Extensive experiments on two benchmark datasets show that our method achieves substantial improvements in accuracy (5%+) compared to solid baselines.

heading: 1 Introduction
text: Dialogue systems such as chatbots and virtual assistants increasingly rely on their ability to understand and identify user intents, which directly impacts the system’s performance. (Tseng et al., 2019; Guo et al., 2021). Real-world dialogue systems have to deal with evolving user needs, resulting in continuously increasing intents. To address these challenges, intent discovery methods that aim to detect unknown user intents from a large number of unlabeled utterances have been developed in recent years (Perkins and Yang, 2019; Min et al., 2020).
Previous intent discovery work focuses on the unsupervised setting with clustering algorithms (Shi et al., 2018; Min et al., 2020). Min et al. (2020) proposed alternating-view k-means for joint multi-view representation learning and clustering
∗ Corresponding author.
analysis. However, the unsupervised approaches often produce unsatisfactory performance because of their low degree of utilization of prior knowledge of known intents. Therefore, recent works (Hsu et al., 2019; Lin et al., 2020; Zhang et al., 2021) invariantly pretrain an intention detection model using limited known intents data, followed by leveraging clustering techniques to infer unknown intention. Zhang et al. (2021) propose an iterative representation learning and clustering method, DeepAligned, for intention detection and discovery.
Learning intention-friendly semantic representation is vital for semi-supervised methods to improve the performance of intent detection and discovery. However, it’s insufficient to merely rely on the labeled data. Methods pretrained on known intents are limited to the syntactic forms of sentences, therefore, fall short of recognizing adequate variations under the same meaning of unknown intents. Figure 1 displays the semantic similarities between the sentences with the overlapping phrase “cash withdrawal on my account” but a different intention. Red and green polylines represent the sentence similarity produced by the BERT (Devlin et al., 2019) and DeepAligned. It can be observed that both BERT and DeepAligned models ill-group U1 and U3 into the same class because the similarity between U1 and U3 is higher than the decision threshold (i.e., 86% illustrated
14254
as a black dash line in Figure 1). But ideally, the model should group U2 and U3 into the same intent class (depicted as the blue line in Figure 1). It is important to enhance the semantic representation learning ability of intent discovery methods. In this paper, we propose an approach utilizing frame knowledge as conceptual semantic guidance to bridge the gap between known intents representation learning and unknown intents clustering. Frame knowledge is defined as the conceptual structure based on FrameNet (Baker et al., 1998; Das et al., 2014), capturing the background knowledge necessary to understand a situation. Specifically, we introduce a novel frame-guided semantic regulation and augmentation method to enhance intention discovery. Semantic regularization minimizes the bidirectional KL divergence between model predictions made on the frame and sentencebased samples to learn conceptual semantic information. In the meantime, we construct a frameguided data augmenter to capture intent-friendly semantic information and implement contrastive learning for semi-supervised sentence embedding learning. Finally, we perform an iterative representation learning and clustering method for intention detection and discovery. Extensive experiments on two benchmark datasets show that our method achieves substantial improvements, over 5% in accuracy compared to solid baselines.

heading: 2 Model

heading: 2.1 Task Formulation
text: Let Iknown = {I1known, I2known, . . . , Inknown} denote the set of n known intents and Iunknown = {I1unknown, I2unknown, . . . , Imunknown} denote the set
of m unknown intents. Given a labeled utterance set Dlabeled = {(x1, y1), (x2, y2), . . . , (xl, yl) | yi ∈ Iknown} which contains n known intents and an unlabeled utterance set Dunlabeled = {(x̃1, ỹ1), (x̃2, ỹ2), . . . , (x̃r, ỹr) | ỹi ∈ Iknown ∪ Iunknown} containing n known intents and m unknown intents, intent discovery aims to group utterances into different clusters using the PLMs trained by labeled utterance. In our work, we exploit KAFSPA (Zhang et al., 2023) to extract the frame annotation xfi , as shown in the Figure 2 and produce augmented sentence xai for the utterance xi.

heading: 2.2 Model Architecture
text: We adopt an iterative representation learning and clustering for intention discovery. In the representation learning stage, the parameters of the intention encoder module, as shown in Figure 2, are updated based on labeled data from known intents. The module proposes frame-guided semantic regularization and augmentation with four objectives to capture the conceptual semantic information of utterances. In the clustering stage, inspired by DeepAligned, we calculate pseudo-labels for unlabeled data and fine-tune Sentence-BERT for self-supervised learning. Specifically, we freeze the Frame-BERT parameters and minimize the KL divergence of Sentence-BERT and Frame-BERT models to retain the conceptual semantic representation of known intents.

heading: 2.3 Intention Encoder Module
text: The module is responsible for learning intentionfriendly semantic representation of utterance and performing intention detection. Specifically, given
an labeled intent samples xi, the corresponding frame annotation xfi and the augmented sample xai , we first obtain their intent representation zi and zai using Sentence-BERT with a pooling layer, and frame-based intention representation zfi with Frame-BERT encoder with a pooling layer. Then we employ four objectives in intention pretraining process: instance/frame-wise intent detection LCE(θ) and LfCE(θ), semantic regularization LKL(θ) and unsupervised semantic augmentation LSCL(θ) via clustering contrastive learning. Instance/frame-wise Intent Detection The objective aims to achieve intention detection based on the labeled utterances. We fine-tune sentenceBERT and frame-BERT with transformation heads respectively by minimizing the cross-entropy loss LCE and LfCE respectively, where
p(Iknown | xi) = softmax(zi ∗W + b))
LCE = − 1
N
∑
j
y · log ( p(Ijknown | xi) ) (1)
and LfCE is similarly defined by replacing xi, zi with xfi , z f i in Equation (1).
In Equation (1), p(Iknown | xi) denotes the probability of assigning to xi or x f i to the known intents. After fine-tuning two BERT models based on labeled utterances, the transformation heads are discarded. In the second clustering stage, BERT with a pooling layer is continuously fine-tuned while Frame-BERT with a pooling layer is frozen, which is regarded as conceptual semantic guidance.
Unsupervised Semantic Regularization To guide the Sentence-BERT concentrating on the conceptual semantic representation, we apply semantic regularization to minimize the bi-directional KLdivergence between model predictions made by sentence-BERT and frame-BERT modules. Given the intent representation zi and z f i , we assume Pϕ(z|zi) and Pψ(z|zfi ) take the following forms:
Pϕ(z|zi) ∼ N (µ, σ2I), Pψ(z|zfi ) ∼ N (µ, σ2I) (2)
The µ and σ are calculated based on the zi,
µ = zi ·Wµ + bµ log σ2 = zi ·Wσ + bσ
z = µ+ σ ⊙ ϵ (3)
and µ, σ are calculated in a similar way by replacing µ, σ, z, zi with µ, σ, z, z f i with learnable param-
eters Wµ,W σ, bµ and bσ in Equation (3). Then we formulate the negative KL divergence:
LKL(θ) = ∑
i Pϕ(z|zi) log ( Pϕ (z|zi) Pψ(z|zfi ) ) (4)
Semi-supervised Semantic Augmentation As shown in the Figure 2, the positive samples xai are generated by replacing the target words in the original samples xi with another word that belongs to the same frame as the target words. Then we employ contrastive learning to model instance-wise semantic similarities by pulling together intents belonging to the same class while pushing apart samples from different classes in one batch B. The equation is as follows:
LSCL(θ) = − log exp(sim(zi, z a i ))∑B
j=1 1[j ̸=i] exp(sim(zi, zj)) (5)
where B denotes the batch size. Finally, The intention encoder module is trained with the loss L.
L = LCE + LfCE + LKL + LSCL (6)

heading: 3 Experiments

heading: 3.1 Set up
text: The experiments are conducted on CLINC (Larson et al., 2019), and Banking77 (Casanueva et al., 2020) datasets. CLINC contains 22, 500 utterances with 150 intents, and Banking77 consists of 13, 083 utterances with 77 intents. Followed DeepAligned (Zhang et al., 2021) model, we randomly select 10% of training data as labeled and 75% of all intents as known intents. The model is compared with strong baselines: BERT-MCL (Hsu et al., 2019), CDAC+ (Lin et al., 2020) and DeepAligned (Zhang et al., 2021). Three widely used metrics are adopted to evaluate the clustering results: Accuracy (ACC), Normalized Mutual Information (NMI), and Adjusted Rand Index (ARI).

heading: 3.2 Training details
text: For a fair comparison with previous work, hyperparameters are adopted as suggested in Zhang et al. (2021). The parameters of Frame-BERT and µ, σ are frozen during clustering. The unlabeled data are aligned per 10 epochs in the clustering stage.

heading: 3.3 Result and Discussion

heading: 3.3.1 Main results
text: We report the results across all models on the CLINC and Banking77 with different known intents ratios in Table 1. Our model achieves the best results on all metrics and datasets compared to the baselines, demonstrating the effectiveness of the proposed approach. Another observation is that the smaller ratios of known intents are included in the fine-tuning, the more significant improvements are achieved compared to baselines. The contribution can be explained in two aspects. Firstly, frame regularization and augmentation alleviate the overfitting problem when fine-tuning. Secondly, freezing Frame-BERT in the clustering stage guides the Sentence-BERT to avoid noisy pseudo-labeled data distribution.

heading: 3.3.2 Ablation study
text: To understand the effect of the proposed method, we perform an ablation study (in Table 2) by removing frame regularization and augmentation, respectively. There is a significant drop in performance on all datasets with all metrics, especially for the Banking77 datasets when frame regularization and augmentation are absent. The dropped performance verifies the effectiveness of frame-based semantic guidance in bridging the gap between known intents representation learning and unknown intent clustering. The degradation is more obvious in Banking77 datasets because the intentions is finegrained and closed compared to CLINC datasets.

heading: 3.3.3 Case study
text: To further verify the model’s effectiveness, we visualize the representation in the clustering stage of DeepAligned and our model (in Figure 3). We
can find the distribution obtained by DeepAligned are sparse, with more utterance clusters indicating higher errors in the output. In comparison, the representation learned from our model appears aggregated to fewer groups, potentially ascribed to the effect of the conceptual semantic learning ability of the proposed method. The results are consistent with the case study shown in Figure 1. Our model performs closer to an ideal model in alleviating the ill influence the syntactic forms bring.

heading: 4 Conclusion and Future Work
text: This paper introduces frame knowledge as conceptual semantic guidance and proposes a frameguided semantic regularization and augmentation method. Experimental results on two benchmarks demonstrate that our method outperforms strong baselines by a wide margin. In future work, we will explore practical ways to reason latent intentions based on the frame ontology knowledge.
Limitations
There are two limitations to this work. (1) the total number of known and unknown intents are predefined, requiring an extension in real-world scenarios; (2) The frame knowledge is predefined and, therefore, inflexible to address complex intents. In addition, some user queries have no frame in FrameNet matching.
There are additional computation costs for frame knowledge learning. The model fine-tunes two BERT(bert-base-uncased, 340M) models in the training stage and runs sentence-BERT in the evaluation stage. The pre-training stage of our model lasts about 10 minutes, and clustering runs for 90 minutes on CLINC with a 75% known intents ratio, both using a single NVIDIA Tesla V100 GPU(32 GB of memory).

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section Limitation behind conclusion
3 A2. Did you discuss any potential risks of your work? in the section Limitation
3 A3. Do the abstract and introduction summarize the paper’s main claims? see in the section abstract and introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts?

heading: In the Section Experiments
text: 3 B1. Did you cite the creators of artifacts you used? In the Section Experiments, set up subsection
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? In the Section Experiments, set up subsection
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? In the section experiments and section Limitation
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? In the Section Experiments, set up subsection
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. In the Section Experiments, set up subsection
C 3 Did you run computational experiments?

heading: In the Section Experiments, training details subsection
text: 3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? In the Section Experiments, training details subsection
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? In the Section Experiments, training details subsection
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Not applicable. Left blank.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? In the Model Architecture Section
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In future work, we will explore practical ways to reason latent intentions based on the frame ontology knowledge. Limitations
There are two limitations to this work. (1) the total number of known and unknown intents are predefined, requiring an extension in real-world scenarios; (2) The frame knowledge is predefined and, therefore, inflexible to address complex intents. In addition, some user queries have no frame in FrameNet matching. There are additional computation costs for frame knowledge learning. The model fine-tunes two BERT(bert-base-uncased, 340M) models in the training stage and runs sentence-BERT in the evaluation stage. The pre-training stage of our model lasts about 10 minutes, and clustering runs for 90 minutes on CLINC with a 75% known intents ratio, both using a single NVIDIA Tesla V100 GPU(32 GB of memory)