Zero-Shot Intent Classification with Siamese Networks | by Duygu ALTINOK | Towards Data Science

Intent recognition is an essential task for goal-oriented dialogue systems. Intent recognition (sometimes also called intent detection) is the task of classifying each user utterance with a label, which comes from a predefined set of labels.
Classifiers train on labelled data and learns to discriminate which utterance belongs to which class. If an utterance that doesn’t look like any of the training utterances comes to the classifier, sometimes result can be quite embarrassing. That’s why we also classify out-of-domain utterances, which don’t belong to the domain at all.
Problem is that the users and we developers live in different words. Developers want to stay in their safe farm (so called a domain) in the semantics jungle, but users aren’t very aware of how classifiers or chatbot NLU work (and they don’t have to). You cannot expect users to stay in correct semantic territory, instead you should donate your chatbot with skills to handle ood utterances.
In this article, we’ll talk about how we keep our domain intact by detecting which utterances fall into the domain and which utterances are out of the domain for Chris, our voice assistant for drivers. First I’ll introduce the Chris domain and utterances, then we’ll go over what zero shot learning text classification with Siamese networks .
Chris is a voice assistant for drivers. Our Chris is a driver assistant, built by German Autolabs. Chris can send/read SMS, WhatsApp messages, make calls to the phone contacts, play music, navigate, respond to weather queries and make a bit chitchat.
Chris is a task-oriented conversational agent, hence user utterances are usually succinct and to the point. These are typical user utterances:
play musicsome music pleasestop the musicsend a messagesend a message to Tesssend a whatsappread my messagesdo I have any new messagesdisplay the latest messagestart the navigationshow navigationshow the mapstop the navigationnearest gas stationfind a parking spotread itno cancelcancelyesnextno nexttell me the timehow is the weather today
Intents names are formed with a subdomain name and an action name. Here are some intent names, corresponding to the above utterances:
music.playmusic.pausemessages.writemessages.readnavigation.startnavigation.stopuniversals.acceptuniversals.rejectuniversals.nextmisc.timemisc.weather
What about out-of-domain utterances? Here are some examples as well:
hey sirilaunch siriwake up siriopen my instagramshare my video on instagramdelete my facebook accountplay my audibooksset an alarmshow alarmsplease delete all alarmsopen the doors
It’s totally normal users to ask for some capabilities that your chatbot does not have at all. For instance, opening the car’s door is not in Chris’s skill set at all. Even though UX team includes a guideline about Chris skills in every packaging, users might skip reading it 😄 That’s why your chatbot NLU should always be prepared to deal with a broad set of out-of-domain utterances, users ask for abilities that they can imagine your chatbot can achieve, not the ones written in user manual.
Short utterances might look “easy” at the first sight, but there are certain challenges as well. Speech recognition mistakes can give hard time especially on short utterances, because semantically crucial words might be missing from the utterance (such as “play” in “play music”). Also speech engine has to start listening at the right time, otherwise it can miss one-word, short utterances such as “yes”, “no” which are crucial for resolving the context. Missing only 3 letters doesn’t look like a great ASR mistake on WER side, but it might create frustration for the user, if your agent asks the user for their approval many times. All voicebots has acoustic and semantic challenges, never underestimate the challenges of working with voice.
Zero short text classification is task of training the classifier on a set of class labels and test the classifier with class labels the classifier didn’t see in the training set. Recent work in NLP focused on zero-shot learning in a broader setting, zero-shot learning NLP now means train a model to do a task it didn’t explicitly trained on to do. GPT-3 is a zero-shot learner and attracted quite a lot attention.
In zero-shot classification, we describe an unseen class to the classifier, either with some visual clues or with the class name. For zero-shot text classification, usually intent name is used to describe the semantics of the intent. When I first started doing Chris NLU, data was intended for “usual” intent classification. Then I started experimenting for ood and found our class naming scheme of domain.action (music.play, navigation.start and so on) indeed very well suited for zero-shot learning.
A common approach in zero-shot text classification is to embed the intent names and utterances into the same space. This way zero-shot algorithm can learn the semantic relation between the intent name and utterances via a semantic group, just like us humans do. This is not a new idea at all, researchers used word vectors to represent the text and intent names in fixed dimensions (such as in Veeranna at all. 2016 ). By the invention of Transformers , dense representations experienced a revolution, now we have much more quality sentence and word embeddings.
In our study, we first used average pooled word vectors to represent the utterances, then BERT to encode our utterances to generate utterance vectors. First let’s see how intent names comes together with utterances when we used word vectors. We used 100-dimensional Glove vectors. We generated sentence embedding for an utterance by averaging the word vectors of its words. In order to get a label’s embedding (e.g. music.play), we averaged the embeddings of the domain (music) and the action (play). After generating an embedding for all utterances, we transformed our dataset into 2 dimensions with t-SNE (for visualization purposes). The following scatter plot shows all dataset utterances and intent names, utterances that has the same intent are colored with the same color:
This is how utterances and intent names align:
If we zoom in a bit, we see that intent names and corresponding utterances indeed align very well:
The dataset utterances creates rather a beautiful scatter plot, there are not many outliers and same intent’s utterances group together with the intent name.
What if we want to embed out utterances and intent names with BERT? That works great on utterances, but intent names are not real sentences and short expressions. BERT is trained for full sentences and may not work great for short expressions such as our intent names. In this case, we can create 768-dimensional embeddings for utterances by BERT and 100-dimensional embeddings for the intent names by word vectors. In order to embed those into the same space, we need to calculate a projection matrix φ that project 768-dim utterance vectors onto 100-dimensional intent vectors. Since we have annotated data, we can just learn the projection matrix via regression. (Though it’s important to include regularization to prevent overfitting). Then we have an alignment similar to the visuals above.
Siamese networks for zero-shot text classification
The exploratory data analysis above show us that semantic similarity between intent names and utterances is pretty obvious and learnable. Then we can ask a Siamese NN to learn the similarity between intent names and utterances.
Traditional intent classifiers input an utterance and generates a class label. Usually we encode the utterance either with an LSTM or BERT, then feed the encoded utterance into a Dense layer and get a class label:
Our zero-shot intent classifier instead learns if a label and an utterance is semantically similar. We used a Siamese network architecture, which is great for calculating semantic similarity. Our Siamese network inputs an intent name and an utterance; outputs if the inputs are related or not related.
The architecture is the same as I described in my previous Siamese NN article. The architecture consists of
an LSTM/BERT layer to encode the utterances and labels
followed by a distance layer to calculate the semantic distance between the utterance and the label
finally a Dense layer to squash the distance vector to a binary value
This architecture is still a text classifier, but this time number of inputs is two and dimension of output vector is only 1. The output is binary, 0 means the label and the utterance is not related; 1 indicates the utterance belongs to this label’s class.
We can ask Siamese classifier any label we want (even though the classifier never seen this label before) as long as we can provide a good embedding for the label. This is great for out-of-domain utterances because even if you don’t have any labelled data for ood class or just few examples, still the Siamese zero-shot classifier can decide if an utterance is relevant to Chris domain or something not really relevant to the domain.
Siamese networks are used for semantic similarity for quite along time, yet playing some tricks can carry us to an effortless zero-shot intent prediction model. Sometimes the idea is in front of you all the time, but you have to take a different perspective to see it.
In this article, we inherited a brand new approach to the “traditional” problem of intent classification. First we made a semantic grouping of Chris utterances. Then, we refreshed our knowledge about Siamese networks. Finally we saw how to use Siamese networks for zero-shot classification.
I hope all readers enjoyed our Chris’ capabilities and data and hope to meet you at more Chris articles. Until then stay tuned and healthy ❤️
Language Models are Few-Shot Learners, https://arxiv.org/abs/2005.14165
Are Pretrained Transformers Robust in Intent Classification?A Missing Ingredient in Evaluation of Out-of-Scope Intent Detection https://arxiv.org/pdf/2106.04564.pdf
Detecting Out-Of-Domain Utterances Addressed to a Virtual Personal Assistant https://www.microsoft.com/en-us/research/wp-content/uploads/2014/09/IS14-Orphan2.pdf
Using Semantic Similarity for Multi-Label Zero-ShotClassification of Text Documents , https://www.esann.org/sites/default/files/proceedings/legacy/es2016-174.pdf