Zero-Shot Learning and Clustering for Semantic Utterance Classification
Authors: Yann N. Dauphin, Gokhan Tur, Dilek Hakkani-Tur, Larry Heck
ICLR 2014

Abstract: We propose two novel zero-shot learning methods for semantic utterance classification (SUC) using deep learning. Both approaches rely on learning deep semantic embeddings from a large amount of Query Click Log data obtained from a search engine. Traditional semantic utterance classification systems require large amounts of labelled data, whereas our proposed methods make use of the structure of the task to allow classification without labeled data. We also develop a zero-shot semantic clustering algorithm for extracting discriminative features for supervised semantic utterance classification systems. We demonstrate the effectiveness of the zero-shot semantic learning algorithm on the SUC dataset collected by cite{DCN}. Furthermore, we show that extracting features using zero-shot semantic clustering for a linear SVM reaches state-of-the-art result on that dataset.


Peer review:
[Review]:
The paper considers the task of categorizing queries into classes revealing user intent (e.g., weather vs. flight booking). In doing this, the authors exploit query log data (specifically, queries paired with URLs clicked after sending the query to a search engine), however they use no or little supervised data. Instead,  they learn query representations predictive of URLs, and use proximity between a query representation and a category name representation to make the classification decision (word(s) in the category name are also treated as a query).  Additionally they consider a regularizer which favors low entropy predictions on unlabelled queries (with a distance-based probability model). Finally, they also use their representation in supervised learning. 

I find the ideas and their implementation interesting and the experiments seem fairly convincing to me as well (however, see (2)).

Comments:
1) The type of regularisation used is similar to entropy minimization  proposed in Grandvalet and Bengio (2004), and can also be regarded as a form of posterior regularization (Ganchev et al., 2010) or generalized expectation criteria (Druck et al., 2008). I think the authors should briefly discuss the relation.
2) The dataset used for testing query classification does not seem to be very standard and actually consists of utterances by users of a spoken dialog system rather than search queries. I am wondering why a more standard dataset is not selected (e.g., KDDCUP).
3) Not being an expert in query classification, I would appreciate some discussion of related approaches, as I assume this is not the first method which considers a semi-supervised approach to this task. 
4) Section 5: 'the best class name would have a meaning P(H | C_r) that is the mean of the meaning of all its utterances  E_{X_r | C_r}{ P(H | X_r)} '. I am not sure in what sense it would be the best. If in terms of classification accuracy then this is not entirely correct -- e.g., a single datapoint can move the mean to an arbitrary position and dramatically affect the error rate. I understand what the authors are trying to say here but I think it is a bit vague.
5) The authors limit themselves to using only 1,000 most frequent  URLs when learning the representations, perhaps because they use the sofmax error function. They might consider using 'standard' techniques which avoid summation over the categories (i.e. URLs)  in training: some form of binarization (e.g., Huffman trees) or a ranking error function with sub-sampled negative examples. 

The paper would also benefit from some polishing, a couple of points:
-- Section 6 (page 5, 1st par): 'However, it is not clear how of a proxy that is for a given task.' ?? 
-- caption Fig 3: sentence 'ZSL achieves …' does not seem grammatical

Overall, I have a feeling that the authors tend to over-generalize -- it is a nice paper about query classification, and this over-generalization makes it less readable (e.g, see section 6 where 'proxy functions' are discussed). 

Pro:
-- a clever application of distributed representation learning for an important task
-- the approach may have applications in other domains (e.g., in opinion summarization -- learning to categorize online review sentences according to product features)
Con:
-- the dataset may not be entirely appropriate 
-- discussion of related work does not seem quite adequate 
-- writing (minor)