Zero-Shot Learning and Clustering for Semantic Utterance Classification
Authors: Yann N. Dauphin, Gokhan Tur, Dilek Hakkani-Tur, Larry Heck
ICLR 2014

Abstract: We propose two novel zero-shot learning methods for semantic utterance classification (SUC) using deep learning. Both approaches rely on learning deep semantic embeddings from a large amount of Query Click Log data obtained from a search engine. Traditional semantic utterance classification systems require large amounts of labelled data, whereas our proposed methods make use of the structure of the task to allow classification without labeled data. We also develop a zero-shot semantic clustering algorithm for extracting discriminative features for supervised semantic utterance classification systems. We demonstrate the effectiveness of the zero-shot semantic learning algorithm on the SUC dataset collected by cite{DCN}. Furthermore, we show that extracting features using zero-shot semantic clustering for a linear SVM reaches state-of-the-art result on that dataset.


Peer review:
[Review]:
This paper introduces a method to classify search queries into a set of classes in an utterance frame classification task.

The main proposed method is to learn from query click logs a representation of words into an embedding space.

The paper has one huge problem which is its lack of comparison to the most obvious baselines which are other word embedding models.

For example, the table 2 of the embedding nearest neighbors looks very similar to what neural network language models would learn. In fact, the paper mentions that the proposed model is also very similar to such neural network language models (NNLM).

Given that the model is very similar to NNLMs like those of Bengio, Collobert and Weston, Huang et al., Mikolov et al., etc it would seem crucial to compare to these existing methods to know whether this model improves over existing literature.

Unlike NNLMs which are truly unsupervised and can be trained on easily accessible abundant large text corpora like wikipedia, the paper instead proposes to use a proprietary dataset that is not and will not be available to anybody to learn essentially the same type of embeddings.

It also seems like, if the click log data includes words like restaurant that are similar to the SUC class names, the task becomes essentially supervised. Even if not, the tasks are related and I would call this approach more of a distant supervision type approach and not zero shot learning since users clicking on semantically important URLs is some type of supervision (albeit one impossible to obtain for anybody except the big search engines).

Frome et al. and Socher et al. both had a NIPS paper last year that also used deep methods for zero shot learning, using embeddings that were learned in an unsupervised way, mention or comparison to these projects would be reasonable.

The terms/ideas of zero-shot clustering seem confusing. Clustering is, by its usual definition, always unsupervised and hence 'zero-shot'. In fact, the point of zero shot learning is that one can do a usually supervised task but without supervision of the classes that are to be predicted.
When I assign an element to a cluster in k-means, am I doing zero-shot clustering in the author's view?

The improvement to a simple existing kernel based method in table 3 is tiny (0.2%) improvement.

Minor comments:
- 'models [15] who learn' -> models which learn
- Table 1 is in an odd place that is way too early since it's referenced only a few pages later.
- it would be better if citations mentioned the names of the authors so the reader familiar with the field doesnt have to go back and forth between the text and references to know what paper is being cited.
- citation 25 just mentions ICML and has no title or authors
- fig.3 is not readable in a black and white printout
- 'to the the 1000 most popular' 


Conclusion:
The merit of the paper is highly questionable without a comparison to similar models that learned word embeddings with just raw text. Several word embeddings are available for download and I encourage the authors to pick one and update their paper on arxiv with an added comparison.