Zero-Shot Learning and Clustering for Semantic Utterance Classification
Authors: Yann N. Dauphin, Gokhan Tur, Dilek Hakkani-Tur, Larry Heck
ICLR 2014

Abstract: We propose two novel zero-shot learning methods for semantic utterance classification (SUC) using deep learning. Both approaches rely on learning deep semantic embeddings from a large amount of Query Click Log data obtained from a search engine. Traditional semantic utterance classification systems require large amounts of labelled data, whereas our proposed methods make use of the structure of the task to allow classification without labeled data. We also develop a zero-shot semantic clustering algorithm for extracting discriminative features for supervised semantic utterance classification systems. We demonstrate the effectiveness of the zero-shot semantic learning algorithm on the SUC dataset collected by cite{DCN}. Furthermore, we show that extracting features using zero-shot semantic clustering for a linear SVM reaches state-of-the-art result on that dataset.


Peer review:
[Review]:
This paper offers 2 contributions: one confirms that zero-shot learning has some practical use for semantic classification. The second one, about zero-shot clustering is much more original, but unfortunately less mature.
When using deep learning for sentence or document-level classification, it has been observed that discriminantly tuning the word embedding significantly improved performance.
This paper does such discriminant tuning *without* labeled data, by assuming that the classifier has been obtained through zero-shot learning. They call the method 'zero-shot clustering', and I find it very neat and original.

Unfortunately, this paper seems to have been hastily written . Explanations  are thought-provoking but very idiosyncratic, thus very hard to follow. Experiments  are limited and poorly explained, especially about zero-shot clustering.

Section 5: 
Zero-shot learning is introduced in section 5, but I had to get back to the ' Zero-Shot Learning with Semantic Output Codes' paper to understand the idea in its full generality. In particular, it would be useful to formalize the 'intuition' behind equation (2), which corresponds to the 'knowledge base'. 

Section 6:
It  is the most interesting: it proposes an excellent, and to my knowledge, novel idea, that I understood as soon as I saw equation (3). However, the 3 paragraphs of explanation that precede are so confusing that I nearly suspected deliberate obfuscation of a good idea. The discussion starts by assuming we want to build density  model of the data P(X) like in auto-encoder, and then show this is a bad idea: why bother? What is proposed here has nothing to do with density estimation. This proxy framework is completely cryptic to me: proxy of what? P(C|X)?

What has this to do with the choice of the entropy (excellent by the way)?
The link seems to be in the sentence:
“The better the proxy function hat{f} the better this measure (H(f(X)) - H( hat{f}(X))^2<= K*(f(X) - hat{f}(X))^2 by Lipschitz continuity).”
What this sentence tells us is that we should get P(C|X) as close as possible to the true posterior to get its entropy close to the true entropy? But we are doing the opposite here: minimizing the estimated entropy, which does not even have to be close to the true entropy.

Section 7: experiments
The part about how zero shot clustering improves SVM classification is very frustrating to read: results are so promising, but very few details are shared (table 3).
-	What are the raw features? N-grams?
-	Why only SVMs are tried on the DNN and ZSC embeddings? It would make sense to try DNNs or DCNs.
-	An interesting further experiment would be if discriminant fine tuning of the embedding further improves performance over ZSC. In this case, ZSC training would be comparable to semi-supervised training, with a mixture of labelled and unlabeled examples.