Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Did you also explore increasing the dimensionality of embeddings (e.g., doubling)? I.e., with 8x compression, can you achieve a model that is both smaller and noticeably more accurate?
review_title: Question

final_score: rating: null
review_content: It is well-known that non-deep method still performs well on the text classification task (particularly for dataset with few classes as in Section 4.1). Could the authors also report results on non-deep baselines, such as a simple bag-of-words + SVM with polynomial kernel of degree 2 or 3 (thus emulating bigrams and trigrams)? How does this kind of baseline compare to fastText.zip in terms of accuracy and model size? I believe that such baseline models can be compressed using simple tricks as well.
review_title: comparison to non-deep baselines

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper describes how to approximate the FastText approach such that its memory footprint is reduced by several orders of magnitude, while preserving its classification accuracy. The original FastText approach was based on a linear classifier on top of bag-of-words embeddings. This type of method is extremely fast to train and test, but the model size can be quite large.

This paper focuses on approximating the original approach with lossy compression techniques. Namely, the embeddings and classifier matrices A and B are compressed with Product Quantization, and an aggressive dictionary pruning is carried out. Experiments on various datasets (either with small or large number of classes) are conducted to tune the parameters and demonstrate the effectiveness of the approach. With a negligible loss in classification accuracy, an important reduction in term of model size (memory footprint) can be achieved, in the order of 100~1000 folds compared to the original size.

The paper is well written overall. The goal is clearly defined and well carried out, as well as the experiments. Different options for compressing the model data are evaluated and compared (e.g. PQ vs LSH), which is also interesting. Nevertheless the paper does not propose by itself any novel idea for text classification. It just focuses on adapting existing lossy compression techniques, which is not necessarily a problem. Specifically, it introduces:
  - a straightforward variant of PQ for unnormalized vectors,
  - dictionary pruning is cast as a set covering problem (which is NP-hard), but a greedy approach is shown to yield excellent results nonetheless,
  - hashing tricks and bloom filter are simply borrowed from previous papers.

These techniques are quite generic and could as well be used in other works. 


Here are some minor problems with the paper:

  - it is not made clear how the full model size is computed. What is exactly in the model? Which proportion of the full size do the A and B matrices, the dictionary, and the rest, account for? It is hard to follow where is the size bottleneck, which also seems to depend on the target application (i.e. small or large number of test classes). It would have been nice to provide a formula to calculate the total model size as a function of all parameters (k,b for PQ and K for dictionary, number of classes).
  
  - some parts lack clarity. For instance, the greedy approach to prune the dictionary is exposed in less than 4 lines (top of page 5), though it is far from being straightforward. Likewise, it is not clear why the binary search used for the hashing trick would introduce an overhead of a few hundreds of KB.
  

Overall this looks like a solid work, but with potentially limited impact research-wise.

review_title: Lossy compression techniques applied to FastText with nice results

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper presents a few tricks to compress a wide and shallow text classification model based on n-gram features. These tricks include (1) using (optimized) product quantization to compress embedding weights (2) pruning some of the vocabulary elements (3) hashing to reduce the storage of the vocabulary (this is a minor component of the paper). The paper focuses on models with very large vocabularies and shows a reduction in the size of the models at a relatively minor reduction of the accuracy.

The problem of compressing neural models is important and interesting. The methods section of the paper is well written with good high level comments and references. However, the machine learning contributions of the paper are marginal to me. The experiments are not too convincing mainly focusing on benchmarks that are not commonly used. The implications of the paper on the state-of-the-art RNN text classification models is unclear.

The use of (optimized) product quantization for approximating inner product is not particularly novel. Previous work also considered doing this. Most of the reduction in the model sizes comes from pruning vocabulary elements. The method proposed for pruning vocabulary elements is simply based on the assumption that embeddings with larger L2 norm are more important. A coverage heuristic is taken into account too. From a machine learning point of view, the proper baseline to solve this problem is to have a set of (relaxed) binary coefficients for each embedding vector and learn the coefficients jointly with the weights. An L1 regularizer on the coefficients can be used to encourage sparsity. From a practical point of view, I believe an important baseline is missing: what if one simply uses fewer vocabulary elements (e.g based on subword units - see https://arxiv.org/pdf/1508.07909.pdf) and retrain a smaller models?

Given the lack of novelty and the missing baselines, I believe the paper in its current form is not ready for publication at ICLR.

More comments:
- The title does not make it clear that the paper focuses on wide and shallow text classification models. Please revise the title.
- The paper cites an ArXiv manuscript by Carreira-Perpinan and Alizadeh (2016) several times, which has the same title as the submitted paper. Please make the paper self-contained and include any supplementary material in the appendix.
- In Fig 2 does the square mark PQ or OPQ? The paper does not distinguish OPQ and PQ properly at multiple places especially in the experiments.
- The paper argues the wide and shallow models are the state of the art in small datasets. Is this really correct? What about transfer learning?

review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a series of tricks for compressing fast (linear) text classification models. The paper is clearly written, and the results are quite strong. The main compression is achieved via product quantization, a technique which has been explored in other applications within the neural network model compression literature. In addition to the Gong et al. work which was cited, it would be worth mentioning Quantized Convolutional Neural Networks for Mobile Devices (CVPR 2016, https://arxiv.org/pdf/1512.06473v3.pdf), which similarly incorporates fine tuning to mitigate losses due to quantization error.

As such, one criticism of the paper is that it is a more-or-less straightforward application of techniques that have already been shown to be effective elsewhere in the model compression literature, and so isn't particularly surprising or deep from a technical perspective. However, this is as far as I am aware the first work applying these techniques to text classification, and the results are strong enough that I think it will be of interest to those working on models for text-based tasks.
review_title: Effective if simple combination of existing techniques for text-classifier compression

Meta-review: The submission describes a method for compressing shallow and wide text classification models. The paper is well written, but the proposed method is not particularly novel, as it's comprised of existing model compression techniques. Overall, the contributions are incremental and the potential impact seems rather limited.
Decision: Reject