Title: Short and Deep: Sketching and Neural Networks
Authors: Amit Daniely
Nevena Lazic
Yoram Singer
Kunal Talwar
Venue: ICLR 2017
Keywords: Theory

Abstract: Data-independent methods for dimensionality reduction such as random projections, sketches, and feature hashing have become increasingly popular in recent years.  These methods often seek to reduce dimensionality while preserving the hypothesis class, resulting in inherent lower bounds on the size of projected data.  For example, preserving linear separability requires $\Omega(1/\gamma^2)$ dimensions, where $\gamma$ is the margin, and in the case of polynomial functions, the number of required dimensions has an exponential dependence on the polynomial degree.
                                                                                                                                                            
Despite these limitations, we show that the dimensionality can be reduced further while maintaining performance guarantees, using improper learning with a slightly larger hypothesis class. In particular, we show that any sparse polynomial function of a sparse binary vector can be computed from a compact sketch by a single-layer neural network, where the sketch size has a logarithmic dependence on the polynomial degree.
                                                                                                                                                           
A practical consequence is that networks trained on sketched data are compact, and therefore suitable for settings with memory and power constraints. We empirically show that our approach leads to networks with fewer parameters than related methods such as feature hashing, at equal or better performance.

TL;DR: For sparse boolean inputs, Neural Networks operating on very short sketches can provably and empirically represent a large class of functions.

Review discussion:
Overall, I like the paper and the ideas.
The paper is about a better dimensionality reduction techniques for sparse binary data using count sketches. The nice thing about sketching is the possibility of reconstruction, and hence any function approximation can be build using NN (as NN are universal approximator on a reconstructed vector). The arguments in the paper can be simplified for general audiences deferring the math. You can add the equation for gradient descent update (of sketches) for wider appeal.
The experiments part can be made much better. Only showing why this is approach is superior over Gaussian projections on sparse binary datasets is not quite enough. It is known that for sparse binary datasets, minwise hashing and its variants are much better than random projections (and even space optimal for binary intersection).  See a series of papers.
1) Hashing algorithms for large scale learning NIPS 2011
2) Fast near-neighbor search for high dimensional binary data ECML 2012
3) In Defense of MinHash over SimHash AISTATS 2014
4) Is Min-Wise Hashing Optimal for Summarizing Set Intersection? PODS 14
4) On b-bit min-wise hashing for large-scale regression and classification with sparse data. Arxiv 2016
Of course, the reconstruction and polynomial approximation with minwise hashing are not clear.
So there is a good argument for sketching and hence the paper has advantages on its own.
However, the experiments to show that Gaussian projections are the best benchmark, for sparse datasets, is little obsolete.
We thank the reviewer for the constructive comments and for the references to the min-hashing papers. We plan to discuss these points and related work in the updated version.
Min-hashing is known to work well for similarity detection when the similarity function of interest is the Jaccard distance. The b-bit min hashing used in citation (1) above shows that they work also for linear regression problems. As pointed out by the authors of (1), it does not work well as a dimension reduction technique since the dimension of the final projection is 2^b k. Since b needs to be about 8 to 10 in order to get good performance, the experiments in (1) end up using, e.g. a projected dimension of 51k for the webspam dataset.
Additionally, as the reviewer points out, it is not clear whether minwise hashing allows for reconstruction or for polynomial approximation. The limited results in (5) on the polynomial case have an exponential dependence on the degree just like other previous work. We view one of the main contributions of our work to be the drastic provable reduction in sketch size in the polynomial case, that results from the fact that we use improper learning.
In light of the above, we believe our approach has significant benefits over b-bit min-hash. In the next revision, we will also add an experimental comparison to it.
We performed additional experiments comparing sketching the input to using as a MinHash as an input layer. We generated synthetic datasets for sparse linear and polynomial regression using settings similar to those described in Section 6. Denote by k be the number of non-zero features per example. We reduced dimensionality using the following settings:
Sketches: 2^3 blocks of size 2^3 * k
MinHash: 2^3 * k blocks of size 2^3 (3 bits)
This resulted in the two representations having equal size (2^6*k) and equal number of nonzero entries (2^3*k).  We used an efficient implementation of MinHash using hash functions, as described in https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/a13-li.pdf. We used 4-universal hash functions for both MinHash and Sketches. Note that MinHash is more expensive to compute, as it scales as k^2, whereas Sketches are linear in k.  We found that Sketches substantially outperform MinHash.
On the linear regression task (k=50, s1=50, |I|=50, 25 “noise” features per example, d=10K), MinHash mean squared error (MSE) was 1.39 (0.35), whereas the MSE with Sketches was 0.23 (0.05), over 50 generated datasets.
On the polynomial regression task (k=100, s2=25, s3=25, |I|=50, 50 noise features per example), the MinHash MSE was 0.28 (0.42) and the Sketches MSE was 0.056 (0.014).
We plan to report a more exhaustive comparison in the next version of the paper.
The paper presents some nice ideas on learning over sketched data.
- Corr3.2 is stated for a fixed x and w. Th.4.1 and Th 4.2 are stated uniformly for all x , I would expect a dependency of m and t on
k log(d/k) to be uniform on B_{d,k}. same thing for w . or the statement in Th 4.1 etc  should be like corr 3.2 , meaning not for all x and all w?
- Sparse linear function: If I understand correctly matrix V is a binary matrix with fixed weights and is not learned, then there is a weight w , that is learned, under a regression or classification cost.
so it seems the paper learns a linear function and not a neural network. N_{Relu} is a deterministic implementation of D^{AND}? or is V learned also in the experiments by back propagation?
Many citations on work on random projections and polynomial kernel approximations are missing and worth the comparison theoretically and practically for instance:
*Random Projections for Linear Support Vector Machines
https://arxiv.org/pdf/1211.6085v5.pdf
* Spherical Random Features for Polynomial Kernels
https://papers.nips.cc/paper/5943-spherical-random-features-for-polynomial-kernels.pdf
- Experiments:  would be interesting to try other datasets than AG news , such as Amazon reviews, to see how m and t depends on the training size and vocabulary size (dimension) in an NLP application.
also might be interesting to have some bounds  for m and t taking in account training size and vocabulary size.
- Minor comments:
proper learning and improper learning are not defined in a clear way in the paper.
We thank the reviewer for the constructive comments and for the references to the papers on fast random projections and spherical random features. We will discuss these works in the updated version, and also clarify the terms proper and improper learning
All our results, much like all results on projections in learning that we cite, are for any single fixed w. Corollary 3. 2 talks about a single x at a time. Thm 4. 1 and 4. 2 need the “uniformity” in the sense that the same neural network parameters work simultaneously for (almost) all x. The reviewer is correct that uniform guarantees should be possible with k log d sized sketches. For our requirement, we need not just a small sketch, but also a decoding step that can be implemented by a low depth neural network.
To our knowledge, the known algorithms in the compressed sensing literature that give for all guarantees with sparse sketching matrices, such as EMP, SMP and l_1 minimization, require larger depth, as they are iterative. In appendix C in our paper, we present a deterministic sketch that gives a for all guarantee. For the simpler sketch, a for-all type guarantee is not true unless we increase the sketch size to about k^2.
In both the sparse linear functions case and the sparse polynomial functions case, we learn all parameters including the matrix V. This is necessary because we do not a priori know the support of w, and hence do not know which bits need to be decoded. Decoding all bits would negate the advantages of projection, which is why we need to learn V.
Sketching is a powerful tool that has found application in a number of classical machine learning methods like least squares, kernel learning and tensor methods. It is very exciting that such application can be extended to deep neural networks, which represent the state-of-the-art for numerous learning tasks.
However, I have doubts on some aspects of the experimental and analytical settings adopted in this paper, which (in my opinion) might limit its impact in both theoretical understanding and practical applicability.
1. On the theoretical side, I found the sparse linear/polynomial classifiers assumption very troublesome. If the classifier that one wishes to learn is indeed sparse linear or sparse polynomial, why should neural networks be a good option compared to, say, Lasso or sparse logistic regression models, which are theoretically sound and even optimal for the particular function class to be learnt?
I think a more relevant setting is where one wants to learn a feedforward neural network, with perhaps sparse weights on each of its layers. It is an interesting question of whether dimensionality reduction preserves the richness of such sparse weight networks, and how aggressively one can reduce the dimension into (i.e., t in the paper), which leads to much smaller parameter space.
Also, from a learning theoretical perspective, the existence of a small neural network that approximates a sparse linear/polynomial classifier does not mean such a network can be effectively learnt. It would be good that some VC dimension /generalization aspects are analyzed as well.
For example, intuitively network that operates on sketched inputs should have smaller model complexity. Does it lead to improved generalization, and how such improvement should be traded off for the richness of the network (which desires *larger* sketch length)?
2. On the practical side, it seems the most relevant application would be in NLP where inputs are naturally sparse high-dimensional vectors. Reducing the input dimension indeed reduces the size of the model and at least save storage cost.
However, in practice the high dimensionality is typically dealt with using a LookupTable structure (encoding), which is not difficult to train at all, because the gradient for each data point only involves one entry in the LookupTable. I think the experiments should include comparison with such traditional approaches and comment on both accuracy and training/testing time.
Also, if the argument is that the vocabulary size is so large that LookupTable cannot be stored in GPU memory, such an example should be demonstrated and show how sketching helps.
The theoretical contribution of our paper is to show that neural networks can learn a sparse linear or polynomial functions from smaller sketches than Lasso can.
For this reason we sometimes refer to our approach as improper learning.
While the data-generating model may attain the best performance on the original training data, it is not necessarily the best choice for sketched inputs, as we also demonstrate empirically in Fig 2 (center). As the reviewer points out, the existence of a network does not convey anything about how difficult it is to learn such a network. Our empirical results are positive, but we provide no theoretical guarantees beyond the formal reconstruction guarantees. We do not understand the comment on generalization as the results of, for instance Bartlett, are applicable in our setting as well.
Our experiments do include an NLP task with a very large vocabulary (entity type tagging), and we do in fact compare our approach to using the original features (mapped to 32bit integers and hashed). We demonstrate that sketches can achieve comparable accuracy, with significant savings in the number of parameters (see Figure 3 right). Training sketches may take slightly longer than original data, since the number of non-zero features per example is increased by a factor of #blocks.
Summary of the paper:
The paper introduces sketches that approximates linear and sparse polynomials on binary data. The paper shows that such sketches can be represented as a one layer neural network. Experiments are conducted on some language processing tasks.
Novelty:
Approximation for linear functions and polynomial have been studied in previous work, hashing here is learned as a neural network which is novel.
Although I have some reserves on the connexion between the theory presented and the learning part in the paper (Please see comments below).
The main concern is that the  single layer neural network that is  learned  is not restricted to the class of  hash functions studied, hence it is hard to know if we are really testing the hashing class or just a one layer neural network.
Clarity:
The paper is  written in a clear way.
Comments :
- The link between neural network and the hashing stems from the implementation of the 'and operation' with a relu and a binary weight V and a bias 1-t. V encodes the support of the linear weight for instance in the linear model.
When Learning V it is regularized with l_1 norm to encourage sparsity, nevertheless the weights are not restricted to be positive to be faithful to the theory introduced in the paper.
When learning V by back propagation when can try projected gradient to restrict V values to be in [0,1]. On a synthetic example where the support of w is known it would be interesting to see if V recovers the support of w.
- Similarly  for polynomial seems group sparsity is a good regularizer, difficulty being here that the groups are unknown. Many works have tackled that issue see for instance http://www.di.ens.fr/~fbach/IEEE_TSP_shervashidze_bach_2015.pdf
- The experiments are limited to small sets.
We would like to emphasize that our main results pertain to existence of neural networks that can implement a “sketch decoding” function. The particular network we describe is just one way of implementing such a function, and there exist other possibilities, including networks with negative weights (and even different non-linearities).
Thus we did not consider it necessary to explicitly enforce non-negativity and block-sparsity. This setting is called improper learning as first described by Valiant over 20 years ago in his seminal on PAC learnability. We hope this clarifies our experimental setting.
Thank you for your comment.
The one layer neural network space may represent functions outside the functions induced by the hashing studied in the paper.
Hence if we don't restrict to this class, we don't know if we are really evaluating this class or not. We might be learning a function outside the function class we are studying. This is my main concern with the empirical evaluation. I think the paper will be more coherent if those issues are addressed.
Otherwise another analysis is needed for the type of learning done in the experiments, using the similarity induced by the hashing between two points, and see how it approximates the original function space.
The reviewer seems to be pointing to the improper learning aspect of our work. Recall that improper learning is the idea of learning a function f from a hypothesis class H by using a function g from a larger hypothesis class H’. This approach is often used to make learning more efficient computationally. From a learning point of view, the primary motivation for restricting f to be from H in the first place is to get good generalization, and using an H’ that is not “much larger” suffices for this purpose. It is very rarely the case that g not being in H is an issue. Most previous applications of improper learning allow us to get around NP-hardness results and get polynomial time learning. In our setting, we use it to get faster running times.
In our experiments, we could have added more constraints to make our H’ even smaller as the reviewer suggests. We view the fact that we did not have to do that as a feature: the simplicity allows us to deploy our sketching approach with little changes to modern learning infrastructures. Our theorems hold for exactly the setting that we use in the experiments: the theorems show that H’ contains (good approximations to) every f in H. They do not require that H’ does not contain anything else.
Summary: This paper presents a novel sketching method in solving sparse polynomial function of a sparse binary vector within a single-layer neural network.
Pros:1. The paper is written clearly.
2. I like the idea of using improper learning to reduce the dimensionality.
3. It's novel to use neural network to do sketching decoding.
Cons:1. The experiments part is kind of weak in comparison with other sketching/projection methods.
2. Any theoretical analysis to show why this approach needs fewer paramerters than feature hash, etc. My other concern is that since t and m are tunable, how to set these parameter for different problems?
3. The authors claim they have faster training.
However it's not shown in experiments.
Comments: It may be interesting to compare with sparse feed-forward neural networks.

Meta-review:
role: PC
data: comment: The reviewers present a detailed set of concerns regarding the paper. In particular, the paper lacks comparison to other sketching works. The sketches used in the paper are rudimentary and in practice, there are more sophisticated sketches employed.
 
 Sketching has a different goal from function approximation. Sketching aims to reconstruct (with a certain error). The paper uses sketching in a naive manner, in the sense that the error due to the sketch is incorporated into the overall error. But presumably, networks can be compressed without achieving reconstruction on each instance, while maintaining a guarantee on the overall accuracy. I would find such a framework to be more interesting and practically relevant.
 
 On the other hand, it is important for the ICLR community to be exposed to sketching frameworks. Hence, I would like to invite the paper to be presented in the workshop, in the hope that it can spur further ideas.

Decision: Invite to Workshop Track