SOLAR: Sparse Orthogonal Learned and Random Embeddings

Abstract: Dense embedding models are commonly deployed in commercial search engines, wherein all the document vectors are pre-computed, and near-neighbor search (NNS) is performed with the query vector to find relevant documents. However, the bottleneck of indexing a large number of dense vectors and performing an NNS hurts the query time and accuracy of these models. In this paper, we argue that high-dimensional and ultra-sparse embedding is a significantly superior alternative to dense low-dimensional embedding for both query efficiency and accuracy. Extreme sparsity eliminates the need for NNS by replacing them with simple lookups, while its high dimensionality ensures that the embeddings are informative even when sparse. However, learning extremely high dimensional embeddings leads to blow up in the model size. To make the training feasible, we propose a partitioning algorithm that learns such high dimensional embeddings across multiple GPUs without any communication. This is facilitated by our novel asymmetric mixture of Sparse, Orthogonal, Learned and Random (SOLAR) Embeddings. The label vectors are random, sparse, and near-orthogonal by design, while the query vectors are learned and sparse. We theoretically prove that our way of one-sided learning is equivalent to learning both query and label embeddings. With these unique properties, we can successfully train 500K dimensional SOLAR embeddings for the tasks of searching through 1.6M books and multi-label classification on the three largest public datasets. We achieve superior precision and recall compared to the respective state-of-the-art baselines for each task with up to 10 times faster speed.

Reviews and author responses:

The work describes an extreme classification strategy which leverages the computational efficiency of multiclass (and multilabel) efficiency on small label sets (circa 10K) composed with the near-orthogonality of random sparse embeddings; while exploiting inherent parallelism of repeated independent instantiations of this primitive technique to mitigate statistical issues.  

The use of fixed near-orthogonal label embeddings is elegantly motivated, the computational properties of the technique are favorable (especially, amenability to inverted indexing), and the statistical performance is competitive.

Inference only gets passing treatment in the entire exposition, without any supporting rationale.   For instance, this reviewer was surprised that inference involves summing over predicted probabilities, rather than summing logs of predicted probabilities.  Essentially there are a collection of K independent predictors that we are trying to ensemble, a problem that has received lots of attention in the literature.  Making these connections would both help the reader and also potentially improve the technique.

This submission addresses the problem of learning document embeddings for document retrieval/recommendation tasks. Such tasks are characterized by a large number of documents and a large set of semantic class labels. In contrast to the now standard approach of representing documents and their labels as dense low dimensional embeddings, this work proposes to use sparse high dimensional embeddings for documents/labels and claims that the proposed approach has certain advantages over state of the art:
	1. It results in memory efficient and load balanced inverted index that facilitates fast index based lookup for retrieval  
	2. It enables distributed training on disjoint parts of the label vector and thereby speeding up training.

Unlike the common approach of jointly learning the document and label embeddings, the proposed approach employs random binary embeddings for labels.  The label embeddings are high dimensional, sparse and mostly orthogonal by design. Each label in the training data is assigned a random K-sparse (only K > 0 bits are 1 and rest are 0) binary embedding independent of other labels as well as documents associated with the label and any semantic features (such as label text/title) the label might have. When they are sufficiently high dimensional, random embeddings ensure that the resulting inverted index is both balanced and  each bucket in the index has only a small number of labels. 

As the label embeddings are random and each bit in the embedding is independent of all other bits by design, the embeddings can be partitioned into multiple disjoint blocks and each block can be used to independently learn partitioned embeddings for the documents. As this requires no communication between the different threads working on different blocks, training can be very fast.  

To keep the model size low, the work proposes to use a simple feed-forward network with only one hidden layer. There is one such network for each block and as they are trained independently, there is no sharing among the models.  Also, unlike other approaches, the proposed work doesn't make use of pre-trained word embeddings for training document embeddings. Instead it uses a feature hashing scheme to map the very high dimensional text data to reasonably low dimension. This eliminates the need to store word embeddings in memory. As feature hashing is memory efficient, the only structures that need to be in memory are the parameters of the feed-forward network and the inverted index thereby achieving memory efficiency at query time.

The submission presents experimental results comparing the proposed approach with a few baselines. The results are promising in terms of retrieval efficacy on multiple data sets.

While the proposed approach is well engineered to achieve its twin goals of memory efficient and load balanced inverted index and distributed training, there are a few concerns about the methodology. 

First is about the rationale behind using random binary embeddings for labels. As observed earlier, such an embedding is independent of other labels as well as documents associated with the label and any semantic features (such as label text/title) the label might have. This seems to be a rather poor utilization of the rich semantic relatedness in the data for the sake of engineering gains. It can be argued that methods that ignore label correlations and other semantic relationships in the data are suboptimal.

Second concern is about the harmful consequences of random embeddings. As authors note in the appendix, unrelated labels fall into the same bucket. As each bucket has only a small number of labels, it is unlikely that a significant number of the labels assigned to a bucket are related. Therefore, it is unclear to this reviewer what does a bucket represent semantically other than being a collection of unrelated labels. Would this not affect the training of the classifier mapping documents to the bucket adversely? Empirical results seem to suggest otherwise, but this is an issue that needs both deeper theoretical and empirical investigation. Further, in the proposed approach, semantically related labels are likely to have dissimilar embeddings. This goes against the traditional view, particularly in semantic hashing, that similar labels should have similar embeddings. 

Third, authors claim that one-sided and two-sided learning are mathematically equivalent. This is true only when label embeddings are orthogonal. When the embeddings are low dimensional as in other methods, orthogonality of label embeddings is neither strictly guaranteed nor necessary. 

Fourth, though authors don't say it, K and B are hyper parameters in the proposed approach. The tuning of these hyper parameters does add a significant overhead to training which is not shown in the experiments. So the claim on significant training speed gains over baselines is suspect. 

Fifth, it is not clear whether the proposed work uses the same tokenization scheme as used by the authors of DSSM, viz Unigrams+Bigrams+Char Trigrams+OOV, in the experimental comparison with DSSM. As observed in DSSM paper, the choice of tokenization affects the retrieval accuracy significantly and a fair comparison would necessarily need to use the best tokenization scheme. 

Some suggested corrections:
	1. In figure 1, classifier 1 appears twice.
	2. In figures 1 and 3, k should be K.
	3. In the feature ablation study, the best performing setting is B = 30K, K = 32 (based on P@1, P@3, P@5 reported in Table 3) whereas the submission claims that B = 20K, K = 32  is the optimal setting.
	

---

Based on author feedback, here are some additional comments:
I would like to thank the authors for their response to the reviews. As noted in my original review, the proposed method is a well-engineered method for a particular type of document recommendation problem. Empirical performance on the chosen data sets is impressive compared to the baselines. The claim on gain in training speed is suspect as there is a hyper parameter tuning step that has been not taken into account while reporting the speed gain over the baselines. Label correlations are not taken into account for label embeddings and might hurt the performance when there are not many documents in the training data for the long tail of labels in many real-world applications. Random embedding puts unrelated labels into the same bucket. Though this doesn't seem to hurt retrieval performance in the experiments reported in the submission thanks to filtering, it is not clear how training will be affected by this non-semantic bucketing of labels. Overall, I think the submission has several things going in its favor though there is substantial scope for strengthening. 

I've updated the rating. 




Yes, we used the same tokenization (unigrams+bigrams+OOV) for both SOLAR and DSSM. And the 4% gain was with same featurization. Also, thank you again for spotting the incorrect order in figures 1 and 3. We have corrected them.

We thank all the reviewers for their insightful comments. We are glad to know that the reviewers find this work's rationale compelling/interesting [R1, R2, R3], the paper well motivated and written [R1,R2,R3], the methodology and properties useful [R1, R2, R4] and the results good [R1, R2, R4].

We have revised the paper and updated our initial responses to each review with more details. We kindly request you to read our clarifications and new additions.

Here is a synopsis of the revision.

1. [R1] Theoretical justification for adding probabilities: In the inference part of section3, we have added a derivation that shows that the original label probability grows linearly with the expected value of corresponding bucket probabilities, justifying our use of summation of probabilities.

2. [R1] In section 5, we added a table 2 comparing different scoring schemes like summing probabilities, log probabilities and logits.

3. [R2] Sensitivity to random seed: We experimented with 5 different sets of random seeds (each having K integers). Our mean P@1/3/5 for Amazon-670K is 34.18,32.49,32.47 with a standard deviation of (0.0145, 0.01,0.004) showing the robustness of SOLAR.

4. [R4] We added a new discussion called 'Filtering Noisy Labels' in section 3. This explains how we omit irrelevant labels that appear due to random pooling.

5. [R4] We added a new table 4 showing the candidate set size with frequency filtering and the improvement in precision and recall.

Thank you for your comments. We are glad that you appreciate the importance of the problem and the clarity/presentation of the paper. Please see the following clarifications:
 
**Q1. In abstract and introduction it is mentioned that "In this paper, we make a novel and unique argument that sparse high dimensional embeddings are superior to their dense low dimensional counterparts." This is simply not true.**

A1. We intend to convey that ultra sparse and orthogonal embeddings are superior than their dense counterparts. SNRM neither enforces orthogonality nor supports other indexing schemes. We will emphasize the orthogonality aspect in the opening sentence of 'Our Contributions'.

**Q2.1  The task they used for evaluating their model is pretty artificial. There is no such real-world task as "product to product recommendation" and the experimental setting is not realistic at all.**

A2.1 Complimentary/Supplementary product recommendation is a very important task for cross-selling and up-selling. The dataset that we used is a real co-purchase data.

**Q2.2  SNRM is the only paper with the same idea of learning high-dimensional sparse representation. However, the results reported in the table are strangely low. Looking at the SNRM paper (which was published in 2018, so it's not an old paper) we can see that it outperformed a lot of strong baselines on a number of benchmark.**

A2.2 We did vary SNRM's sparsity weight among (0.1,0.01,0.001). However, we still did not get good numbers. We will be adding our SNRM code (official code with a custom data generator). SNRM's load balance and precision issue has also been corroborated by another team working on a new method called alpha-SVS for passage retrieval on MS-MARCO dataset. They reported that SNRM coagulates into 1.33% of the buckets only and MRR@10 is 0.0.

**Q3. It is not clear how scalable the model is and actually I have some doubt about its scalability. What will happen when the number of items goes beyond tens of millions or even billions?**

A3. We are surprised with the concerns about SOLAR's scalability. We have comfortable 10x lead in training time over industry scale algorithms like SLICE, Parabel and DSSM and our algorithm is trivially parallelizable. We have shown results on three largest Extreme Classification datasets where many other leading embedding models have failed. And every algorithm is bound to have the same issue of scaling up when there are 100s of millions of a billion labels.

**Q5. It is not clear whether the proposed work uses the same tokenization scheme as used by the authors of DSSM, viz Unigrams+Bigrams+Char Trigrams+OOV, in the experimental comparison with DSSM. The choice of tokenization affects the retrieval accuracy significantly and a fair comparison would necessarily need to use the best tokenization scheme.**

A5. Thank you for the suggestion, we used unigrams+bigrams+OOV as our featurization. We did not use character trigrams. However, in a subsequent run of SOLAR, we did notice 0.39% gain in P@1 by using character trigrams. We did not have time to run all the baselines. But going by DSSM's reporting trends, we can expect DSSM to also be marginally better. Given our 4% lead over DSSM, we would still be comfortably ahead. And with a larger input dimension, we still use the same feature hashing for SOLAR while we DSSM's embedding layer grows in memory.

We've corrected k to K in figures 1 and 3 and also corrected the highlighting in ablation study table.

Thank you for the positive comments and the detailed review. Please see the following clarifications.
 
**Q1.  What is the rationale behind using random binary embeddings for labels? This seems to be a sub-optimal utilization of the rich semantic relatedness in the data.**

A1. We did experiment with MinHash based indexing for labels (using label titles). We also used pre-trained language models to extract sentence embeddings followed by Locality Sensitive Hashing. We encountered two problems, first is the obvious load balance problem and the second is that there were multiple labels with exactly same sparse representation (in the case of MinHash). And the P@1 was a good 4% lower than that with random orthogonal embeddings. Most real large-scale datasets have power law distribution where most labels belong to a few popular categories and only few labels belong to the rest of the categories. Using any label dependent embedding is bound to coagulate labels into few buckets and disturb the load balance. Random embeddings mitigate both the issues.

**Q2. As each bucket has only a small number of labels, it is unlikely that a significant number of the labels assigned to a bucket are related. Therefore, it is unclear to this reviewer what does a bucket represent semantically other than being a collection of unrelated labels. Would this not affect the training/inference of the classifier mapping documents to the bucket adversely?**

A2. Thank you for raising this important concern. We added a discussion 'Filtering Noisy Labels' in the inference part in section 3. We also added table 3 which shows how we omit unrelated labels during inference. The basic idea is that any label that appears less than a threshold d times (out of K) in the top-m buckets is a noisy irrelevant label. Table 3 empirically shows that number of active labels goes down exponentially as we increase d and the recall jumps from 25% without filtering to 34% with d=8. Beyond K/2, we end up with too few candidates and lose recall.

As far as training is concerned, we already had an explanation for the phenomenon of 'Positive Only Association' in Appendix B - "By choosing an ‘OR’ operation over the true labels, we enforce the model to reasonably fire up the bucket for all queries containing terms ‘waves’, ‘gravity’, ‘rabbit’ etc. This is called ‘positive-only association’. 
Each independent model is a feebly correct signal towards accurate prediction. Although a single bucket might correspond to unrelated labels, the labels that survive with high scores across K components would be truly related to the input."

**Q3. Authors claim that one-sided and two-sided learning are mathematically equivalent. This is true only when label embeddings are orthogonal. When the embeddings are low dimensional as in other methods, orthogonality of label embeddings is neither strictly guaranteed nor necessary.**

A3. It is true that our theorem only holds when the embeddings are orthogonal. And orthogonality is not  sufficient for dense embeddings. However, works like Spread Out Regularization[1], GLaS [2] use 'Global orthogonal regularization' which enforce labels to be well separated in the space so that there is good distinguishability and faster convergence. Prior works demonstrate the empirical necessity of orthogonality and we show a theoretical and empirical advantage of orthogonality.

**Q4. K and B are hyper parameters in the proposed approach. The tuning of these hyper parameters does add a significant overhead to training which is not shown in the experiments.**

A4. Tuning hyperparameters is integral to any neural network training and in some Vision tasks, even a minor tweak to learning rate warrants training from scratch again. Table 6 in the ablation study part shows how we tuned our hyperparameters. In practice, across a range of datasets (labels > 500K), we realize that K=16 and B=30000 is a good choice. The gains from K=16 to 32 are negligible and so is the case from B=30K to B=40K. Any embedding model has to choose an appropriate dimension through experiments. And SOLAR is also robust to learning rates and each individual model is sufficiently regularized due to random pooling. Hence, SOLAR is as sensitive or even less to hyperparameters than other methods.

Thank you very much for the positive comments. We are glad that you appreciate our idea, presentation and results. Please see the following clarifications.
 
**Q1. Why are the baselines in Table 1 trained for only 5 epochs and not until convergence. Can the performance of the baselines be improved by training more?**

A1. We re-trained DSSM till 10 epochs. And the metrics saturate at 7 epochs. The best numbers we could get was P@1/5/10 = (32.09, 27.93, 24.78) as opposed to the previous (31.34, 27.55, 24.41). And the training time increased by a good 10 hrs (from 25.27 hrs for 5 epochs  to ~35 hrs for 7 epochs). SOLAR only takes 2.65 hrs for 10 epochs and gets P@1/5/10 (35.24,29.71,26.98). 

**Q2. I could not find information about the compute used during inference for the baselines and SOLAR. Can the inference speed of the baselines and SOLAR be improved by simply using more compute?**

A2. In the paper, we did mention that "the evaluation for DSSM was totally done on GPU while SOLAR spends most of its time on CPU". We are clarifying it further. The inference process for baselines was all done on a V-100 GPU with 32GB RAM (including the cosine similarity computation and nearest neighbor). While we could afford that for 1.67M labels, we would not be able to do that beyond 5 million labels. That is where SOLAR would be very advantageous. As for SOLAR, we used 24 parallel CPU cores with C++ OMP parallelism (or python multiprocessing) to aggregate and sort candidates.

**Q3. How sensitive is SOLAR with respect to the random seeds chosen for the K models? In other words, how much would the performance vary if we start with a different set of random seeds?**

A3. We re-trained the model 5 times for Amazon-670K dataset with K=32 and B=20K, each time picking K random integers in the range of 1000 as random seeds for the label initialization. For m=50, the mean for P@1/3/5 is (34.18,32.49,32.47) with standard deviation (0.0145, 0.01,0.004). This shows that SOLAR is very robust to the choice of random seed. SOLAR is and ensemble of weak random models which is sufficiently regularized and hence very robust.




Thank you very much for the positive comments, appreciation for our motivation and computational properties. 
 
We wish to clarify that the tasks that we deal with have classes ranging from 500K-3M. The 10K buckets that we refer to are the embedding dimensions in each of the independent instantiations.
 
**Q1. This reviewer was surprised that inference involves summing over predicted probabilities, rather than summing logs of predicted probabilities. Essentially there are a collection of K independent predictors that we are trying to ensemble, a problem that has received lots of attention in the literature. Making these connections would both help the reader and also potentially improve the technique**

A1. Thank you for bringing this interesting point. We added a discussion 'Alternate Scoring Methods' in the inference part of section 3. We also added a theoretical derivation that shows the original label probability can be estimated from the expected value of predicted bucket probabilities.  This justifies our use of sum of probabilities.

However, we also added table 2 that compares the precision of summing probabilities, logits and log-probabilities. While theory shows that summing probabilities is an appropriate scheme, empirical evidence points to summing logits being slightly better. It could be due to the fact that logits have wide range (all real values) and they are more expressive then probabilities. We are investigating this interesting phenomenon. We hope to come up with better estimators in the future.

The paper studies the problem of document retrieval using embedding based models. It argues that performing near-neighbour search on a large number of dense embeddings hurts performance and accuracy. As an alternative, the paper proposes SOLAR (SPARSE ORTHOGONAL LEARNED AND RANDOM EMBEDDINGS), a model which uses high-dimensional, ultra sparse embeddings, on which near-neighbour search can be done using simple lookup operations. In SOLAR the document labels are divided into equal chunks of sparse vector, and independent models are learned for mapping the query to each chunk. Hence, SOLAR could be trained on multiple GPUs in an embarrassingly parallel way without requiring any communication between the GPUs. The paper demonstrates the effectiveness of SOLAR by comparing it against strong baselines on various recommendation and extreme classification datasets. It also provides theoretical justification for SOLAR by showing how “one-sided” learning (i.e. fixing label embedding but learning mapping from query to label) is mathematically equivalent to “two-sided” learning (i.e. jointly mapping the label and query to a common space).

#### Strong points: 
- **Well-motivated problem:** The paper does a very good job in motivating the problem, and highlighting the potential issue with dense embedding based models for large scale problems.
- **Interesting result:** The paper argues the current convention for document retrieval is of using low dimensional dense embeddings. It is thus interesting that a method that against the standard notions could perform so well. 
- **Practical approach:** The proposed method seems to be simple to implement and easily scalable to a large number of parallel processes. This makes it practically valuable, especially in large-scale problems.
- **Empirical evidence:** The paper provides strong empirical evidence. SOLAR outperforms industry standard embedding models while taking less query time on a 	Product to Product Recommendation dataset. SOLAR also performs equally well on various extreme classification benchmarks.
- The paper is well written and well structured. The theoretical results are also presented in an easy-to-follow format with consistent notations.

#### Weak points: 
- **Potential Issues/Questions about the baselines:** I was wondering why are the baselines in Table 1 trained for only 5 epochs and not until convergence.  Can the performance of the baselines be improved by training more?
- Similarly, I could not find information about the compute used during inference for the baselines and SOLAR. Can the inference speed of the baselines and SOLAR be improved by simply using more compute?
- How sensitive is SOLAR with respect to the random seeds chosen for the K models? In other words, how much would the performance vary if we start with a different set of random seeds? 
 
#### Overall Recommendation:
Although some information about the baselines is missing, overall I feel that the paper presents an interesting and practical method with strong empirical evidence. The paper is also well presented and is easy to follow. I currently support acceptance, but would form a final opinion based on the authors response.


The paper proposes to use high-dimensional sparse representation as opposed to low-dimensional dense representation for representing input text. This can be useful in information retrieval applications. The proposed model is evaluated on a "product to product recommendation" dataset and shows superior performance compared to the prior work.

The paper's strengths:
1. Studying an important and interesting research problem (i.e., learning sparse representation for information retrieval)
2. The paper is well-written and easy to follow. It is also well-structured. The results are well presented.

The paper's weaknesses:
1. In several places there are some overclaims that should be fixed. For example, in abstract and introduction (when describing the paper's contributions) it is mentioned that "In this paper, we make a novel and unique argument that sparse high dimensional embeddings are superior to their dense low dimensional counterparts." This is simply not true. As we go forward in the related work, we can see that the authors mention that SNRM [40] has exactly the same idea with empirical evidences. I think the authors should be more careful in claiming what is novel in their paper and what is borrowed from prior work.

2. I think the weakest part of the paper is its evaluation. Here is the detailed comments:
2.1. First of all, it has been only evaluated on a single dataset. It would be nice to see how the results can be generalized across datasets. Second, the task they used for evaluating their model is pretty artificial. There is no such real-world task as "product to product recommendation" and the experimental setting is not realistic at all. I believe using such synthetic datasets for evaluating models is acceptable when there is no good alternative publicly available. However, in this case, there are numerous large-scale datasets that can be used to evaluate the model. For example, MS MARCO dataset from Microsoft has been widely adopted by the research community and can be used for evaluating the model. A lot of Open-Domain QA datasets are available too. 

2.2. As mentioned multiple times in the paper, SNRM is the only paper with the same idea of learning high-dimensional sparse representation. However, the results reported in the table are strangely low. Looking at the SNRM paper (which was published in 2018, so it's not an old paper) we can see that it outperformed a lot of strong baselines on a number of benchmark. I understand that sometimes it is difficult to reproduce the results from some papers. However, the SNRM paper seems to be cited by over 60 papers and some of them reported its results on different datasets and they all reported a reasonable performance for the model. I personally think that the authors did not carefully select the sparsity hyper-parameter in the SNRM model. I also recommend the authors to include some non-neural baselines, such as BM25 and RM3 as they sometimes outperform neural network models on information retrieval related tasks.

3. It is not clear how scalable the model is and actually I have some doubt about its scalability. What will happen when the number of items goes beyond tens of millions or even billions? These questions are important to be answered when a model is proposed for information retrieval or recommender systems.

All in all, I believe the authors are working on an interesting problem but the experimental results are not convincing and some claims should be changed. Some questions about scalability remain unanswered.

Meta-review: The paper proposes an approach to learn sparse embeddings for documents/labels which can be trained by using multiple GPUs in parallel, and are more amenable to nearest neighbor search. 

The paper certainly seemed to have botched  comparison to SNRM and requires to fix the claims in  section 5.1. 
But, the impressive performance on extreme classification tasks is quite convincing. Also, reviewers in general are quite enthusiastic about the paper. So we would recommend the paper for acceptance, but authors certainly need to take comments of reviewers into account (especially around baselines and comparison to SNRM).

Decision: poster-presentations