Venue: NAACL 2021

# _Query2Prod2Vec_

Grounded Word Embeddings for eCommerce

 Federico Bianchi

Bocconi University

Milano, Italy

f.bianchi@unibocoroni.it &Jacopo Tagliabue\({}^{*}\)

Coveo Labs

New York, USA

jtagliabue@coveo.com & Bingqing Yu

Coveo

Montreal, Canada

cyu2@coveo.com

Corresponding author. All authors contributed equally and are listed alphabetically.

###### Abstract

We present **Query2Prod2Vec**, a model that grounds lexical representations for product search in product embeddings: in our model, _meaning_ is a mapping between words and a latent space of products in a digital shop. We leverage shopping sessions to learn the underlying space and use merchandising annotations to build lexical analogies for evaluation: our experiments show that our model is more accurate than known techniques from the NLP and IR literature. Finally, we stress the importance of data efficiency for product search outside of retail giants, and highlight how **Query2Prod2Vec** fits with practical constraints faced by most practitioners.

+
Footnote †: Corresponding author. All authors contributed equally and are listed alphabetically.

## 1 Introduction

The eCommerce market reached in recent years an unprecedented scale: in 2020, 3.9 trillion dollars were spent globally in online retail Cramer-Flood (2020). While shoppers make significant use of search functionalities, improving their experience is a never-ending quest Econsultancy (2020), as outside of few retail giants users complain about sub-optimal performances Baymard Institute (2020). As the technology behind the industry increases in sophistication, neural architectures are gradually becoming more common Tsagkias et al. (2020) and, with them, the need for accurate word embeddings for Information Retrieval (IR) and downstream Natural Language ProcessingNLP tasks Yu and Tagliabue (2020); Tagliabue et al. (2020).

Unfortunately, the success of standard and contextual embeddings from the NLP literature Mikolov et al. (2013); Devlin et al. (2019) could not be immediately translated to the product search scenario, due to some peculiar challenges Bianchi et al. (2020), such as short text, industry-specific jargon Bai et al. (2018), low-resource languages; moreover, specific embedding strategies have often been developed in the context of high-traffic websites Grbovic et al. (2016), which limit their applicability in many practical scenarios. In _this_ work, we propose a sample efficient word embedding method for IR in eCommerce, and benchmark it against SOTA models over industry data provided by partnering shops. We summarize our contributions as follows:

1. we propose a method to learn dense representations of words for eCommerce: we name our method **Query2Prod2Vec**, as the mapping between words and the latent space is mediated by the product domain;
2. we evaluate the lexical representations learned by **Query2Prod2Vec** on an analogy task against SOTA models in NLP and IR; benchmarks are run on two independent shops, differing in traffic, industry and catalog size;
3. we detail a procedure to generate synthetic embeddings, which allow us to tackle the "cold start" challenge;
4. we release our implementations, to help the community with the replication of our findings on other shops1. Footnote 1: Public repository available at: [https://github.com/coveoss/ecommerce-query-embeddings](https://github.com/coveoss/ecommerce-query-embeddings).

While perhaps not fundamental to its industry significance, it is important to remark that grounded lexical learning is well aligned with theoretical considerations on _meaning_ in recent (and less recent) literature Bender and Koller (2020); Bisk et al. (2020); Montague (1974).



Embeddings for Product Search: an Industry Perspective

In product search, when the shopper issues a query (e.g. "sneakers") on a shop, the shop search engine returns a list of products matching the query intent and possibly some contextual factor - the shopper at that point may either leave the website, or click on products to further explore the offering and eventually make a purchase.

Unlike web search, which is exclusively performed at massive scale, product search is a problem that both big and small retailers have to solve: while word embeddings have revolutionized many areas of NLP Mikolov et al. (2013), word embeddings for product queries are especially challenging to obtain at scale, when considering the huge variety of use cases in the overall eCommerce industry. In particular, based on industry data and first-hand experience with dozens of shops in our network, we identify four _constraints_ for effective word embeddings in eCommerce:

1. **Short text**. Most product queries are very short - 60% of all queries in our dataset are one-word queries, > 80% are two words or less; the advantage of contextualized embeddings may therefore be limited, while lexical vectors are fundamental for downstream NLP tasks Yu and Tagliabue (2020); Bianchi et al. (2020). For this reason, the current work specifically addresses the quality of _word_ embeddings2. Footnote 2: Irrespectively of how the lexical vectors are computed, query embeddings can be easily recovered with the usual techniques (e.g. sum or average word embeddings Yi et al. (2020)): as we mention in the concluding remarks, investigating compositionality is an important part of our overall research agenda.
2. **Low-resource languages**. Even shops that have the majority of their traffic on English domain typically have smaller shops in low-resource languages.
3. **Data sparsity**. In _Shop X_ below, only 9% of all shopping sessions have a search interaction3. Search sparsity, coupled with vertical-specific jargon and the usual long tail of search queries, makes data-hungry models unlikely to succeed for most shops. Footnote 3: This is a common trait verified across industries and sizes: among dozens of shops in our network, 30% is the highest _search vs no-search_ session ratio; _Shop Y_ below is around 29%.
4. **Computational capacity**. The majority of the market has the necessity to strike a good trade-off between quality of lexical representations and the cost of training and deploying models, both as hardware expenses and as additional maintenance/training costs.

The embedding strategy we propose - **Query2Prod2Vec** - has been designed to allow efficient learning of word embeddings for product queries. Our findings are useful to a wide range of practitioners: large shops launching in new languages/countries, mid-and-small shops transitioning to dense IR architectures and the raising wave of multi-tenant players4: as A.I. providers grow by deploying their solutions on multiple shops, "cold start" scenarios are an important challenge to the viability of their business model.

Footnote 4: As an indication of the market opportunity, only in 2019 and only in the space of AI-powered search and recommendations, we witnessed Coveo Techcrunch, Algolia Techcrunch, and Lucidworks Techcrunch (2019) raising more than 100M USD each from venture funds.

## 3 Related Work

The literature on learning representations for lexical items in NLP is vast and growing fast; as an overview of classical methods, Baroni et al. (2014) benchmarks several count-based and neural techniques Landauer and Dumais (1997); Mikolov et al. (2013); recently, context-aware embeddings Peters et al. (2018); Devlin et al. (2019) have demonstrated state-of-the-art performances in several semantic tasks Rogers et al. (2020); Nozza et al. (2020), including document-based search Nogueira et al. (2020), in which target entities are long documents, instead of product Craswell et al. (2020). To address IR-specific challenges, other embedding strategies have been proposed: _Search2Vec_Grbovic et al. (2016) uses interactions with ads and pages as context in the typical context-target setting of skip-gram models Mikolov et al. (2013); _QueryNGram2Vec_Bai et al. (2018) additionally learns embeddings for n-grams of word appearing in queries to better cover the long tail. The idea of using vectors (from images) as an aid to query representation has also been suggested as a heuristic device by Yu et al. (2020), in the context of personalized language models; _this_ work is the first to our knowledge to benchmark embeddings 

tuned for domain-specific tasks), _and_ investigate sample efficiency for small-data contexts.

## 4 Query2Prod2Vec

In **Query2Prod2Vec**, the representation for a query \(q\) is built through the representation of the objects that \(q\) refers to. Consider a typical shopper-engine interaction in the context of product search: the shopper issues a query, e.g. "shoes", the engine replies with a noisy set of potential referents, e.g. pairs of shoes from the shop inventory, among which the shopper may select relevant items. Hence, this dynamics is reminiscent of a cooperative language game Lewis (1969), in which shoppers give noisy feedback to the search engine on the meaning of the queries. A full specification of **Query2Prod2Vec** therefore involves a representation of the target domain of reference (i.e. products in a digital shop) and a denotation function.

### Building a Target Domain

We represent products in a target shop through a _prod2vec_ model built with anonymized shopping sessions containing user-product interactions. Embeddings are trained by solving the same optimization problem as in classical _word2vec_Mikolov et al. (2013a): _word2vec_ becomes _prod2vec_ by substituting _words_ in a _sentence_ with _products_ viewed in a _shopping session_Mu et al. (2018). The utility of _prod2vec_ is independently justified Grbovic et al. (2015); Tagliabue and Yu (2020) and, more importantly, the referential approach leverages the abundance of browsing-based interactions, as compared to search-based interactions: by learning _product_ embeddings from abundant behavioral data first, we sidestep a major obstacle to reliable word representation in eCommerce. Hyperparameter optimization follows the guidelines in Bianchi et al. (2020a), with a total of \(26{,}057\) (_Shop X_) and \(84{,}575\) (_Shop Y_) product embeddings available for downstream processing5.

Footnote 5: Final parameters for _prod2vec_ are: \(dimension=50\), \(win\_size=10\), \(iterations=30\), \(ns\_exponent=0.75\).

### Learning Embeddings

The fundamental intuition of **Query2Prod2Vec** is treating clicks after \(q\) as a noisy feedback mapping \(q\) to a portion of the latent product space. In particular, we compute the embedding for \(q\) by averaging the product embeddings of all products clicked after it, using frequency as a weighting factor (i.e. products clicked often contribute more). The model has one free parameter, _rank_, which controls how many embeddings are used to build the representation for \(q\): if _rank=k_, only the \(k\) most clicked products after \(q\) are used. The results in Table 1 are obtained with _rank=5_, as we leave to future work to investigate the role of this parameter.

The lack of large-scale search logs in the case of new deployments is a severe issue for successful training. The referential nature of **Query2Prod2Vec** provides a fundamental competitive advantage over models building embeddings from past linguistic behavior _only_, as synthetic embeddings can be generated as long as cheap session data is available to obtain an initial _prod2vec_ model. As detailed in the ensuing section, the process happens in two stages, event generation and embeddings creation.

### Creating Synthetic Embeddings

The procedure to create synthetic embeddings is detailed in Algorithm 1: it takes as input a list of words, a pre-defined number of sampling iterations, a popularity distribution over products6, and it returns a list of synthetic search events, that is, a mapping between words and lists of products "clicked". Simulating the _search_ event can be achieved through the existing search engine, as, from a practical standpoint, _some_ IR system must already be in place given the use case under consideration. To avoid over-relying on the quality of IR and prove the robustness of the method, all the simulations below are not performed with the actual production API, but with a custom-built inverted index over product meta-data, with a simple TF-IDF weighting and Boolean search.

Footnote 6: Please note that data on product popularity can be easily obtained through marketing tools, such as Google Analytics.

For the second stage, we can treat the synthetic click events produced by Algorithm 1 as a drop-in replacement for user-generated events - that is, for any query \(q\), we calculate an embedding by averaging the product embeddings of the relevant products, weighted by frequency7. Putting the two stages together, **Query2Prod2Vec** can not only produce reliable query embeddings based on historical data, but also learn approximate embeddings for a large vocabulary _before_ being exposed 

to any search interaction: in Section 7 we report the performance of **Query2Prod2Vec** when using only synthetic embeddings8.

Footnote 8: All the experiments are performed with \(N=500\) simulated search events per query.

## 5 Dataset and Baselines

### Dataset

Following best practices in the multi-tenant literature Tagliabue et al. (2020), we benchmark all models on different shops to test their robustness. In particular, we obtained catalog data, search logs and anonymized shopping sessions from two partnering shops, _Shop X_ and _Shop Y_: _Shop X_ is a sport apparel shop with Alexa ranking of approximately _200k_, representing a prototypical shop in the middle of the long tail; _Shop Y_ is a home improvement shop with Alexa ranking of approximately _10k_, representing an intermediate size between _Shop X_ and public companies in the space. Linguistic data is in Italian for both shops, and training is done on random sessions from the period June-October 2019: after sampling, removal of bot-like sessions and pre-processing, we are left with \(722{,}479\) sessions for _Shop X_, and \(1{,}986{,}452\) sessions for _Shop Y_.

### Baselines

We leverage the unique opportunity to join catalog data, search logs and shopping sessions to extensively benchmark **Query2Prod2Vec** against a variety of methods from NLP and IR.

* **Word2Vec and FastText**. We train a CBOW Mikolov et al. (2013) and a FastText model Bojanowski et al. (2017) over product descriptions in the catalog;
* _UmBERTo9_. The \(\langle s\rangle\) embedding of the last layer of the architecture is the query embedding; Footnote 9: [https://huggingface.co/Musixmatch/umberto-commoncrawl-cased-v1](https://huggingface.co/Musixmatch/umberto-commoncrawl-cased-v1)
* **Search2Vec**. We implement the skip-gram model from Grbovic et al. (2016), by feeding the model with sessions composed of search queries and user clicks. Following the original model, we also train a time-sensitive variant, in which time between actions is used to weight query-click pairs differently;
* **Query2Vec**. We implement a different context-target model, inspired by Egg (2019): embeddings are learned by the model when it tries to predict a (purchased or clicked) item starting from a query;
* **QueryNGram2Vec**. We implement the model from Bai et al. (2018). Besides learning representations through a skip-gram model as in Grbovic et al. (2016), the model learns the embeddings of _unigrams_ to help cover the long tail for which no direct embedding is available.

To guarantee a fair comparison, all models are trained on the same sessions. For all baselines, we follow the same hyperparameters found in the cited works: the dimension of query embedding vectors is set to 50, except that 768-dimensional vectors are used for **UmBERTo**, as provided by the pre-trained model.

As discussed in Section 1, a distinguishing feature of **Query2Prod2Vec** is _grounding_, that is, the relation between words and an external domain - in this case, products. It is therefore interesting not only to assess a possible _quantitative_ gap in the quality of the representations produced by the baseline models, but also to remark the _qualitative_ difference at the core of the proposed method: if words are _about_ something, pure co-occurrence patterns may be capturing only fragments of lexical meaning Bianchi et al. (2021).

 

Solving Analogies in eCommerce

As discussed in Section 2, we consider evaluation tasks focused on _word meaning_, without using product-based similarity (as that would implicitly and unfairly favor referential embeddings). Analogy-based tasks [16] are a popular choice to measure semantic accuracy of embeddings, where a model is asked to fill templates like _man : king = woman :_?; however, preparing analogies for digital shops presents non trivial challenges for human annotators: these would in fact need to know both the language and the underlying space ("air max" is closer to "nike" than to "adidas"), with the additional complication that many candidates may not have "determinate" answers (e.g. if _Adidas_ is to _Gazelle_, then _Nike_ is to what exactly?). In building our testing framework, we keep the intuition that analogies are an effective way to test for lexical meaning and the assumption that human-level concepts should be our ground truth: in particular, we programmatically produce analogies by leveraging existing human labelling, as indirectly provided by the merchandisers who built product catalogs10.

Footnote 10: It is important to note that this categorization is done by product experts for navigation and inventory purposes: all product labels are produced independently from any NLP consideration.

### Test Set Preparation

We extract words from the merchandising taxonomy of the target shops, focusing on three most frequent fields in query logs: _product type_, _brand_ and _sport activity_ for _Shop X_; _product type_, _brand_ and _part of the house_ for _Shop Y_. Our goal is to go from taxonomy to analogies, that is, showing how for each pair of taxonomy **types** (e.g. _brand : sport_), we can produce two pairs of **tokens** (_Wilson : tennis_, _Cressi : scubalying_), and create two analogies: _b1 : s1 = b2 : ? (target: s2)_ and _b2: s2 = b1 : ? (target: s1)_ for testing purposes. For each **type** in a pair (e.g. _brand : sport_), we repeat the following for all possible values of _brand_ (e.g. "Wilson", "Nike") - given a brand \(B\):

1. we loop over the catalog and record all values of _sport_, along with their frequency, for the products made by \(B\). For example, for \(B=Nike\), the distribution may be: {"soccer": 10, " basketballall": 8, "scubadviving": 0 }; for \(B=Wilson\), it may be: {"tennis": 8};
2. we calculate the Gini coefficient [35] over the distribution on the values of _sport_ and choose a conservative Gini threshold, i.e. \(75th\) percentile: the goal of this threshold is to avoid "undetermined" analogies, such as _Addias : Gazelle = Nike :_?. The intuition behind the use of a dispersion measure is that product analogies are harder if the relevant label is found across a variety of products11. Footnote 11: In other words, _Wilson : tennis = Atomic : ? (skining)_ is a better analogy than _Adidas : Gazelle = Nike :_?.

 

data) has the best performance12, while maintaining a very competitive coverage. More importantly, following our considerations in Section 2, results confirm that producing competitive embeddings on shops with different constraints is a challenging task for existing techniques, as models tend to either rely on specific query distribution (e.g. **Search2Vec (time)**), or the availability of linguistic and catalog resources with good coverage (e.g. **Word2Vec**). **Query2Prod2Vec** is the only model performing with comparable quality in the two scenarios, further strengthening the methodological importance of running benchmarks on more than one shop if findings are to be trusted by a large group of practitioners.

Footnote 12: HR@20 was also computed, but omitted for brevity as it confirmed the general trend.

### Sample Efficiency and User Studies

To investigate sample efficiency, we run two further experiments on _Shop X_: first, we run **AT** giving only 1/3 of the original data to **Query2Prod2Vec** (both for the _prod2vec_ space, and for the denotation). The small-dataset version of **Query2Prod2Vec** still outperforms all other full-dataset models in Table 1 (_HR@5,10 = 0.276 / 0.380_). Second, we train a **Query2Prod2Vec** model _only with simulated data_ produced as explained in Section 4 - that is, with _zero_ data from real search logs. The entirely simulated **Query2Prod2Vec** shows performance competitive with the small-dataset version (_HR@5,10 = 0.259 / 0.363_)13, outperforming all baselines.

Footnote 13: A similar result was obtained on _Shop Y_, and it is omitted for brevity.

As a further independent check, we supplement **AT** with a small semantic similarity task (**ST**) on _Shop X14_: two native speakers are asked to solve a small set (\(46\)) of manually curated questions in the form: "Given the word _Nike_, which is the most similar, _Adidas_ or _Wilson?_". **ST** is meant to (partially) capture how much the embedding spaces align with lexical intuitions of generic speakers, independently of the product search dynamics. Table 1 reports results treating human ratings as ground truth and using cosine similarity on the learned embeddings for all models15. **Query2Prod2Vec** outperforms all other methods, further suggesting that the representations learned through referential information capture some aspects of lexical knowledge.

Footnote 14: _Shop X_ is chosen since it is easier to find speakers familiar with sport apparent than DIY items.

Footnote 15: Inter-rater agreement was substantial, with _Cohen Kappa Score_=0.67 [McHugh, 2012].

### Computational Requirements

As stressed in Section 2, accuracy and resources form a natural trade-off for industry practitioners. Therefore, it is important to highlight that, our model is not just more accurate, but significantly more efficient to train: the best performing **Query2Prod2Vec** takes 30 minutes (CPU only) to be completed for the larger _Shop Y_, while other competitive models such as **Search2Vec(time)** and **QueryNGram2Vec** require 2 to 4 hours16. Being able to quickly generate many models allows for cost-effective analysis and optimization; moreover, infrastructure cost is heavily related to ethical and social issues on energy consumption in NLP Strubell et al. (2019).

\begin{table}
\begin{tabular}{l c c c c} \hline \hline _Model_ & **HR@5,10 for X** & **HR@5,10 for Y** & **CV (X/Y)** & **Acc on ST** \\ \hline _Query2Prod2Vec (real data)_ & **0.332 / 0.468** & **0.277 / 0.376** & 0.965/0.924 & **0.88** \\ \hline _Word2Vec_ & 0.206 / 0.242 & 0.005 / 0.009 & 0.47 / 0.03 & 0.68 \\ _Query2Vec_ & 0.077 / 0.113 & 0.065 / 0.120 & 0.97 / 0.93 & 0.54 \\ _QueryNGram2Vec_ & 0.071 / 0.122 & 0.148 / 0.216 & **0.99** / 0.92 & 0.82 \\ _FastText_ & 0.068 / 0.116 & 0.010 / 0.012 & 0.52 / 0.03 & 0.57 \\ _UmBERTo_ & 0.019 / 0.042 & 0.030 / 0.103 & 0.99 / **1.00** & 0.57 \\ _Search2Vec (time)_ & 0.018 / 0.025 & 0.232 / 0.329 & 0.23 / 0.90 & 0.17 \\ _Search2Vec_ & 0.016 / 0.024 & 0.095 / 0.150 & 0.23 / 0.90 & 0.17 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Hit Rate (HR) and coverage (CV) for all models and two shops on **AT**; on the rightmost column, Accuracy (Acc) for all models on **ST**.

 



## 8 Conclusion and Future Work

In _this_ work, we learned reference-based word embeddings for product search: **Query2Prod2Vec** significantly outperforms other embedding strategies on lexical tasks, and consistently provides good performance in small-data and zero-data scenarios, with the help of synthetic embeddings. In future work, we will extend our analysis to _i)_ specific IR tasks, within the recent paradigm of the dual encoder model Karpukhin et al. (2020), and _ii)_ compositional tasks, trying a systematic replication of the practical success obtained by Yu et al. (2020) through image-based heuristics.

When looking at models like **Query2Prod2Vec** in the larger industry landscape, we hope our methodology can help the field broaden its horizons: while retail giants indubitably played a major role in moving eCommerce use cases to the center of NLP research, finding solutions that address a larger portion of the market is not just practically important, but also an exciting agenda of its own17.

Footnote 17: Please note that a previous draft of this article appeared on _arxiv_ – [https://arxiv.org/abs/2104.02061](https://arxiv.org/abs/2104.02061) – _after_ the review process, but before the camera-ready submission.

## 9 Ethical Considerations

_Coveo_ collects anonymized user data when providing its business services in full compliance with existing legislation (e.g. GDPR). The training dataset used for all models employs anonymous UUIDs to label events and sessions and, as such, it does not contain any information that can be linked to shoppers or physical entities; in particular, data is ingested through a standardized client-side integration, as specified in our public protocol.

## Acknowledgements

We wish to thank Nadia Labai, Patrick John Chia, Andrea Polonioli, Ciro Greco and three anonymous reviewers for helpful comments on previous versions of this article. The authors wish to thank _Coveo_ for the support and the computational resources used for the project. Federico Bianchi is a member of the Bocconi Institute for Data Science and Analytics (BIDSA) and the Data and Marketing Insights (DMI) unit.

## References

* Bai et al. (2018) Xiao Bai, Erik Ordentlich, Yuanyuan Zhang, Andy Feng, Adwait Ratnaparkhi, Reena Somvanshi, and Aldi Tjahjadi. 2018. Scalable query n-gram embedding for improving matching and relevance in sponsored search. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, KDD '18, pages 52-61, New York, NY, USA. ACM.
* Baroni et al. (2014) Marco Baroni, Georgiana Dinu, and German Kruszewski. 2014. Don't count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 238-247, Baltimore, Maryland. Association for Computational Linguistics.
* Institute (2020) Baymard Institute. 2020. _Site Search for Ecommerce_.
* Bender and Koller (2020) Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5185-5198, Online. Association for Computational Linguistics.
* Bianchi et al. (2021) Federico Bianchi, Ciro Greco, and Jacopo Tagliabue. 2021. Language in a (Search) Box: Grounding Language Learning in Real-World Human-Machine Interaction. In _NAACL-HLT_. Association for Computational Linguistics.
* Bianchi et al. (2020a) Federico Bianchi, Jacopo Tagliabue, Bingqing Yu, Luca Bigon, and Ciro Greco. 2020a. Fantastic embeddings and how to align them: Zero-shot inference in a multi-shop scenario. In _Proceedings of the SIGIR 2020 eCom workshop_.
* Bianchi et al. (2020b) Federico Bianchi, Bingqing Yu, and Jacopo Tagliabue. 2020b. Bert goes shopping: Comparing distributional models for product representations. _arXiv preprint arXiv:2012.09807_.
* Bisk et al. (2020) Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. 2020. Experience grounds language. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 8718-8735, Online. Association for Computational Linguistics.
* Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. _Transactions of the Association for Computational Linguistics_, 5:135-146.
* Catalano et al. (2009) Michael Catalano, Tanya Leise, and Thomas Pfaff. 2009. Measuring resource inequality: The gini coefficient. _Numeracy_, 2.
* Cramer-Flood (2020) Ethan Cramer-Flood. 2020. _Global Ecommerce 2020. Ecommerce Decelerates amid Global Retail Contraction but Remains a Bright Spot_.



Nick Craswell, Bhaskar Mitra, E. Yilmaz, Daniel Fernando Campos, and E. Voorhees (2020)Overview of the tree 2019 deep learning track. ArXivabs/2003.07820. Cited by: SS4.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 4171-4186. External Links: Link, Document Cited by: SS5.
* E. Epg (2019)Query2vec: search query expansion with query embeddings. Vol. , N. Djuric, V. Radosavljevic, F. Silvestri, R. Baeza-Yates, A. Feng, E. Ordentlich, L. Yang, and G. Owens (2016)Scalable semantic matching of queries to ads in sponsored search advertising. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '16, New York, NY, USA, pp. 375-384. External Links: ISBN 978-1-4503-3845-3 Cited by: SS1.
* M. Grbovic, V. Radosavljevic, N. Djuric, N. Bhamidipati, J. Savla, V. Bhagwan, and D. Sharp (2015)E-commerce in your nibox: product recommendations at scale. In Proceedings of KDD '15, H. Walls, S. Edunov, D. Chen, and W. Yih (Eds.), Online, pp. 6781. External Links: Link, Document Cited by: SS1, SS2.
* P. K. Landauer and S. T. Dumais (1997)A solution to plato's problem: the latent semantic analysis theory of acquisition, induction, and representation of knowledge. External Links: Link Cited by: SS1, SS2.
* D. Lewis (1969)Convention. Mass. Harvard UP. External Links: Link Cited by: SS1.
* M. McHugh (2012)Interrater reliability: the kappa statistic. Biochemia medica : casopis Hvatskoga drutskoga drutskva medicinskhi biokemara / HDMB22, pp. 2276-82. External Links: Link, Document Cited by: SS1.
* T. Mikolov, K. Chen, G. Sorrado, and J. Dean (2013a)Efficient estimation of word representations in vector space. CoRRabs/1301.3781. External Links: Link, 1301.3781 Cited by: SS1.
* Volume 2, NIPS'13, Red Hook, NY, USA, pp. 3111-3119. External Links: Link, Document Cited by: SS1.
* R. Montague (1974)English as a formal language. In Formal Philosophy: Selected Papers of Richard Montague, Yale University Press, New Haven, London, pp. 188-222. External Links: Link, Document Cited by: SS1.
* C. Mu, G. Yang, and Z. Yan (2018)Revisiting skip-gram negative sampling model with regularization. CoRRabs/1804.00306. External Links: Link, 1804.00306 Cited by: SS1.
* R. Nogueira, Z. Jiang, and J. Lin (2020)Document ranking with a pretrained sequence-to-sequence model. In EMNLP, Cited by: SS1.
* Debora Nozza, Federico Bianchi, and Dirk Hovy (2020)What the [MASK]? making sense of language-specific BERT models. arXiv preprint arXiv:2003.02912. External Links: Link, 2003.02912 Cited by: SS1.
* Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer (2018)Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, Louisiana, pp. 2227-2237. External Links: Link, Document Cited by: SS1.
* A. Rogers, O. Kovaleva, and A. Rumshisky (2020)A primer in BERTology: what we know about how BERT works. Transactions of the Association for Computational Linguistics8, pp. 842-866. External Links: Link, ISSN 1558-1221, Link, Document Cited by: SS1.
* E. Strubell, A. Ganesh, and A. McCallum (2019)Energy and policy considerations for deep learning in mlp. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3645-3650. External Links: Link, Document Cited by: SS1.
* J. Tagliabue and B. Yu (2020)Shopping in the multiverse: a counterfactual approach to insession attribution. In Proceedings of the SIGIR 2020 Workshop on eCommerce (ECOM 20), Cited by: SS1.
* J. Tagliabue, B. Yu, and M. Beaulieu (2020)How to grow a (product) tree: personalized category suggestions for eCommerce type-ahead. In Proceedings of The 3rd Workshop on e-Commerce and NLP, Seattle, WA, USA, pp. 7-18. External Links: Link, Document Cited by: SS1.
* J. Tagliabue, B. Yu, and M. Beaulieu (2020)How to grow a (product) tree: personalized category suggestions for eCommerce type-ahead. In Proceedings of The 3rd Workshop on e-Commerce and NLP, Seattle, WA, USA, pp. 77-18. External Links: Link, Document Cited by: SS1.

 

Techncrunch. covoe-raises-227m-at-1b-valuation.
* [Techncrunch(2019a)] Techcrunch. 2019a. Algolia finds S110m from accel and salesforce.
* [Techcrunch(2019b)] Techcrunch. 2019b. Lucidworks raises S100m to expand in ai finds.
* [Tsagkias et al.(2020)] Manos Tsagkias, Tracy Holloway King, Surya Kallumadi, Vanessa Murdock, and Maarten de Rijke. 2020. Challenges and research opportunities in ecommerce search and recommendations. In _SIGIR Forum_, volume 54.
* [Vasile et al.(2016)] Flavian Vasile, Elena Smirnova, and Alexis Conneau. 2016. Meta-prod2vec: Product embeddings using side-information for recommendation. _Proceedings of the 10th ACM Conference on Recommender Systems_.
* [Yu and Tagliabue(2020)] Bingqing Yu and Jacopo Tagliabue. 2020. Blending search and discovery: Tag-based query refinement with contextual reinforcement learning. In _EComNLP_.
* [Yu et al.(2020)] Bingqing Yu, Jacopo Tagliabue, Ciro Greco, and Federico Bianchi. 2020. An image is worth a thousand features: Scalable product representations for in-session type-ahead personalization. _Companion Proceedings of the Web Conference 2020_.