Venue: NAACL 2022

# Augmenting Training Data for Massive Semantic Matching Models in Low-Traffic E-commerce Stores

Ashutosh Joshi1 Shankar Vishwanath1 Choon Hui Teo1

**Vaclav Petricek1 Vishy Vishwanathan1 Rahul Bhagat1 Jonathan May1,2 1**Amazon.com, Inc., 2Information Sciences Institute, University of Southern California {jasutos, shavis, choonhui, petricek, vishy, rbhagat}@amazon.com jonnay@isi.edu

###### Abstract

Extreme multi-label classification (XMC) systems have been successfully applied in e-commerce Shen et al. (2020); Dahiya et al. (2021) for retrieving products based on customer behavior. Such systems require large amounts of customer behavior data (e.g. queries, clicks, purchases) for training. However, behavioral data is limited in low-traffic e-commerce stores, impacting performance of these systems. In this paper, we present a technique that augments behavioral training data via query reformulation. We use the Aggregated Label eXtreme Multi-label Classification (AL-XMC) system Shen et al. (2020) as an example semantic matching model and show via crowd-sourced human judgments that, when the training data is augmented through query reformulations, the quality of AL-XMC improves over a baseline that does not use query reformulation. We also show in online A/B tests that our method significantly improves business metrics for the AL-XMC model.

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

E-commerce search engines are primarily keyword-based information retrieval (IR) systems comprising two main operations-matching and ranking Manning et al. (2008). Lexical and/or semantic matching algorithms generate a recall-focused _matchset_ which is then ranked based on the match quality of the document (product) to the query Joachims et al. (2007). Lexical matching algorithms such as Okapi-BM25 Robertson Walker (194); Robertson and Zaragoza (2009) score a query-product pair as a weighted sum of overlapping keywords. These approaches, used in many retrieval tasks Lee et al. (2019); Boytsov and Nyberg (2020), do not capture customer behavior signals (purchase, stream, etc) and thus do not capture customer preferences.

Semantic matching learns representations of queries and products based on customer behavior and hence captures the products that customers prefer. Semantic matching can be implemented using dual encoders Nigam et al. (2019); Huang et al. (2013), that separately build query and product representations, then combine the two in a final shared space to determine the similarity of the pair. It is also implemented using extreme multi-label classification (XMC) systems. In particular, Aggregated Label eXtreme Multi-label Classification (AL-XMC) Shen et al. (2020) partitions the label (product) space by clustering labels into hierarchically granular clusters in a b-ary tree structure. Irrespective of the approach, semantic matching requires a large amount of behavioral data to train. But in newly launched sites or for new products, this behavioral data is not abundant.

In this paper, we propose a method to augment the data available to train semantic models in low-resource e-commerce stores. We use item-to-item collaborative filtering to identify queries that show strong behavioral associations. Such query pairs elicit a similar behavioral response from customers and so represent the same purchase intent. We use these query pairs to fine-tune the text-to-text-transfer-transformer (T5) language model Raffel

Figure 1: Overview of our approach. We induce query reformulation pairs from behavioral training data to finetune a reformulation model (T5). We then augment the original behavioral data with reformulated queries to train an AL-XMC semantic model.



et al., 2020) to generate query paraphrases, which we then use to augment the data available to train the semantic model.

We test our method on an AL-XMC semantic model, though it can be easily used to augment training data for any semantic model. We show, by offline crowd-sourced human judgments as well as online A/B tests, that our data augmentation technique generates reliable training data that improves the performance of the AL-XMC semantic model.

## 2 Approach

A pipeline diagram of our system is shown in Figure 1. The main goal of this work is to increase good quality query-product pairs with which to train an AL-XMC model; we do this by generating alternative queries from known queries using a fine-tuned T5 model. To fine-tune the T5 model, we identify purchase-intent-preserving query pairs from historical customer data. In Section 3 we describe our approach to constructing the data used to fine-tune the T5 model.

## 3 Constructing query reformulations

We regard customer interaction data as tuples of the form \(\langle q,t,p\rangle\), each consisting of a query \(q\), interaction type \(t\) (eg: search, purchase, etc.) and product \(p\). It is generally the case that newer stores do not have sufficient interaction data to train robust semantic models. To increase the data available to train semantic models, we can either increase the number of queries associated with the product, the number of products associated with the queries, or both. In a typical e-commerce site the number of products is usually fixed. It is not usually possible or desirable to artificially augment this set. Instead we fine-tune a language model to generate _query_ reformulations and associate the reformulated queries to products using the same interaction type \(t\) as the original query. The reformulated queries increase the variance of the tokens, including those from rare queries, in accordance with their distribution in the target store. In order for this to happen, the reformulated queries need to encompass the same purchase intent as the original queries.

For a language model to create high-fidelity reformulations, we need an adequate corpus of intents preserving query paraphrase pairs. However, generating such a corpus is not trivial. Queries vary widely in specificity, from highly generic (_gifts for teens_) to highly specific (_HP 63XL_). Also, a product can satisfy multiple shopping intents. For example, the same product can be purchased for _kids laptop_ and _cheap laptop_. In this section, we describe our methodology to generate high fidelity query reformulations.

### Identifying related queries using Collaborative Filtering

To generate fine-tuning data for a query reformulation model, we need to identify query pairs that have the same purchase intent. We adapt item-to-item collaborative filtering Linden et al. (2003) to generate query-to-query similarities. In this formulation, we interpret common product interactions of queries as query co-occurrence. For our case, we say two queries _co-occur_ iff they _lead to the purchase of the same product_. Given this interpretation, we can then use different measures like pointwise mutual information (PMI) to quantify dependence between queries. Figures 1(a) and 1(b) show the top 6 similar queries (or _similarities_) associated with seed queries _aquarium filter brush_ and _heart name tag for dog_, respectively. As we can see from these figures, the similarities are generally related to the seeds. However, as seen in Figure 1(b), the similarities, while related, do not preserve purchase intent in all cases.

### Clustering the query graph

Since our purpose is to identify queries that have the same purchase intent, for every seed query, we filter out any similarities with a _query specificity_ that is not within 10% of the specificity of the seed.

Figure 2: Top 6 similarities of two example queries, sorted by PMI. While all the top matches to _aquarium filter brush_ appear to refer to the same product as the key, many of the top matches to _heart name tag for dog_ refer to products that are not good matches.

 

We define specificity, \(s_{Q}(q)\), as the inverse normalized click entropy (\(H(q)\)) of clicks for a query, \(q\).

\[H(q)=-\sum_{p\in c(q)}P(p|q)\log P(p|q)\] \[s_{Q}(q)=1-\frac{H(q)}{\max_{q^{\prime}\in Q}H(q^{\prime})}\]

where \(P(p|q)\) is the probability that a customer will click on product \(p\) after issuing query \(q\) and \(c(q)\) is the set of all products clicked by users after issuing \(q\). A very specific query (e.g.: _k-cup coffee pods 72 count_) will have low \(H(q)\) and thus high specificity \(s_{Q}(q)\) while broad queries (e.g.: _party dress_) will have high \(H(q)\) and low \(s_{Q}(q)\).

Even after specificity filtering, similarities can still show associations that do not preserve query intent (Figure 1(b)). To further reduce noise, we identify clusters in the query similarities graph. We form this graph by considering every query to be a vertex, and an edge to connect vertices if there is a similarity relation between them. The whole graph, though extremely sparse (\(<0.0001\%\) of the edges of a fully connected graph), can still contain tens of millions of vertices and hundreds of millions of edges for some e-stores. Since our objective is to identify queries that lead to the purchase of similar products, we break the similarities graph into product sub-graphs for each product \(p\) as \(G_{p}=(V_{p},E_{p})\) where \(V_{p}=(V_{Q_{p}}\cup V_{N_{p}})\) is the set of vertices formed as follows: \(V_{Q_{p}}\) is the set of queries associated with (i.e. that lead to the purchase of) \(p\), and \(V_{N_{p}}\) is the set of similarities of \(V_{Q_{p}}\). An edge connecting \((v_{i},v_{j})\) is in \(E_{p}\) iff \(v_{i},v_{j}\in V_{p}\) and there exists a similarity relation between them. By processing each product sub-graph independently, we can parallelize the clustering problem and operate only on small sub-graphs.

Figure 3 shows the sub-graph of the product _Lily's 8-Inch Japanese 440C Pet Dog Chunker Shears_. The graph naturally separates the queries into multiple sets: thinning shears, grooming shears, and professional grooming scissors, each marked in different colors in the figure.

We cluster each similarities subgraph to identify sets of queries that show high connectivity within the cluster and low connectivity outside it. The aim of clustering is to find groups of behaviorally related queries that satisfy the same customer intent. The subgraph edges already tell us that two queries are related, in that they lead to the purchase of the same product more frequently than would be expected by chance alone. By clustering we can identify groups of mutually related queries. If two queries are behaviorally related to a similar set of queries we can consider them to be related as well.

Graph clustering, though, is an ambiguous problem, with no universal definition of a cluster. Depending on the algorithm used, we can detect one, two, or three clusters in the graph in Figure 4.

Edge clustering Ahn et al. (2010) combines features of soft clustering, where clusters can share the same members, with hierarchical clustering, where clusters are nested into dendrograms that represent progressive subdivision of a single cluster into a set of singletons. We next describe an edge clustering algorithm that is hierarchical but does not prohibit vertex sharing between clusters.

### Seeding query clusters

Hierarchical clustering recursively merges clusters according to some merge criterion, beginning with each vertex in its own cluster. This imposes a restriction that clusters not share vertices. To avoid this, we let some vertices' base clusters consist of themselves _and_ their neighbors (i.e. their similarities). To determine which initial clusters are singleton vertices and which contain vertices and neighbors, we use the _Clustering Coefficient (C3)_, which measures the cliquishness of the neighborhood of a vertex in a graph Lind et al. (2005). \(C3(i)\) is defined as the fraction of the number of triangles observed in the graph out of the total number of possible triangles which may appear. For a vertex \(i\) with a degree \(d_{i}\), the total number of

Figure 3: Queries and similarities associated with product _Lily’s 8-Inch Japanese 440C Pet Dog Chunker Shears_. This sub-graph shows three clusters that can be visually identified.

 

possible triangles is just the number of pairs of neighbors, i.e. \(d_{i}(d_{i}-1)/2\). Then \(C3\) is given by:

\[C3(i)=\frac{2T_{i}}{d_{i}(d_{i}-1)} \tag{1}\]

where \(T_{i}\) is the number of observed triangles incident on vertex \(i\).

If a vertex has a high (\(>0.33\)) C3 coefficient, we are assured that the neighborhood has a high internal degree (a large number of edges between vertices in the neighborhood), and so we include its neighbors in its initial cluster. Vertices not meeting this criteria are added as singleton initial clusters.

### Merging query clusters

We then merge clusters hierarchically, based on the size of the vertex intersection between clusters. Starting with the seed clusters (either neighborhoods or singleton vertices), in each round, pairs of clusters are chosen in order by intersection size. A round terminates when for some cluster pair \(c_{i},c_{j}\) to be merged, \(|c_{i}\cap c_{j}|<\theta\times\min(|c_{i}|,|c_{j}|)\) for some predefined hyperparameter \(\theta\),1 where \(|c_{i}|\) is the number of vertices in cluster \(c_{i}\).2

Footnote 1: We use \(\theta=0.4\).

Footnote 2: We can interpret the round termination condition as a modified Jaccard similarity, where our condition,

\(\frac{|c_{i}\cap c_{j}|}{\min(|c_{i}|,|c_{j}|)}<\theta\), is more amenable to clusters of dissimilar sizes than \(\frac{|c_{i}\cap c_{j}|}{|c_{i}\cup c_{j}|}\), the Jaccard coefficient.

### Pruning query clusters

Since we include entire neighborhoods as seed clusters, we may also end up including rogue edges that are tenuously connected to the clusters. After the final merge, we prune the vertices that have an internal-degree to external-degree ratio less than a threshold.3 This removes noise and bolsters the community structure (high intra-cluster connectivity and low inter-cluster connectivity).

Footnote 3: We use 0.5.

Since query graphs are behavioral, it may not be possible to get semantically distinct clusters. We observe that overly large clusters (\(|c|>10\)) include queries with multiple (albeit related) purchase intents. For example, in Figure 5, though mostly semantically grouped, we see a mix of _dog halloween costumes_ and _dog sweaters_ in one cluster. We thus only consider clusters with fewer than 10 vertices.

## 4 Experiments

From a low resource store we collect 2.4m instances of behavioral association between queries and products. We select behaviors over a threshold level of activity (e.g. clicks and purchases), comprising 0.83m pairs, and use that to train a baseline AL-XMC model; we call this data _local_. To apply query reformulation (_QR_), we first fine-tune a pre-trained T5-Base model for a sequence-to-sequence prediction task that uses query pairs from the clusters we generated in Section 3.4 We fine-tune for two epochs using batches of 32 intent-preserving query pairs and gradient accumulation with a factor of 16, for an effective batch size of 512. We

Figure 4: Different ways to cluster a graph

Figure 5: Due to the behavioral nature of the graph, some queries with different intents get grouped together. In this example, ‘dog elsa costume’ and ’dog dress warm’ are grouped together, yet represent different intents.

 

optimize with Adam, with a learning rate of \(3{\rm e}-4\). We fine-tune using 8 NVIDIA V100 GPUs; each epoch takes around 24 hours to run.

We use the fine-tuned T5 model to reformulate the query for each of the 2.4m collected query-product-behavior tuples, and associate the reformulation with the product and behavior, as if it was additional data. We can then re-select tuples with behavior over the same threshold level of activity used to form the local data; now, however, more tuples are above threshold and some new tuples have been introduced. Altogether, this procedure dramatically increases the size of the AL-XMC training corpus, to 16.14m training tuples.

In addition to the local baseline and local+QR reformulation approaches, we compare our data augmentation method to integrative knowledge transfer (IKT) Pan et al. (2008); Zhuo et al. (2008) from a higher-resource store. To do this, we identify queries in the high-resource store that show strong interaction with a product that is available in both stores. We add those query-product associations to the training set in the target store. Such an approach is of course only possible in a mature ecosystem where previously-established high-resource stores exist.

## 5 Evaluation

To intrinsically evaluate the impact of fine-tuning data size on reformulation intent preservation, we randomly select 300 QR tuples and determine to what degree intent is preserved during reformulation, using a variety of data conditions to fine-tune T5. We count a reformulation as accurate if it has the same manually judged purchase intent as the input query and a search engine returns a similar set of products for both queries. Table 1 shows that for the reformulation fine-tuned on the 460k pairs obtained as described in Section 3, accuracy is about 87%. To put this in context, Table 1 also shows results for T5 reformulation when fine-tuned on a 130k subset of that data, as well as on larger query reformulation fine-tuning data sets obtained by including query pairs from other high-resource stores. The trend indicates that intent preservation is generally a function of the amount of data used to fine-tune the T5 model. In the rest of this work we use a T5 reformulation model fine-tuned on the 460k pairs obtained as described in Section 3, since the availability of high-resource store data is not guaranteed.

To extrinsically evaluate, we consider the effectiveness of the AL-XMC models when built under different data augmentation conditions. We consider both offline and online evaluation paradigms.

### Offline evaluation

Typically, evaluations are done by computing precision and recall metrics using customer purchases as ground truth. Low-resource stores however, do not have sufficient purchase data and suffer from significant display bias--if a relevant product isn't shown to a customer, they cannot purchase it, resulting in an artificial drop in precision of a new (not deployed) model. Thus, we train crowd-sourced judges using the Toloka platform5 to evaluate if the products predicted by the AL-XMC model for a particular query are indicative of the query text. We use 1,000 randomly sampled query-product pairs to evaluate the models. Table 2 gives the product accuracy improvement and coverage (average number of products generated per query) for AL-XMC models. AL-XMC models trained using any data-augmentation (local+\(*\)) significantly outperform the model trained using only local data, both in terms of product accuracy (p-value in a one-sided t-test is < 0.01 for all three results) and coverage. Our QR data augmentation scheme outperforms IKT (local+IKT18) when the size of data augmentation is similar. Using multiple high-resource stores and hand-tuning the parameters for IKT, we can increase the training data by almost 3x to a total size of 47.58m. However, even with this massive data increase (local+IKT48), IKT only achieves a product accuracy comparable to our augmentation scheme. If no high-resource store is available, IKT is not even an option. Query reformulation, on the other hand, only requires information from the

\begin{table}
\begin{tabular}{l l} size & accuracy \\ \hline
130k & 79.67\% \\
460k & 86.67\% \\ \hline
4m & 90.33\% \\
40m & 93.00\% \\ \end{tabular}
\end{table}
Table 1: Impact of data size on fine-tuned T5 reformulation’s intent preservation, showing that accuracy generally correlates with data size. The 130k and 460k data points use only reformulated local data, while the 4m and 40m data points use data from other high-resource stores. Our QR experiments use the local-only 460k-tuned T5 model.

 

low-resource store and a language model like T5.

### Online evaluation

For the analysis of a single A/B test, it is common to compute a p-value from the t-statistic. However, in modern industrial settings, a large number of randomized experiments are run every day. In this setting, using only the Null Hypothesis test essentially ignores the information from the whole population of experiments. Stein's paradox (Stein, 1956) states that when estimating multiple parameters, there exist combined estimators more accurate on average than any method that handles the parameters separately. In the case of A/B tests, this means that the true effect of any one experiment can benefit by including the information from prior tests. Also, while the Null Hypothesis test is proper for testing whether the true effect is below or above zero, it is inconvenient to determine the true effect with respect to a loss/utility function. For our evaluations, we use an empirical Bayesian approach for analysis of large-scale experiments proposed by Guo et al. (2020), that uses the Normal-Normal model to determine the posterior probability of a positive return on a loss metric. Adding a risk buffer, we consider our effect positive if the posterior probability of a positive return, i.e., the treatment metric being greater than the control metric, is \(>0.66\).

We ran a 14 day A/B test on a popular e-commerce site in a low-resource English-speaking store, using the AL-XMC model trained with purchase data augmented with query reformulations as Treatment and the purchase data alone as Control, and observed a significant improvement in business metrics--Search Click Rate (SCR, the proportion of all searches that have at least one click) and Customer Reformulation Rate (CRR, the proportion of all searches where the customer had to reissue the query with different wording). A reduction in the CRR implies that the search returned relevant products in the first query. Table 3 shows a significant increase in SCR and a significant reduction (probability of positive effect < 0.33) in CRR.

## 6 Discussion

We compare the generated queries to those transferred by IKT48 and find that only 20k of the reformulated queries were not present in the IKT48 set. This indicates that the T5 model is able to generate high fidelity queries that customers are likely to use. In addition, data augmentation with query reformulations is able to achieve the same results as that with IKT48 with only about a third of the data. Thus, the reformulated queries are able to capture the relevant information of the target locale. Although IKT48 contains nearly the same information as QR, it is also noisier in the sense that it transfers queries that are relevant in the source (high-resource) store but may not be relevant in the target low-resource store. This noise necessitates several times more data to achieve the same performance. Model training with the larger IKT data set takes approximately 4 times as long as training with the query reformulations dataset.

## 7 Related Work

Data augmentation in the e-commerce space has dealt with reformulating queries in several ways.

\begin{table}
\begin{tabular}{l r r r} Training & Product accuracy & Avg number of & Num training \\ data & improvement & products/query & pairs \\ \hline local & N/A & 16.85 & 0.83m \\ local+IKT18 & 13.58\% & 22.27 & 18.11m \\ local+IKT48 & 19.85\% & 25.94 & 47.58m \\ local+QR (ours) & 19.57\% & 21.08 & 16.14m \\ \end{tabular}
\end{table}
Table 2: Data augmentation method comparison. Product accuracy (proportion of relevant products) improvement measures percent increase from local baseline. Product accuracy numbers use offline crowd sourced evaluations. p-value is ¡ 0.01 for all three data augmentation cases in a one-sided t-test.

\begin{table}
\begin{tabular}{c c r r r} Training data & Improvement & P(Treatment(SCR)\textgreater{}) & Improvement & P(Treatment(CRR)\textgreater{}) \\  & in SCR & Control(SCR)) & in CRR & Control(CRR)) \\ \hline local+QR & 0.26\% & 0.82 & -0.30\% & 0.15 \\ \end{tabular}
\end{table}
Table 3: Online A/B test where control is the local model. For Search Click Rate (SCR), positive effect is better, for Customer Reformulation Rate (CRR) negative effect is better.

 

Kuzi et al. (2016) expand user queries using semantically similar terms according to a learned embedding space, while Wang et al. (2021) use query annotations to identify words to expand queries. Both methods rely on a large amount of natural query reformulation by users to train their models, which we avoid. Zhang et al. (2015) and Wang and Yang (2015) propose synonym or paraphrase replacement reformulation approaches. The noisiness of these approaches, some of which expect significant context, are a bad fit for the product query use case. Other works that leverage data enhancement via model-based generation include Mao et al. (2021), who augment queries in black-box question answering tasks, Sennrich et al. (2016), who generate bilingual parallel data via backtranslation, and Fadaee et al. (2017) who, similar to us, though in a machine translation context, reformulate training data via language model-based generation, in this case focusing on rare word replacement.

## 8 Conclusion

In this paper, we have presented a method to augment training data for semantic models in low-resource e-commerce stores. Our method is generic enough to be applied as training data augmentation for any model and does not require transfer from high-resource store data. We have shown that augmenting training data using query reformulations improves upon a baseline store-specific AL-XMC semantic matching model in both offline evaluations as well as online business metrics. Our methods increase the training data of a test low-traffic store from 0.83m to 16.14m, resulting in a quality improvement of 19.57% over the baseline model. We also see a significant boost to business metrics in online A/B tests. Next we will explore similar data augmentation techniques for generating multilingual query reformulations, using the mT5 pretrained model Xue et al. (2021). In addition to low traffic stores, this technique may even be applied to yet-to-be-launched locales where training data is missing completely, by forming pseudo-queries from product descriptions.

## 9 Ethics Statement

We have performed our research so far only for the English language. Though we believe that similar results can be obtained for non-English languages we have yet to demonstrate this. We only use query and product interaction data for our work. Any identifiable user information is completely stripped before we can access the data. As our method is ultimately used to retrieve a set of products in an e-commerce store, incorrect predictions will not cause harm to the user besides an unsatisfactory experience.

## References

* J. M. Blei, R. K. Lee, C. L. Lin, and D. E. Silverman (2016)A survey of deep learning with text documents. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-10. Cited by: SS1.
* T. Joachims, H. Li, T. Liu, and C. Zhai (2007)Learning to rank for information retrieval (lr4ir 2007). SIGIR Forum41 (2), pp. 58-62. External Links: ISSN 0037-8695, Link, Document Cited by: SS1.
* X. Chen, Y. Gao, and W. Q. Weinberger (2016)An efficient approach for sentence parsing. In Proceedings of the European Conference on Empirical Methods in Natural Language Processing, pp. 1-10. Cited by: SS1.
* G. Cho, B. Choi, and F. He (2016)Semantic modeling of document parsing and inference in document parsing. In Proceedings of the International Conference on Empirical Methods in Natural Language Processing, pp. 1-10. Cited by: SS1.
* P. Huang, X. He, J. Gao, L. Deng, A. Acero, and L. Heck (2013)Learning deep structured semantic models for web search using clickthrough data. ACM International Conference on Information and Knowledge Management (CIKM). Cited by: SS1.
* N. Srivastava, I. Sutskever, and V. Lebedev (2014)Attention-aware neural networks. arXiv preprint arXiv:1412.6698. Cited by: SS1.
* K. Daihya, D. Saini, A. Mittal, A. Shaw, K. Dave, A. Soni, H. Jain, S. Agarwal, and M. Varma (2021)Deepxml: a deep extreme multi-label learning framework applied to short text documents. In Proc. 14th ACM Int'l Conf on Web Search and Data Mining, WSDM '21, pp. 31-39. Cited by: SS1.
* M. Fadaee, A. Bisazza, and C. Monz (2017)Data augmentation for low-resource neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Vancouver, Canada, pp. 567-573. Cited by: SS1.
* F. Richard Guo, J. McQueen, and T. S. Richardson (2020)Empirical bayes for large-scale randomized experiments: a spectral approach. arXiv: Methodology. Cited by: SS1.
* Po-Sen Huang, X. He, J. Gao, L. Deng, A. Acero, and L. Heck (2013)Learning deep structured semantic models for web search using clickthrough data. ACM International Conference on Information and Knowledge Management (CIKM). Cited by: SS1.
* T. Joachims, H. Li, T. Liu, and C. Zhai (2007)Learning to rank for information retrieval (lr4ir 2007). SIGIR Forum41 (2), pp. 58-62. Cited by: SS1.
* Saar Kuzi, A. Shtok, and O. Kurland (2016)Query expansion using word embeddings. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, CIKM '16, New York, NY, USA, pp. 1929-1932. External Links: ISBN 978-1-4503-3871-1, Link, Document Cited by: SS1.
* K. Lee, M. Chang, and K. Toutanova (2019)Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 6086-6096. Cited by: SS1.

 

* Lind et al. (2005) Pedro G. Lind, Marta C. Gonzalez, and Hans J. Herrmann. 2005. Cycles and clustering in bipartite networks. _Physical Review E_, 72(5).
* Linden et al. (2003) G. Linden, B. Smith, and J. York. 2003. Amazon.com recommendations: item-to-item collaborative filtering. _IEEE Internet Computing_, 7(1):76-80.
* Manning et al. (2008) Christopher D. Manning, Prabhakar Raghavan, and Hinriche Schutze. 2008. XML retrieval. In _Introduction to Information Retrieval_. Cambridge University Press.
* Mao et al. (2021) Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2021. Generation-augmented retrieval for open-domain question answering. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 4089-4100, Online. Association for Computational Linguistics.
* Nigam et al. (2019) Priyanka Nigam, Yiwei Song, Vijai Mohan, Vihan Lakshman, Weitian (Allen) Ding, Ankit Shingavi, Choon Hui Teo, Hao Gu, and Bing Yin. 2019. Semantic product search. In _Proceedings of the 25th ACM SIGKDD_, KDD '19, page 2876-2885.
* Volume 3_, AAAI'08, page 1383-1388. AAAI Press.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of Machine Learning Research_, 21(140):1-67.
* Robertson and Walker (1994) S. E. Robertson and S. Walker. 1994. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In _Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_, SIGIR '94, page 232-241, Berlin, Heidelberg. Springer-Verlag.
* Robertson and Zaragoza (2009) Stephen Robertson and Hugo Zaragoza. 2009. _The probabilistic relevance framework: BM25 and beyond_. Now Publishers Inc.
* Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 86-96, Berlin, Germany. Association for Computational Linguistics.
* Shen et al. (2020) Yanyao Shen, Hsiang-fu Yu, Sujay Sanghavi, and Indejit Dhillon. 2020. Extreme multi-label classification from aggregated labels. In _Proceedings of the 37th International Conference on Machine Learning_, ICML'20. JMLR.org.
* Stein (1956) Charles Stein. 1956. Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. In _Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics_, pages 197-206, Berkeley, Calif. University of California Press.
* Wang and Yang (2015) William Yang Wang and Diyi Yang. 2015. That's so annoying!!!: A lexical and frame-semantic embedding based data augmentation approach to automatic categorization of annoying behaviors using #petpeeve tweets. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 2557-2563, Lisbon, Portugal. Association for Computational Linguistics.
* Wang et al. (2021) Yaxuan Wang, Hanqing Lu, Yunwen Xu, Rahul Goutam, Yiwei Song, and Bing Yin. 2021. Queen: Neural query rewriting in e-commerce. In _The Web Conference 2021, Workshop on Knowledge Management in E-Commerce_.
* Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 483-498, Online. Association for Computational Linguistics.
* Volume 1_, NIPS'15, page 649-657, Cambridge, MA, USA. MIT Press.
* Zhuo et al. (2008) Hankui Zhuo, Qiang Yang, Derek Hao Hu, and Lei Li. 2008. Transferring knowledge from another domain for learning action models. In _PRICAI 2008: Trends in Artificial Intelligence_, pages 1110-1115, Berlin, Heidelberg. Springer Berlin Heidelberg.