SlowGuess/ACL_2020_OCR huggingface dataset:
Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. arXiv preprint arXiv:1804.06059.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. arXiv preprint arXiv:1707.07328.
Dongyeop Kang, Tushar Khot, Ashish Sabharwal, and Eduard Hovy. 2018. Adventure: Adversarial training for textual entailment with knowledge-guided examples. arXiv preprint arXiv:1805.04680.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Volodymyr Kuleshov, Shantanu Thakoor, Tingfung Lau, and Stefano Ermon. 2018. Adversarial examples for natural language classification problems.
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119.
Bhaskar Mitra, Fernando Diaz, and Nick Craswell. 2017. Learning to match using local and distributed representations of text for web search. In Proceedings of the 26th International Conference on World Wide Web, pages 1291-1299. International World Wide Web Conferences Steering Committee.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. 2018. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957.
Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2016. Text matching as image recognition. In Thirtieth AAAI Conference on Artificial Intelligence.
Nicolas Papernot, Patrick McDaniel, Ananthram Swami, and Richard Harang. 2016. Crafting adversarial input sequences for recurrent neural networks. In MILCOM 2016-2016 IEEE Military Communications Conference, pages 49-54. IEEE.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kocisky, and Phil Blunsom. 2015. Reasoning about entailment with neural attention. arXiv preprint arXiv:1509.06664.
6866
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2013. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199.
Jun Wang, Lantao Yu, Weinan Zhang, Yu Gong, Yinghui Xu, Benyou Wang, Peng Zhang, and Dell Zhang. 2017. Irgan: A minimax game for unifying generative and discriminative information retrieval models. In Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 515-524. ACM.
Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. 2017. End-to-end neural ad-hoc ranking with kernel pooling. In Proceedings of the 40th International ACM SIGIR conference on research and development in information retrieval, pages 55-64. ACM.
A.1 LSTM Classifier
We adapt our classifier from that for textual entailment in (Rocktäschel et al., 2015), but with a few key differences. Unlike standard textual entailment problems for natural language, user queries and item titles tend to follow different language patterns, with both of them being different from "natural" language. For example, queries "red bike running shoes", "running bikes shoes, red" and "red running shoes bike" all refer to the same general product, despite differing in structure. On the other hand, item titles are structured, with brand, size, color, etc. all mentioned in a long sequence, which is also not how a conventional sentence is structured. To account for these differences between query strings and item titles, we separately train word embeddings using word2vec (Mikolov et al., 2013) on anonymized query logs and item titles. Thus, the same word can have two embeddings, one for the query and one for the title. The overall classifier structure is shown in Figure 5
We implement the word-by-word attention layer as follows: Let  $k$  be the output dimension of the LSTMs,  $K \in \mathbb{R}^{k \times m}$  and  $H = [h_1, h_2, \dots, h_n] \in \mathbb{R}^{k \times n}$  be the LSTM output matrices for the item title and query respectively, with the  $i^{th}$  column corresponding to the output of the  $i^{th}$  LSTM cell. Let  $m$  and  $n$  denote respective lengths of title and query sequences.
Figure 5: LSTM based classifier for a query-item pair (best seen in color). The LSTMs are fed word embeddings, separately learned for queries and titles. The word-by-word attention layer is adapted from (Rocktäschel et al., 2015), and  $h^*$  is defined in (2)
For each word  $t = 1,2,\ldots ,n$  in the query, we compute attention scores for every word in the title and its weighted representation  $r_t$  at that step. The representation  $r_{t - 1}$  is helpful to inform the next step what the model previously paid more focus on. We use the additive attention (Bahdanau et al., 2014) here, but other alternatives can be used as well.
$$
M _ {t} = \tanh  ([ K ^ {\top}, 1 h _ {t} ^ {\top}, 1 r _ {t - 1} ^ {\top} ] W _ {h}),
$$
$$
\alpha_ {t} = \tanh (M _ {t} w), \quad w \in \mathbb {R} ^ {k}
$$
$$
r _ {t} = K ^ {\top} \alpha_ {t} + \mathrm {t a n h} (W _ {r} r _ {t - 1}).
$$
where  $W_{h}\in \mathbb{R}^{3k\times k}$  and  $W_{r}\in \mathbb{R}^{k\times k}$ . The final representation for the query and title that is passed to fully-connected layers is:
$$
h ^ {*} = \tanh  \left(W _ {x} \left[ r _ {n} ^ {\top}, q _ {n} ^ {\top}, \left| r _ {n} - q _ {n} \right| ^ {\top} \right]\right), \tag {2}
$$
where  $W_{x} \in R^{k \times 3k}$ . In the above equations,  $W_{h}, w, W_{r}$  and  $W_{x}$  are weight matrices to be learned, and  $q_{n}$  is the output of the LSTM that encodes the query. Passing  $|r_{n} - q_{n}|$  in (2) to the dense layers improves classification performance. We observe in mismatched query-item pairs that a slight word substitution or deletion often leads to mismatched items; for example, "iPhone screen protector" and "iPhone screen" or "iPad screen protector" are textually very similar, but are completely different items from a shopping point of view. Hence, we use the term  $|r_{n} - q_{n}|$  in (2) to explicitly account for such word changes. Traditional sentence classification methods also pass
6867
$r_n \circ q_n$  to the dense layers, where  $0$  is the hadamard product. We noticed that this did not improve the model performance, and hence choose to not use it. A desirable side effect is reduced computations. We expect that  $|r_n - q_n|$  somehow captures words that are in the query but not in the title and vice versa.
Let  $f_{\theta}(\cdot)$  denote the classifier in Figure 5. Given  $N$  samples  $\{(x_i, y_i)\}_{i=1}^N$ , our objective function is a weighted cross-entropy loss:
$$
\begin{array}{l} L _ {\theta} (X, y) = \frac {1}{N} \sum_ {i = 1} ^ {N} L _ {\theta} \left(x _ {i}, y _ {i}\right) \tag {3} \\ = \frac {1}{N} \sum_ {i} \beta y _ {i} \log \left(f _ {\theta} \left(x _ {i}\right)\right) \\ + \left(1 - y _ {i}\right) \log \left(1 - f _ {\theta} (x _ {i})\right), \quad (4) \\ \end{array}
$$
where  $\beta$  adjusts the weight on the positive samples. We set  $\beta > 1$  to account for the fact that the number of positive samples (i.e. mismatched) is much larger than negative samples in our datasets.
A.2 Variational Query Generator
For the applications we are interested in, the training datasets are highly unbalanced, as a reasonable search engine will have far more matched query-item examples than mismatched. We thus need ways to account for this class imbalance. Generating trivially mismatched examples is easy: we can randomly sample an item from the entire catalog for a given query. But these will be examples that are easy-to-classify for  $f_{\theta}(\cdot)$ , and are hence uninformative. Here we aim to train a model that can generate hard-to-classify mismatched examples, which tend to occur due to the query and product title being lexically similar. Specifically, we want to generate mismatched query-item examples that have a realistic chance of appearing in the search results for said query.
We train a Variational Encoder-Decoder (VED) model to this end. The model takes as input a matched pair  $(I, Q)$ , and outputs a new query  $Q_{\text{gen}}$  so that the pair  $(I, Q_{\text{gen}})$  is mismatched, but being lexically similar to  $Q$ . As an example, if  $I = \text{puma}$  running shoe, size 11, black and  $Q = \text{running shoes for men}$ , we can generate  $Q_{\text{gen}} = \text{insoles for running shoes. In this case, } Q_{\text{gen}} \text{ is similar to } Q \text{ in that the item is somewhat related to } Q_{\text{gen}}$ , and there's a chance that  $I$  may be matched to  $Q_{\text{gen}}$  due to keyword stuffing by sellers, or poor semantic
matching. On the other hand, another mismatched query  $Q_{\text{gen}} = \text{pizza cutter}$  is not a good candidate to generate, since it's highly unlikely that a reasonable search engine will show shoes for a query about pizza cutters.
Figure 6: Variational encoder-decoder query generator (best seen in color). The encoder is reused from the classifier in the previous section. The decoder is an LSTM with attention (Luong et al., 2015).  $Q_{\mathrm{gen}}$  is a generated query via beam search.
To train the model, we make use of an labeled  $\{(I,Q,y)\}$  dataset and create a new one as follows: we consider only those items  $I$ , for which there exist both matched and mismatched queries, and construct samples  $(I,Q,Q_{\mathrm{mis}})$  so that  $y(I,Q) = 0$  and  $y(I,Q_{\mathrm{mis}}) = 1$ . The model is the variational sequence to sequence model proposed in (Bahuleyan et al., 2017), which we adapt to our case (Figure 6). Our architecture can reuse the title-query encoder of the classifier in Section A.1. The variational decoder allows us to generate diverse output sequences for the same input. We equip the decoder with an attention mechanism (Luong et al., 2015) to generate  $Q_{\mathrm{gen}}$ . Using an existing annotated dataset to pretrain the VED allows us to accurately warm start the end to end model described in the next section.
Source	Targets
kate spade yoga mat	kate spade
kate spade sale
kate spade wallet
dickies overalls striped	dickies work pant
32x32 mens dickies shorts
kahki overalls for women
puppy training 101	dog training pad
dog training collar
puppy
plastic stacking bins	stackable storage bins
storage bins
foldable storage bins
Table 5: Nearest neighbors by cosine similarity for a few queries. Note that the LSTM and mean-pooling method accurately represents queries based on various intents. In the first case, customers looking for Kate Spade items tend to look for more items of the same brand. In the second case, the model groups queries with similar intents together. In the last 2 cases, the model groups similar queries together.
6868
source
target 1	screen replacement for iphone 7 plus in white including all tools instruction 2 screen protectors
target 2	iphone 7 plus screen replacement whiteLCD display 3d touch screen digitizer frame assembly white
target 3	iphone 6 screen replacement white p zone 4 7 inchLCD display touch screen digitizer frame assembly for iphone 7 screen replacementLCD touch screen digitizer frame assembly full set
source	imagine by rubie s dc superheroes harley quinn mallet costume
target 1	rubie s harley quinn mallet costume accessory
target 2	rubie s women s suicide squad harley quinn mallet as as shown one size
target 3	rubie s women s batman harley quinn inflatable mallet multi one size
source	nike unisex core golf visor dark grey anthracite white one size
target 1	nike golf unisex legacy91 hat white black one size
target 2	nike men s flex core golf shorts dark grey dark grey size 36
target 3	nike golf tech visor black adjustable one size
source	mercer culinary genesis 6 piece forged knife block set tempered glass block
target 1	dalstrong knife set block gladiator series knife set german hc steel 8 pc
target 2	j a henckels international 13550 005 statement knife block set 15 pc light brown
target 3	top chef by master cutlery 5 piece chef basic knife set with nylon carrying case
source	hicksholsters purple dark punisher edition wallet
target 1	hicksholsters kydex dark punisher edition wallet
target 2	silk iphone 6 6s wallet case vault protective credit card grip cover wallet slayer vol 1 black onyx
target 3	kalmore genuine leather RFID protected slim thin pocket wallet minimalist wallet money clip light blue
Table 6: We show nearest neighbors by cosine similarity based on embedding for a few item titles. In each cell, the first line (bolded) represents the source, and the next 3 lines represent its three nearest neighbors. Unlike the query case, the nearest neighbors are always substitutable items.
A.3 Learned LSTM Embeddings
Next, we verify that the learned query and item embeddings from the LSTM models are informative. To compute a vector representation of a query, we mean-pool the query LSTM outputs  $H$ . Similarly, for an item, we mean-pool the item LSTM outputs  $K$ . Table 5 shows the 3-nearest neighbors for user queries. The neighbors are computed based on the cosine similarity between the embeddings of the source and target query. We can see that, depending on the specific query, the model learns to group queries that have the same product intent, or brand (for higher-end) items.
Along the same line, Table 6 shows the 3-nearest neighbors in terms of cosine similarity for items. Note that this case is not the same as queries, since an item by itself is meaningless. Indeed, the outcome of the word by word attention model is to achieve an item representation conditioned on the query. More specifically, the item nike running shoe by itself cannot be deemed as matched or mismatched, unless seen in the context of a user typed query. Hence, the LSTM + mean pooling output for the items tend to cluster similar (substitutable) items together. The upshot if this is, conditioned on a given query  $Q$ , the item embeddings for similar items will be similar, which is a desirable outcome for our use case.
A.4 Related Work
The DSSM (Huang et al., 2013) model and its variants (Mitra et al., 2017; Xiong et al., 2017) have
been commonly applied in learning to rank tasks. Such models are useful for web search, where there are several related documents and it's easier for natural language based models to distinguish between related and unrelated documents. These models do not easily carry over for product search, due to the issues alluded to in the previous section. Recently, (Kang et al., 2018) developed means to generate adversarial samples to improve entailment, via the use of additional datasets to learn "rules" to aid in sample generation. These rules do not carry over to the product search domain, nor do the assumption of existing datasets to learn such rules. To the best of our knowledge, we are the first to work on generating adversarial representations of text for the purpose of improving product relevance for e-commerce.
Adversarial example generation has been studied in the context of images (Szegedy et al., 2013; Chen et al., 2018), speech (Carlini and Wagner, 2018) and text (Cheng et al., 2018; Ebrahimi et al., 2017; Kuleshov et al., 2018; Iyyer et al., 2018; Papernot et al., 2016; Kang et al., 2018; Wang et al., 2017). In (Alzantot et al., 2018; Ebrahimi et al., 2017), the authors develop a means to perturb the sequence in order to fool an underlying classifier, and in (Iyyer et al., 2018), the authors use the concept of backtranslation (Sennrich et al., 2015). The aims in these works is to generate adversarial text samples themselves, separate from generating samples that will make the underlying classifier more robust.
6869