SlowGuess/NAACL_2021_OCR huggingface dataset:
SemQA-CQA-DS: Our proposed model that was trained with CQA data and distantly supervised with predictions of syntactic match system on user queries as described in section 3.
We collect training data from the CQA section and user query logs for CA, DE, FR and IN locales of a popular e-commerce service. For each locale, to generate the CQA triplets and user query triplets (for distant supervision), we use data from CQA section of products, and user query logs and follow the sampling strategy described in section 3.3. The dataset statistics are described in table 2.
CQA Triplets	User Query Triplets
CA	5,317,904	1,063,580
DE	5,000,000	4,949,766
FR	1,500,000	173,258
IN	7,176,824	10,641,498
Table 2: Training data statistics
We use the bert-base-uncased $^2$  as the base transformer for our English models (for CA and IN locale), camembert-base (Martin et al., 2020) $^3$  as the base transformer for FR locale, and bert-base-multilingual-uncased $^4$  as the base transformer for DE locale. We train our models upto 10 epochs, with a batch size of 16, Adam optimizer with learning rate of  $2e - 5$  with a schedule of linear warmup of first 10000 steps and then linear decay. We set  $\epsilon = 1$  in the loss equation 2, and  $\alpha = 0.4$  in the inference equation 3. For the joint training (CQA triplets and user query triplets), we have two training runs (data mixing and multi-task as described in section 3.3) per locale and picked the best models (data mixing for CA, FR and multi-task for DE, IN). We use the Pytorch $^5$ , Huggingface (Wolf et al., 2019) and Sentence-Transformers (Reimers and Gurevych, 2019) libraries to develop our models on an Nvidia V100 GPU and hence our training time per batch and inference time per sample are same as that of Sentence-Transformers with BERT (base-model, 110M parameters).
4.3 Offline Evaluation
We do offline evaluation of our models under two settings: (1) on CQA test sets collected from the product pages at a popular e-commerce service, and (2) on user queries test set collected from query logs of the instant QA system on product pages of
<sup>1</sup>https://www.sbert.net/docs/pretrained_models.html
<sup>3</sup>https://huggingface.co/camembert-base
<sup>4</sup>https://huggingface.co/bert-base-multilingual-uncased
<sup>5</sup>https://pytorch.org
the same e-commerce service.Table 3 contains the test data statistics.
CQA Test Set	User Queries Test Set
#Questions	#Queries	#Query-Response Pairs
CA	2722	1485	5992
DE	2871	1351	5591
FR	2547	1762	5127
IN	2773	1459	4225
Evaluation on CQA Dataset: The goal of this section is to evaluate the relevance between community questions and answers learned by different approaches. For all locales we randomly sample questions posted on product pages. The paired answers to those questions are considered to be relevant answers and all other answers (from other CQA pairs) of the product are assumed to be irrelevant answers. We only sampled products that at least have 5 CQA pairs posted. For each question, the task is to rank all the candidate answers according to relevance. We report precision@1  $(\mathrm{P}@\mathrm{1})$ , mean average precision (mAP) and mean reciprocal rank (MRR) in table 4. Since there may be multiple paired answers to a community posted question, the rank (for MRR) of a relevant answer is the number of irrelevant answers ranked above it plus one. We observe that both SemQA-CQA and SemQA-CQA-DS are able to significantly outperform other methods. This is expected since both of these methods were trained using CQA data and hence is able to learn a good QA relevance function, whereas the sentence-transformers-STS-NLI were trained using STS and NLI tasks and they failed to generalize. However, CQA pairs are significantly different from the language of user queries and in the next section, we will evaluate on those queries (the main goal of this paper).
Evaluation on User Queries: To evaluate on user queries, we sample user queries (and their corresponding top responses) uniformly at random from the query logs of the instant QA system. We also retrieve the top responses generated by the different models we trained. These query, response pairs are labeled as relevant or irrelevant by a team of human annotators. We use the area under the precision recall curve (PR-AUC) as our quality metric. We report the absolute percentage points
Table 3: Test data statistics
M0	M1	M2	M3
P@1	CA	39.02	52.87	74.10	73.55
DE	40.82	38.66	73.04	71.65
FR	37.42	42.25	73.85	75.34
IN	26.51	35.67	53.05	53.62
mAP	CA	41.02	51.23	73.37	72.46
DE	45.24	43.35	74.80	73.04
FR	41.24	44.46	74.27	75.38
IN	43.93	51.12	72.17	71.89
MRR	CA	31.21	42.41	65.17	64.17
DE	37.05	35.48	68.30	66.20
FR	32.26	35.03	66.78	67.37
IN	34.70	41.23	58.44	58.46
Table 4: Evaluation on CQA pairs. M0: BM25, M1: sentence-transformers-STS-NLI, M2: SemQA-CQA, M3: SemQA-CQA-DS.
change in PR-AUC with respect to the E-commerce Baseline in table 6 (+ve sign implies PR-AUC has improved and -ve sign implies PR-AUC has decreased). We make the following observations: (1) the vanilla BM25 baseline performs the worst which is expected as it relies solely on syntactic matches and fails to capture semantic intent; (2) both the sentence-transformers-STS-NLI and our SemQA-CQA models fail to generalize validating our hypothesis that learning a general semantic matching model or a QA relevance model is not sufficient to learn the nuances of user querying language; (3) the SemQA-CQA-DS models significantly outperform all other models. There are two underlying reasons for these improvements. Firstly, SemQA-CQA-DS is able to leverage the semantic understanding capabilities (that Pretrained Transformers and SemQA-CQA are also able to do), and secondly, SemQA-CQA-DS is also able to learn the nuances of the task specific query language leading to a better relevance model between user queries and CQA pairs (that are potential candidate responses).
Next, we do a qualitative analysis on the cases where SemQA-CQA-DS is able to improve on the E-commerce Baseline. We identify two main areas of improvement: (1) improving relevance in cases where the baseline fails to capture the semantic intent, and (2) improving coverage in cases where the baseline fails to retrieve any response. We present examples of both cases in table 5. The examples include cases where the language is ill-formed and incoherent and our distantly supervised
User query	Top CQA pair retrieved by SemQA-CQA-DS	Top CQA pair retrieved by E-commerce Baseline
Improving semantic relevance
Do you have size variation??? Like i need this in bigger wood..	Q: Is this available in still large size
A: yes ... available size is 7*5,8*6,9*7,12*9 in inches	Q: Is the wood and print waterproof?
?
A: YES
It is compatible in gaming	Q: Does it run gta v
A: Yesss... Very fine	Q: Is it compatible with Amd A6 processor ?
A: Yes it's compatible DDR4
Total weight	Q: Each 1 how to kgs
A: 10 kgs	Q: Total diameter of the plates?
A: Plate Dia is 9.5 inches Hole Dia is 30 mm
Improving coverage
What is fabric	Q: Which material is the scarf made up of
A: It is like soft satin silk	No response
The dress with hands or seelveless	Q: Is it sleeveless
A: we give a extra sleeves so u can attach or not..as ur wish	No response
Bettry perfon	Q: Batrrey capictiy
A: This Phone has a Wonderful 4000 Mah Battery with Battery Saver Options & Can Watch videos continuously for 18 Hours!!!	No response
Table 5: Qualitative examples.
M0	M1	M2	M3
CA	-19.75	-1.11	+1.53	+9.25
DE	-13.03	-11.90	+5.46	+12.15
FR	-11.66	-4.54	+4.93	+7.68
IN	-16.66	-0.26	+0.39	+4.37
Table 6: PR-AUC on user queries evaluation set. M0: BM25, M1: sentence-transformers-STS-NLI, M2: SemQA-CQA, M3: SemQA-CQA-DS. Numbers denote the absolute percentage points change with respect to the E-commerce Baseline.
model still captures the intent and retrieve relevant responses.
4.4 Online Evaluation
We also ran a large scale online A/B experiment with  $50\%$  of the user traffic. All locales were experimented at least for two weeks to ensure diversity in periodic patterns and have enough queries to achieve statistically significant conclusions (p-values  $< 0.01$  in Chi-Square tests) about the improvement in metrics. Here the SemQA-CQADs model is used to complement the existing E-commerce Baselinen to improve the coverage of
the system. There are two metrics of interest: (1) the coverage (percentage of queries answered by the system), and (2) the new question asking rate (percentage of queries for which even after seeing the response, a user asks a question in the CQA forum; if the relevance of the answers improves, the question asking rate should decrease). We report the change in absolute percentage points with respect to the E-commerce Baseline (for coverage +ve is better, and for question asking rate -ve is better). The results are present in table 7. SemQACQA-DS was able to improve coverage while reducing the rate of new questions posted by users in all locales thereby showing the efficacy of our approach at scale.
Coverage	Question Asking Rate
CA	+2.96	-0.69
DE	+3.12	-0.44
FR	+4.56	-1.60
IN	+6.92	-0.97
Table 7: A/B test evaluation. Numbers denote the absolute percentage points change of Treatment with respect to Control.
<sup>6</sup>Details can't be disclosed due to proprietary information
5 Conclusions & Future Works
In this paper we presented 'SemQA', a practical transformer-based framework to provide instant QA efficiently on the product pages of e-commerce services. Given a user query, our framework directly retrieves the relevant CQA pairs from the product page, where user queries and CQA pairs have significantly different language characteristics. Our model is able to learn a robust relevance function between user queries and CQA pairs by learning representations that leverage the strengths of both syntactic and semantic features, without the need for any explicit human labeled data. Our model is able to scale to large scale real-time e-commerce systems and at inference time only requires model encoding of user queries for by index lookups, and candidate responses are encoded offline into the index in a space efficient manner. Extensive offline evaluation shows our approach generalizes to multiple locales spanning different languages with a PR-AUC gain by upto  $12.15\%$  over the existing system at a popular e-commerce service. We also ran a large scale online A/B experiment with  $50\%$  of the user traffic and our framework was able to improve coverage by upto  $6.92\%$  by complementing the existing system.
As a future direction, we would like to expand our SemQA system to include responses from additional content on the product pages (reviews, descriptions etc.). We believe some of the existing approaches to leverage reviews (discussed in section 2) can be used to complement our system to expand our relevance model beyond CQA data. Another direction of research will be to include features such as accuracy, sentiment, freshness etc. within our proposed SemQA system's responses.
We thank all the anonymous reviewers for providing their valuable comments that helped us improve the quality of our paper. We also thank our colleagues in the science, product, and engineering teams at Amazon for their valuable inputs.
B. Bi, Chen Wu, Ming Yan, Wei Wang, Jiangnan Xia, and Chenliang Li. 2019. Incorporating external knowledge into machine reading for generative question answering. In EMNLP/IJCNLP.
Kurt D. Bollacker, C. J. Evans, Praveen Paritosh, Tim
Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In SIGMOD Conference.
R. Burke, K. Hammond, Vladimir A. Kulyukin, S. Lytinen, Noriko Tomuro, and Scott Schoenberg. 1997. Question answering from frequently asked question files: Experiences with the faq finder system. AI Mag., 18:57-66.
Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio. 2009. Large scale online learning of image similarity through ranking. In IbPRIA.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In ACL.
L. Chen, Ziyu Guan, W. Zhao, Wanqing Zhao, Xiaopeng Wang, Zhou Zhao, and Huan Sun. 2019a. Answer identification from product reviews for user questions by multi-task attentive networks. In AAAI.
Shiqian Chen, Chenliang Li, Feng Ji, W. Zhou, and Haiqing Chen. 2019b. Review-driven answer generation for product-related questions in e-commerce. Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining.
Yang Deng, Wenxuan Zhanng, and Wai Lam. 2020. Opinion-aware answer generation for review-driven question answering in e-commerce.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT.
Shen Gao, Z. Ren, Yihong Zhao, Dongyan Zhao, D. Yin, and Rui Yan. 2019. Product-aware answer generation in e-commerce question-answering. Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining.
Mansi Gupta, Nitish Kulkarni, Raghuveer Chanda, Anirudha Rayasam, and Zachary Chase Lipton. 2019. Amazonqa: A review-based question answering task. In IJCAI.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. ArXiv, abs/1503.02531.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9:1735-1780.
Ashish Kulkarni, Kartik Mehta, Shweta Garg, Vidit Bansal, Nikhil Rasiwasia, and Srinivasan Sengamedu. 2019. Productqna: Answering user questions on e-commerce product pages. In Companion Proceedings of The 2019 World Wide Web Conference, pages 354-360.
Sawan Kumar, Shweta Garg, K. Mehta, and Nikhil Rasiwasia. 2019. Improving answer selection and answer triggering using hard negatives. In EMNLP/IJCNLP.
M. Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, A. Mohamed, Omer Levy, V. Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In ACL.
Christopher D Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to information retrieval. Cambridge university press.
Louis Martin, B. Muller, Pedro Javier Ortiz Suárez, Y. Dupont, L. Romary, 'Eric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020. Camembert: a tasty french language model. ArXiv, abs/1911.03894.
Julian McAuley and A. Yang. 2016. Addressing complex and subjective product-related queries with customer reviews. ArXiv, abs/1512.06863.
Bhaskar Mitra and Nick Craswell. 2019. An updated duet model for passage re-ranking. ArXiv, abs/1903.07666.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,  $000+$  questions for machine comprehension of text. In EMNLP.
Nils Reimers and Iryna Gurevych. 2019. Sentencebert: Sentence embeddings using siamese bert-networks. In EMNLP/IJCNLP.
Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. ArXiv, abs/2004.09813.
S. Robertson, S. Walker, Susan Jones, M. Hancock-Beaulieu, and Mike Gatford. 1994. Okapi at trec-3. In TREC.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. The 2015 IEEE conference on computer vision and pattern recognition.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS.
E. Voorhees and Dawn M. Tice. 1999. The trec-8 question answering track evaluation. In TREC.
Jiang Wang, Yang song, Thomas Leung, Chuck Rosenberg, Jinbin Wang, James Philbin, Bo Chen, and Ying Wu. 2014. Learning fine-grained image similarity with deep ranking. The 2014 IEEE conference on computer vision and pattern recognition.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Yi Yang, Wen tau Yih, and Christopher Meek. 2015. Wikiqa: A challenge dataset for open-domain question answering. In EMNLP.
Shiwei Zhang, Joy Han Lau, Xiuzhen Zhang, Jeffrey Chan, and Cecile Paris. 2019. Discovering relevant reviews for answering product-related queries. 2019 IEEE International Conference on Data Mining (ICDM), pages 1468-1473.
Shiwei Zhang, Xiuzhen Zhang, Joy Han Lau, Jeffrey Chan, and C. Paris. 2020a. Less is more: Rejecting unreliable reviews for product question answering. ArXiv, abs/2007.04526.
Wenxuan Zhang, Yang Deng, and Wai Lam. 2020b. Answer ranking for product-related questions via multiple semantic relations modeling. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval.