SlowGuess/NAACL_2021_OCR huggingface dataset:
Yes/No Answer Labeling Questions are typically answered by free-text answers, posted independently by multiple users. In order to convert these answers into a single yes/no answer, we first classified each answer into one of three classes: yes, no and maybe, and then used majority vote among the classified answers. We used a pre-trained RoBERTa-based classifier, and trained the model on McAuley's dataset (McAuley, 2016), taking only yes/no questions. See Appendix A for details.
We experiment with eleven product categories covered by our Amazon-PQA dataset (Section 4.2), training a SimBA answer prediction model for each of the categories independently. Next, we describe the data preparation steps for each of the SimBA components.
Sibling Retrieval Using AKNN For each record  $r \in C$  ( $C$  is the category dataset), we use AKNN to retrieve the top- $K$  similar siblings from  $C$ , while
Majority vote	Jaccard similarity	USE cosine	RoBERTa cosine	Q2Q
53.1	62.0	69.6	70.7	83.2
Table 3: Classification accuracy of question similarity models.
making sure that neither of them share the same product with  $r$ . We collect training example pairs by coupling each record  $r$  with each of its siblings:  $D^{\prime}(C) = \bigcup_{r_i\in C}\{(r_i,r_j)|r_j\in S(r_i)\}$ .
For retrieval we use Universal Sentence Encoder (USE) (Cer et al., 2018) to embed each question  $q_{i}$  into a 512-length vector  $e_{i}$ . We use the Annoy<sup>11</sup> python library for the implementation of efficient AKNN retrieval. In all experiments, for each record we retrieve the top-K ( $K = 500$ ) similar records, based on the cosine-similarity between the embedding vectors.
Twin Detection Using the Q2Q Model For each sibling pair  $(r_i, r_j) \in D'(C)$ , we use our Q2Q model to score their question-similarity and keep only those with  $Q2Q(q_i, q_j) > \gamma$  to yield a collection of twin pairs,  $D(C)$ . We use  $\gamma = 0.9$  to ensure only highly similar question pairs.
For our Q2Q model, we apply a standard pretrained RoBERTa (Liu et al., 2019) classifier. Specifically, we use Hugging-Face base-uncased pre-trained model $^{12}$  and fine-tune $^{13}$  it for the classification task on our Q2Q dataset $^{14}$ , while splitting the data into train, dev and test sets with  $80\% - 10\% - 10\%$  partition, respectively. For  $\gamma = 0.5$  (its minimal value) the model achieves test accuracy of  $83.2\%$  with a precision of  $81.3\%$  and a recall of  $87.7\%$ . When setting the twin confidence level threshold to  $\gamma = 0.9$ , the precision of the Q2Q model raises to  $89.9\%$  with a recall of  $69.5\%$ .
We compare the performance of the Q2Q similarity classifier with several unsupervised baselines, namely: (a) Jaccard similarity, (b) cosine similarity over USE embedding, and (c) cosine similarity over RoBERTa $^{15}$  embedding. The results are summarized in Table 3, showing that the Q2Q model significantly outperforms these baselines.
11https://github.com/spotify/annoy
<sup>12</sup>https://github.com/huggingface/transformers
13We use batch size 32, maximum sequence length of 128, learning rate 5e-5, and 3 epochs.
14We only used the examples with full agreement.
$^{15}$ Hugging-Face sentence-transformers roberta-large-nli-stsb-mean-token model.
Category	Acc	Majority	Δ
Light Bars	73.9	61.1	+12.8
Monitors	78.2	68.2	+9.9
Smartwatches	80.0	65.6	+14.4
Receivers	77.5	67.6	+9.9
Backpacks	83.9	76.0	+7.9
Jeans	71.3	59.3	+11.9
Beds	84.6	72.0	+12.6
Home Office Desks	73.9	63.4	+10.5
Masks	75.1	66.9	+8.2
Posters & Prints	72.3	60.9	+11.5
Accessories	79.1	72.4	+6.6
Macro Average	77.2	66.7	+9.7
Table 4: CPS model test set results on the CPS auxiliary datasets and the majority baseline of each category.
5.2 CPS Model
Training The CPS model predicts the contextual similarity between a pair of twin records. In our experiments, the textual content of a product consists of the product title concatenated with the product bullet points, separated by semicolons. The question text is the original query as appeared in the Amazon PQA-dataset. For the encoding modules of the CPS model we use a standard pre-trained RoBERTa-based model as well, while using the [SEP] token for separating the two inputs to each encoder. For training, twin pairs are labeled according to their contextual similarity using Equation 2.
We train, fine-tune, and test, an independent CPS model for each category set  $C$ , using  $D(C)$ ,  $D_{dev}(C)$ , and  $D_{test}(C)$  (details of the data split described in Appendix A). The training set  $D(C)$  is created as described in Section 5.1.  $D_{dev}(C)$  and  $D_{test}(C)$ , are created the same with one modification - rather than retrieving the siblings for a record from the dataset it belongs to, the siblings are retrieved from  $D(C)$ , for both  $D_{dev}(C)$ , and  $D_{test}(C)$ . This represents a real-world scenario where existing products with their related questions are used as a corpus for predicting the answer to a question about a new product. Each product with all related questions appear only in one of these sets.
Evaluation We evaluate the CPS model by measuring the accuracy of its contextual similarity prediction over  $D_{test}(C)$ . The accuracy per category is presented in Table 4. The model achieves a relatively high accuracy with a macro average of  $77.2\%$  over all categories, presenting a significant lift of  $9.7\%$  over the majority decision baseline. This is an encouraging result, considering the fact that the
answers for many questions cannot be directly inferred from the product textual information. We conjecture that the model is able to learn the affinity between different products, in the context of a given question, for predicting their contextual similarity. For example, the two backpacks Ranwoo Laptop Backpack and Swiss Gear Bungee Backpack, were correctly classified by the CPS model as similar  $(\psi \geq 0.5)$  in context of the question "Will this fit under a plane seat?", and classified as different  $(\psi < 0.5)$  in context of the question "Does it have a separate laptop sleeve?"
5.3 Answer Prediction Methods
We experiment with our SimBA model and with a few baselines over the test set of all categories. The first one is Majority which returns the majority answer among all records in the category. Other methods are described next.
SimBA Given a target record  $r_t$ , SimBA scores each of its twins by the CPS model and predicts the answer for  $q_t$ , using Equation 3.  $w_{min}$  was finetuned on the combined dev set of all categories and was set to 0.38.
Question Similarity Only (QSO) We modify the SimBA model to ignore the CPS classification score when implementing the Mixture-of-Experts model (Eq. 3), by setting an equal weight of 1.0 to all twin votes:  $Pred(a_{t}|r_{t}) = \text{sign}\left(\sum_{r_{j} \in T(r_{t})} \delta(a_{j})\right)$ .
Product Similarity Only (PSO) We modify the SimBA model by setting  $q_{t}$  and  $q_{j}$  to empty strings at the input of the CPS model, both during training and during inference, forcing it to rely on the products' textual content alone. The twin retrieval process remains untouched.
Answer Prediction Classifier (APC) We experiment with a direct prediction approach that only considers the product textual content and the question for answer prediction. For each category  $C$ , we fine-tune a pre-trained RoBERTa-based classifier over all records  $r_j \in C$ , using  $q_j$  and  $p_j$  (separated by the [SEP] token) as input and  $\delta(a_j)$  as the training label.
SimBA+APC The experimental results show that different answer-prediction methods (e.g. SimBA vs APC) may be preferable for different product categories. Therefore, we combine both methods, for achieving optimal results, by mixing
# Twins	Answer
(Monitors) Does this require WiFi?	51	91% No
(Backpacks) Will it fit under a plane seat?	213	90% Yes
(Smartwatches) Can it measure blood sugar level?	34	97% No
(Smartwatches) Does it come with a charger?	269	96% Yes
Table 5: Examples for popular questions with high answer agreement over different products.
Figure 3: Macro-average test accuracy over all categories. The horizontal axis indicate the minimal number of twins and the percentages of test questions each subset represents.
the vote of APC with the twin votes, using the Mixture-of-Experts approach:
$$
\begin{array}{l} \operatorname {P r e d} \left(a _ {t} \mid r _ {t}\right) = \\ \operatorname {s i g n} \left(\eta (r _ {t}) \delta (\alpha_ {t}) + \sum_ {r _ {j} \in T (r _ {t})} \lambda (r _ {j}) \delta (a _ {j})\right) \\ \end{array}
$$
where  $\alpha_{t}$  is the APC predicted answer, and  $\eta (r_t) =$ $\eta_1,\eta_2$  and  $\eta_{3}$  for  $|T(r_t)|\leq 10,10 <   |T(r_t)| <   50$  and  $|T(r_t)|\geq 50$  , respectively<sup>16</sup>. All  $\eta$  values  $(\eta >0)$  are fine-tuned on the development set for each category separately. The values we used are detailed in Table 10 in Appendix A.
5.4 Answer Prediction Evaluation
The answer prediction accuracy results of all tested predictors, macro-averaged over  $D_{test}(C)$  of all categories, are presented in Figure 3. We inspect the performance of the methods on different subsets of the test data, where each subset is determined by all records having at least  $x$  twins,  $x \in [0..130]$ . The horizontal axis indicates the minimal number of twins in the subset and the percentage of the data each subset represents. For example, the results at  $x = 0$  represent the entire test set, while the results at  $x = 10$  represents the subset of questions with at least 10 twins, account for  $40.2\%$  of the test set.
The performance of Majority begins with  $66\%$  (the percent of 'yes' questions in the entire population) and drops for questions with many twins. We
hypothesize that "obvious" questions, for which the answer is the same across many products, are rarely asked hence have fewer twins. In contrast, informative questions, for which the answer is varied across products, are frequently asked w.r.t. many products, hence have many twins. Therefore we see a drop in accuracy of the Majority baseline as the number of twins grows.
The accuracy of QSO is significantly higher than the majority-vote baseline. This demonstrates an interesting phenomena in the data of similar questions that tend to have the same answer over variety of products, typically of the same type. A few examples are presented in Table 5. The QSO method successfully detects these groups of questions and predicts the majority answer for each such group. We find that PSO method generally doesn't improve over QSO. This is somewhat surprising, as we expected that using product similarity information, such as brand, model, or key features, would increase the prediction accuracy. This demonstrates the importance of question-context, as used in SimBA, in addition to the product information alone.
Moving to SimBA, we can see a large performance improvement over the QSO and PSO methods, which we attribute directly to the CPS model. We also see consistent improvement in accuracy with the number of twins, likely due to the larger support the model has for predicting the answer.
The APC method, despite its relative simplicity, performs very well and greatly outperforms the majority-vote and the QSO and PSO baselines. For the segment of questions with less than 10 twins, APC outperforms the SimBA method. This segment represents roughly  $60\%$  of the questions. However, for the segment of questions with 60 or more twins, which accounts for  $13.6\%$  of the questions, SimBA method consistently outperforms the inductive baseline by  $1 - 2\%$ . When inspecting the results by category, as shown in Table 6, we can see that considering all questions with at least 1 twin, the APC method dominates in 7 out of the 11 categories, while for questions with at least 60 twins, SimBA method dominates in 6 out of the 11 categories.
Finally, we see that the two approaches compliment each other and can be effectively joined, as the SimBA+APC method outperforms both of them over all subsets.
16We also tried a few different splits on the development set
Questions with 1+ Twins	Questions with 60+ Twins
% of data	SimBA	APC	SimBA+APC	% of data	SimBA	APC	SimBA+APC
Light bars	62.5	75.3	74.8	77.7	9.0	74.1	68.1	75.0
Monitors	79.2	76.0	76.4	76.5	27.0	78.5	78.5	78.5
Smartwatches	88.3	77.3	79.3	79.0	31.9	79.9	81.7	80.8
Receivers	57.5	70.1	70.3	72.0	4.8	83.2	77.9	83.2
Backpacks	74.7	80.7	82.7	82.3	21.5	82.7	83.2	82.9
Jeans	63.3	67.4	65.9	67.4	13.4	74.8	70.9	75.7
Beds	70.4	77.7	76.4	77.9	16.8	82.5	80.4	82.5
Home & office desks	65.0	71.8	76.2	75.8	4.7	80.0	84.4	84.4
Masks	76.5	70.9	74.2	73.0	4.2	80.4	82.6	80.4
Posters & prints	60.1	73.4	73.4	74.4	9.3	75.9	69.6	75.9
Accessories	71.7	78.1	79.0	79.2	7.2	82.3	81.6	82.7
Macro Average	69.9	74.4	75.3	75.9	13.6	79.5	78.1	80.2
Table 6: Answer prediction accuracy by category. Left: accuracy over records with at least one twin, representing  $69.9\%$  of the records on average. Right: accuracy over records with at least 60 twins, representing  $13.6\%$  of the records. The highest accuracy between SimBA and APC is in bold.
We presented SimBA, a novel answer prediction approach in the PQA domain, which directly leverages similar questions answered with respect to other products. Our empirical evaluation shows that on some segments of questions, namely those with roughly ten or more similar questions in the corpus, our method can outperform a strong inductive method that directly utilizes the question and the textual product content. We further show that the two approaches are complementary and can be integrated to increase the overall answer prediction accuracy.
For future work, we plan to explore how SimBA can be extended and be applied beyond yes-no questions, e.g., for questions with numerical answers or open-ended questions. Another interesting research direction is combining additional voters to the Mixture-of-Experts model, such as a review-aware answer predictor or a product details-based predictor. Additionally, our current evaluation considered a static view of the answered product-question corpus, we plan to explore temporal aspects of our method, for example, considering questions age or ignoring answers of obsolete products that might be irrelevant.
Ohad Rozen would like to express his gratitude to Yanai Elazar, Vered Shwartz, and Ido Dagan for providing him valuable advice while he was conducting this research during his internship at Amazon.
David Carmel, Liane Lewin-Eytan, and Yoelle Maarek. 2018. Product question answering using customer
generated content - research challenges. SIGIR '18, page 1349-1350. Association for Computing Machinery.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder. CoRR, abs/1803.11175.
Long Chen, Ziyu Guan, Wei Zhao, Wanqing Zhao, Xiaopeng Wang, Zhou Zhao, and Huan Sun. 2019. Answer identification from product reviews for user questions by multi-task attentive networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 45-52.
Lei Cui, Shaohan Huang, Furu Wei, Chuanqi Tan, Chaoqun Duan, and Ming Zhou. 2017. Superagent: A customer service chatbot for e-commerce websites. In Proceedings of ACL 2017, System Demonstrations, pages 97-102.
Manirupa Das, Zhen Wang, Evan Jaffe, Madhuja Chattopadhyay, Eric Fosler-Lussier, and Rajiv Ramnath. 2019. Learning to answer subjective, specific product-related queries using customer reviews by adversarial domain adaptation. CoRR, abs/1910.08270.
Yang Deng, Wenxuan Zhang, and Wai Lam. 2020. Opinion-aware answer generation for review-driven question answering in e-commerce. In Proceedings of CIKM 2020, page 255-264. Association for Computing Machinery.
Miao Fan, Chao Feng, Mingming Sun, Ping Li, and Haifeng Wang. 2019. Reading customer reviews to answer product-related questions. In Proceedings of the 2019 SIAM International Conference on Data Mining, pages 567-575. SIAM.
Shen Gao, Zhaochun Ren, Yihong Eric Zhao, Dongyan Zhao, Dawei Yin, and Rui Yan. 2019. Product-aware answer generation in e-commerce question-answering. In Proceedings WSDM 2019, pages 429-437. ACM.
Jing He and Decheng Dai. 2011. Summarization of yes/no questions using a feature function model. volume 20 of Proceedings of Machine Learning Research, pages 351-366, South Garden Hotels and Resorts, Taoyuan, Taiwan. PMLR.
Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. 1991. Adaptive mixtures of local experts. *Neural Comput.*, 3(1):79-87.
Jiwoon Jeon, W Bruce Croft, and Joon Ho Lee. 2005. Finding similar questions in large question and answer archives. In Proceedings CIKM 2005, pages 84-90.
T. M. Lai, T. Bui, N. Lipka, and S. Li. 2018. Supervised transfer learning for product information question answering. In Proceedings of ICMLA 2018, pages 1109-1114.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Julian McAuley. 2016. Amazon question/answer data. https://jmcauley.ucsd.edu/data/amazon/qa/"
Julian J. McAuley and Alex Yang. 2016. Addressing complex and subjective product-related queries with customer reviews. In Proceedings WWW 2016, pages 625-635. ACM.
Dae Hoon Park, Hyun Duk Kim, ChengXiang Zhai, and Lifan Guo. 2015. Retrieval of relevant opinion sentences for new products. In Proceedings SIGIR 2015, pages 393-402. ACM.
Fatemeh Pourgholamali. 2016. Mining information for the cold-item problem. In Proceedings of RecSys 2016, page 451-454. ACM.
Mengting Wan and Julian McAuley. 2016. Modeling ambiguity, subjectivity, and diverging viewpoints in opinion question answering systems. In Proceeding of ICDM 2016, pages 489-498. IEEE.
Jianfei Yu, Minghui Qiu, Jing Jiang, Jun Huang, Shuangyong Song, Wei Chu, and Haiqing Chen. 2018. Modelling domain relationships for transfer learning on retrieval-based question answering systems in e-commerce. In Proceedings of WSDM 2018, page 682-690. ACM.
Jianxing Yu, Zheng-Jun Zha, and Tat-Seng Chua. 2012. Answering opinion questions on products by exploiting hierarchical organization of consumer reviews. In Proceedings of EMNLP 2012, pages 391-401. ACL.
Qian Yu and Wai Lam. 2018. Review-aware answer prediction for product-related questions incorporating aspects. In Proceedings of WSDM 2018, pages 691-699. ACM.
A.1 Amazon-PQSim dataset
The Amazon-PQSim dataset includes question pairs, where all questions are published on Amazon website. Each pairs has a corresponding label: 1 for similar, else 0. The labels were collected via Appen crowd sourcing service. We took the following filtering steps (step b in 4.1) for each question:
- Removed any question with less than five words.
- Removed any question with more than 15 words.
- Removed any none-English questions.
- Removed any question with multiple question-marks (may indicate multiple questions).
- Removed questions with rare words (any word which is not in the top 2000 most frequent words).
A.2 Amazon-PQA dataset
The Amazon-PQA dataset includes questions and their answers that are published on Amazon website, along with the public product information and category (Amazon Browse Node name). The data includes the following fields:
'question_id',
‘asin_id',
- 'question_text',
- 'answer_text',
bullet_points
- 'product_description',
- 'brand_name',
- 'item_name',
  - 'item_name',
In addition, we augment this data with fields derived from our current work:
‘is_yes-no_question',
- 'yes-no_answer' (yes, no, maybe),
Yes/No Question Classification We followed (He and Dai, 2011) for detecting Yes/No questions using simple heuristics, such as checking if the question starts with a Be verb (am, is, are, been, being, was, were), Modal verb (can, could, shall, should, will, would, may, might) or an Auxiliary verb (do, did, does, have, had, has), and additionally ends with a question mark. We tested the classifier on McAuley's dataset (McAuley, 2016), identified yes/no questions with  $98.4\%$  precision at  $96.5\%$  recall, while considering McAuley as ground truth<sup>17</sup>.
Yes/No Answer Labeling As described in Section A.2, we used the McAuley dataset (McAuley, 2016) to train a RoBERTa-based classifier, taking only yes/no questions. For testing, we used 5 annotator to annotate 583 question-answer pairs, randomly sampled from our raw data. The model achieved  $97\%$  and  $88\%$  precision for the yes and no labels, respectively, and a recall of  $65\%$  and  $51\%$  on the entire test set<sup>18</sup>.
Next, to determine each question's final yes/no answer, we first omitted answers classified as maybe. When a question is answered by a verified seller, we considered it as most reliable and used it as the final label. Otherwise we used the majority votes among the remaining answers. In our experiments, we ignore questions with an equal number of yes and no answers.
Dataset Split Each item in our dataset is a (product, question, answer) triplet. We split the labeled triplets into train  $(80\%)$ , dev  $(10\%)$ , and test  $(10\%)$  sets for each category, relating to the number of products. Each product with all related questions appear only in one of these sets. The statistics for this dataset are given in Table 8.
A.3 CPS Model Details
The CPS has a total of 254.6M parameters. For all incorporated RoBERTa models we use a maximum sequence length of 256, dropout of 0.1, and a 32 batch size for training. We applied different learning rates and number of epochs for each product-category. The specific values we used after tuning are shown in Table 9.
17McAuley and Yang reported identifying yes/no questions with  $97\%$  precision at  $82\%$  recall on their dataset.
18 McAuley and Yang reported  $98\%$  accuracy after keeping only the  $50\%$  of instances about which their classifier was the most confident.
Category	# Products	# Questions	# Y/N Questions	# Answers	# Q. Answered Yes	# Q. Answered No
Light Bars	6,151	48,736	23,956	95,853	10,146	5,243
Monitors	6,246	170,529	86,837	316,126	35,353	22,947
Smartwatches	8,611	166,400	94,936	289,945	41,683	22,033
Receivers	3,442	58,618	33,511	135,700	14,488	7,364
Backpacks	12,096	68,598	38,914	138,996	19,902	6,090
Jeans	38,008	61,908	17,518	129,346	7,708	5,548
Beds	17,202	108,723	46,722	238,786	17,757	13,917
Home Office Desks	6,986	55,303	23,202	112,958	9,523	5,971
Masks	13,384	51,295	24,989	100,789	9,757	5,759
Posters & Prints	33,803	53,939	20,737	99,926	8,171	5,450
Accessories	38,825	238,603	159,272	438,447	60,990	23,772
Rest of 89 Categories	1,288,754	8,906,362	4,833,639	18,565,933	2,219,022	1,055,816
Total	1,473,508	9,989,014	5,404,233	20,662,805	2,454,500	1,179,910
Table 7: Statistics of the Amazon-PQA dataset extracted from Amazon.com. # Y/N Questions as detected by our Yes/No Question detector; # Answers is the total number of answers before any filtering; # Q. Answers Yes (No) is the number of Yes/No questions with answers labeled as Yes (No) (See Section 4.2)
Category	Train Set	Dev Set	Test Set	Total
# P	# Q	% Yes	# P	# Q	% Yes	# P	# Q	% Yes	# P	# Q	% Yes
Light bars	2,552	8,675	68.1	319	1,080	68.7	319	1,296	69.2	3,190	11,051	68.3
Monitors	3,421	29,886	63.3	427	3,890	64.7	427	4,260	63.0	4,275	38,036	63.4
Smartwatches	4,128	34,734	68.5	516	3,730	66.4	516	3,778	67.8	5,160	42,242	68.3
Receivers	1,725	11,991	69.2	215	1,827	68.1	215	2,356	65.7	2,155	16,174	68.5
Backpacks	4,834	14,740	78.4	604	1,397	75.9	604	1,908	77.3	6,042	18,045	78.0
Jeans	5,365	6,056	61.3	670	773	59.8	670	769	58.1	6,705	7,598	60.8
Beds	5,912	16,792	59.1	739	2,017	58.3	739	2,276	58.3	7,390	21,085	58.9
Home Office Desks	2,576	8,637	62.7	322	1,059	64.3	322	962	62.9	3,220	10,658	62.9
Masks	4,332	8,541	64.8	541	1,180	64.0	541	1,099	63.1	5,414	10,820	64.5
Posters & Printed	5,739	7,226	62.7	717	,868	62.4	717	850	66.0	7,173	8,944	63.0
Accessories	14,422	54,125	73.5	1,802	6,038	73.7	1,802	6,706	74.5	18,026	66,869	73.6
Total	55,006	201,403	66.5*	6,872	23,859	66.0*	6,872	26,260	66.0*	68,750	251,522	66.4*