SlowGuess/ACL_2020_OCR huggingface dataset:
1. Train generator with S.
2. Prepare the reward  $G(qa, r)$  as generator reward for each pair of  $(qa, r)$  (each answer  $a$  in  $qa$  pairs is regarded as a review for  $q$ ).
3. Adapt S via removing  $\mu$  instances with low reward.
4. Train ranker according to the objective in Eqn 10.
5. Augment S via adding  $\mu$  pairs of instances, which are  $(q,r)$  pairs with top  $s(qa,r)$  in ranker.
6. Collect  $\alpha$  and  $p^\alpha$  for instances in S from ranker.
Algorithm 1: Learning algorithm of AITA.
consider the influence of the aspect distribution  $\mathbf{p}^{\alpha}$  in the copy probability  $\beta$  for interpolation:
$$
\beta = \sigma \left(\mathbf {p} ^ {\alpha} \mathbf {W} _ {c} \mathbf {s} _ {t} + \mathbf {b} _ {c}\right) \tag {12}
$$
The incorporation of  $\mathbf{p}^{\alpha}$  helps the pointer network to consider the overall aspect distribution of context in addition to the semantics in the current position for copying words. Finally, the  $t$ -th word is generated from the mixture of the two distributions:
$$
p \left(q _ {t}\right) = (1 - \beta) \cdot p _ {0} \left(q _ {t}\right) + \beta \cdot \mathbf {z} _ {t}. \tag {13}
$$
The generator is trained via maximizing the likelihood of the question  $q$  given the review  $r$ :
$$
p (r | q) = \sum_ {i} p \left(r _ {i} \mid q, r _ {1}, \dots , r _ {i - 1}\right) \tag {14}
$$
3.3 Iterative Learning Algorithm
The purpose of our iterative learning, as by Alg 1, is to update the generator gradually via the instance augmentation. The input data for the iterative learning consists of the transferred instance set of question-answer pairs  $\mathbf{S}_{qa}$ , an unlabeled review set  $\mathbf{S}_r$ , and an adaption parameter  $\mu$ . When the learning is finished, two outputs are produced: the final training instances  $\mathbf{S}$ , and the learned generator. The training set  $\mathbf{S}$  for generator is initialized
with  $\mathbf{S}_{qa}$ . In each iteration of the algorithm, the generator is trained with current  $\mathbf{S}$ , and then  $\mathbf{S}$  is adapted accordingly. The ranker is trained based on the rewards from the generation, which is used for instance augmentation in  $\mathbf{S}$ . Thus, the training set  $\mathbf{S}$  is updated during the iterative learning, starting from a pure (question, answer) set. Analysis on the influence of the composition of  $\mathbf{S}$ , i.e., instance numbers of two types, is presented in Section 4.5.
There are two kinds of updates for the instance set  $\mathbf{S}$ : (1) adaption via removing  $(q, a)$  pairs with low generator reward, in order to avoid "negative transfer"; (2) augmentation via adding  $(q, r)$  pairs that are top ranked by ranker, in order to increase the proportion of suitable review-question instances in training set. The instance number hyperparameter  $\mu$  for removing and adding can be set according to the scale of  $\mathbf{S}_{qa}$ , and more details are given in our experimental setting.
To guarantee the effective instance manipulation, two interactions exist between generator and ranker. First, aspect-related variables for reviews obtained by ranker are part of the generator input. The second interaction is that a reward from generator is part of the learning objective for ranker, in order to teach ranker to capture the suitable reviews for generating the corresponding question.
We exploit the user-written QA dataset collected in (Wan and McAuley, 2016) and the review set collected in (McAuley et al., 2015) as our experimental data. The two datasets are collected from Amazon.com separately. We filter and merge the two datasets to obtain products whose associated QA pairs and reviews can both be found. The statistics for our datasets can be found in Table 2, where the numbers of product for several very large product categories are restricted to 5000. According to the average lengths, we can find that the whole review tend to be very long. It justified our assumption that it is not easy for users to exploit reviews, and questions with short length can be a good catalogue for viewing reviews.
To test our question generation framework, we manually labeled 100 ground truth review-question pairs for each product category. 6 volunteers are asked to select user-posed questions and the corresponding review sentences that can serve as answers. Specifically, the volunteers are given pairs
#p	#q	#a	#r	#(s)
Auto	0.8k	5.5k	18.7k	9.4k	46.5k
Baby	1.9k	11.9k	38.7k	75.3k	450.7k
Beauty	2.5k	15.9k	53.7k	62.4k	338.6k
Phones	3.6k	23.8k	87.4k	104.5k	561.8k
Cloth	0.4k	0.30k	10.7k	6.9k	32.2k
Elec	5k	31.0k	101.2k	229.4k	1461.8k
Health	5k	32.4k	114.2k	136.9k	749.9k
Music	0.4k	2.7k	8.9k	5.2k	27.9k
Sports	5k	34.2k	120.6k	122.6k	648.5k
Tools	4.1k	29.8k	104.1k	70.7k	425.6k
Lq	La	Lr	Ls
Auto	14.4	23.3	88.3	17.8
Baby	15.2	22.9	106.4	17.8
Beauty	13.1	22.0	88.6	16.3
Phones	13.2	19.2	97.0	18.1
Cloth	13.0	19.8	71.2	15.3
Elec	16.1	24.8	119.5	18.8
Health	13.0	22.5	96.0	17.5
Music	14.6	24.0	94.2	17.7
Sports	13.6	22.3	91.0	17.2
Tools	14.7	23.2	110.2	18.3
Table 2: Data statistics. #: number;  $p,q,a,r$  : product, question, answer, whole review;  $s$  : review sentence,  ${L}_{q}$  ,  ${L}_{a},{L}_{r},{L}_{s}$  are their average lengths.
of question and review, and only consider the relevance between question and review. The answer to the question is also accessible but it is only used for helping annotators to understand the question. All labeled pairs are validated by two experienced annotators with good understanding for the consumer information need in E-commerce.
The labeled instances are removed from the training set.
For each product category, we train the AITA framework and use the learned generator for testing. The fixed 300 dimension GloVe word embeddings (Pennington et al., 2014) are used as the basic word vectors. For all text including question, answer and review, we utilize StanfordNLP for tokenizing, lower casing, and linguistic features extraction, e.g., NER & POS for the encoder in generator. In ranker, the dimension of aspect distribution is set to 20 and the  $\lambda$  in the final loss function in Eqn 10 is set to 0.8. In the multi-head self-attention, the head number is set to 3 and the dimension for Q, K, V is 300. The dimensions of matrices can be set accordingly. The hidden dimension in generator is set to 200. In the iterative learning algorithm, we set the epoch number to 10 and the updating instance number  $\mu$  to  $0.05 \times |\mathbf{S}_{qa}|$ . In testing, given a review  $r$  as input for generator, the additional
input variables  $\alpha (r)$  and  $\mathbf{p}^{\alpha}(r)$  are obtained via the review encoder (Eqn 3) and aspect extraction (Eqn 8), which are question-independent.
For testing the effectiveness of our learning framework and the incorporation of aspect, we compare our method with the following models:  $\mathbf{G}_a$  (Du et al., 2017): A sentence-based Seq2Seq generation model trained with user-written answer-question pairs.  $\mathbf{G}_a^{PN}$  (Wang et al., 2018): A pointer network is incorporated in the Seq2Seq decoding to decide whether to copy word from the context or select from vocabulary.  $\mathbf{G}_{ar}^{PN}$ : Review data is incorporated via a retrieval-based method. Specifically, the most relevant review sentence for each question is retrieved via BM25 method, and such review-question pairs are added into the training set.  $\mathbf{G}_a^{PN}+$  aspect (Hu et al., 2018): Aspect is exploited in this model. We trained the aspect module in our framework, i.e. only using the reconstruction objective to obtain an aspect feature extractor from reviews. Then the aspect features and distributions can be used in the same way as in our method. AITA refers to our proposed framework. AITA-aspect: All the extracted aspect-related features are removed from AITA as an ablation for evaluating the effectiveness of the unsupervised module for aspect. For every product category, we run each model for 3 times and report the average performance with four evaluation metrics, including BLEU1 (B1), BLEU4 (B4), METEOR (MET) and ROUGE-L ( $R_L$ ).
4.3 Evaluation of Question Generation
The results are demonstrated in Table 3. AITA achieves the best performance on all product categories regarding different evaluation metrics. The significant improvements over other models demonstrate that our instance transfer and augmentation method can indeed reduce inappropriate answer-question pairs and provide helpful review-question pairs for the generator. The performance of  $\mathbf{G}_a$  is very poor due to the missing of attention mechanism. Both  $\mathbf{G}_a^{PN}$  and  $\mathbf{G}_a^{PN}+$  aspect have worse performance than ours, even though some product categories have large volume of QA pairs ( $>100\mathrm{k}$ ), e.g., Electronics, Tools, etc. This indicates that the answer-question instances are not capable of learning a review-based question generator because of the different characteristics between the answer set and review set.  $\mathbf{G}_{ar}^{PN}$  performs much worse than  $\mathbf{G}_a^{PN}$ , which proves that a simple retrieval method
BLEU1	BLEU4	METEOR	ROUGE-L	BLEU1	BLEU4	METEOR	ROUGE-L
Automotive	Baby
Ga	0.103	0.047	0.062	0.089	0.104	0.055	0.065	0.068
GPN	0.162	0.090	0.091	0.140	0.153	0.088	0.087	0.195
GPNar	0.147	0.082	0.078	0.118	0.133	0.060	0.068	0.102
GPN+aspect	0.165	0.090	0.093	0.140	0.157	0.088	0.091	0.203
AITA-aspect	0.179	0.094	0.094	0.146	0.157	0.089	0.092	214
AITA	0.184	0.097	0.099	0.148	0.167	0.089	0.094	0.221
Beauty	Cell Phone
Ga	0.133	0.088	0.118	0.218	0.203	0.125	0.130	0.104
GPN	0.235	0.122	0.128	0.257	0.250	0.122	0.150	0.217
GPNar	0.194	0.098	0.119	0.205	0.215	0.117	0.136	0.141
GPN+aspect	0.240	0.122	0.132	0.257	0.251	0.134	0.154	0.223
AITA-aspect	0.240	0.127	0.132	0.257	0.261	0.139	0.184	0.230
AITA	0.249	0.129	0.136	0.259	0.267	0.142	0.193	0.244
Clothing & Jewelry	Electronics
Ga	0.224	0.093	0.091	0.178	0.099	0.048	0.107	0.144
GPN	0.283	0.134	0.118	0.227	0.124	0.069	0.131	0.171
GPNar	0.258	0.110	0.101	0.198	0.100	0.053	0.121	0.156
GPN+aspect	0.298	0.139	0.125	0.241	0.120	0.069	0.126	0.171
AITA-aspect	0.306	0.152	0.138	0.246	0.125	0.069	0.131	0.174
AITA	0.316	0.157	0.145	0.263	0.127	0.073	0.131	0.175
Health	Musical Instruments
Ga	0.114	0.062	0.091	0.095	0.088	0.054	0.096	0.091
GPN	0.130	0.080	0.089	0.108	0.114	0.110	0.121	0.119
GPNar	0.124	0.069	0.086	0.104	0.090	0.072	0.106	0.103
GPN+aspect	0.133	0.100	0.123	0.175	0.118	0.110	0.130	0.192
AITA-aspect	0.137	0.100	0.121	0.179	0.124	0.110	0.136	0.201
AITA	0.142	0.109	0.132	0.194	0.129	0.112	0.141	0.205
Sports & Outdoors	Tools
Ga	0.079	0.046	0.042	0.064	0.098	0.059	0.093	0.105
GPN	0.091	0.052	0.079	0.102	0.107	0.077	0.112	0.135
GPNar	0.087	0.050	0.071	0.083	0.100	0.072	0.103	0.119
GPN+aspect	0.091	0.052	0.079	0.102	0.110	0.079	0.110	0.136
AITA-aspect	0.094	0.052	0.080	0.102	0.112	0.079	0.116	0.142
AITA	0.097	0.057	0.083	0.102	0.117	0.083	0.120	0.149
is not effective for merging the instances related to reviews and answers. AITA adapts and augments the QA set to select suitable review-question pairs considering both aspect and generation suitability, resulting in a better generator. In addition, effectiveness of aspect feature and aspect pointer network can be illustrated via the slight but stable improvement of  $\mathbf{G}_a^{PN}+$  aspect over  $\mathbf{G}_a^{PN}$  and the performance drop of AITA-aspect on all the categories. This proves that even without precise aspect annotation, our unsupervised aspect-based regularization is helpful for improving generation.
4.4 Human Evaluation and Case Study
We conduct human evaluation on two product categories to study the quality of the generated questions. Two binary metrics Relevance and Aspect are used to indicate whether a question can be answered by the review and whether they share the same or related product aspect. The third metric,
Table 3: Overall performance on question generation.
Clothing & Jewelry
Relevance	Aspect	Fluency
GaPN	0.58	0.62	2.58
GarPN	0.47	0.58	2.29
GaPN+aspect	0.66	0.72	2.76
AITA	0.80	0.80	2.86
Cell Phone
Relevance	Aspect	Fluency
GaN	0.42	0.55	2.79
GnAR	0.35	0.41	2.44
GaN+aspect	0.58	0.63	2.83
AITA	0.72	0.72	2.90
Table 4: Performance of human evaluation.
Fluency with the value set  $\{1,2,3\}$ , is adopted for judging the question fluency. 1 means not fluent and 3 means very fluent. We selected 50 generated questions from each model and asked 4 volunteers
The entire length of the watch is 9 inches, but the effective length from the last hole to clasp is about 8 inches.
-  $\mathbf{G}_a^{PN}$ : What is the difference between gear 2 neo and this watch?
-  $\mathbf{G}_a^{PN}$  +aspect: How is the length?
- AITA: What is the dimension in mm?
If you have a huge wrist this watch mayn't look good nor fit you well.
-  $\mathbf{G}_a^{PN}$ : What is the wrist size?
-  $\mathbf{G}_a^{\bar{P}N}$  +aspect: How does it fit?
- AITA: Will it fit my huge hand?
The stainless steel case back can be pried off from the 12 o'clock position (from the back), and the battery CAN be replaced.
-  $\mathbf{G}_a^{PN}$ : Is the material good quality and not easy to tore?
-  $\mathbf{G}_a^{PN}$  +aspect: Can the lid be removed?
- AITA: Can you tell me how to replace the battery?
The watch has a Japanese Miyota movement inside, and has a Japanese Sony 626sw battery which requires you to loosen a very small flat head screw and slide a little metal arm out of the way to remove the battery.
-  $\mathbf{G}_a^{PN}$ : What is the battery life on this watch?
-  $\mathbf{G}_a^{PN}$  +aspect: Can I remove the battery?
-AITA: Can I remove the battery?
Table 5: Case study of generated questions.
for evaluation. The average scores are reported in Table 4, which shows that our framework achieves the best performance regarding all the metrics, especially for Relevance, showing that our AITA can help generate more accurate questions based on reviews and thus facilitates exploiting reviews. Due to the incorporation of implicit aspect information, both AITA and  $\mathbf{G}_a^{PN}+$  aspect significantly outperform  $\mathbf{G}_a^{PN}$  regarding both Aspect and Relevance. Again,  $\mathbf{G}_{ar}^{PN}$  with a simple retrieval method for augmenting training instances cannot perform well.
The blue sentences in Table 5 are from a long review talking about some important information of a wat ch, and the questions generated by different models are also given. These questions are more user-friendly and potential consumers can browse them to quickly locate the information they care about. For example, if a user wants to know more about the battery replacement, the portion before the third sentence can be skipped. According to the generated questions via three methods in the Table 5, we can find that the questions from AITA are asking about major aspects of the review sentences.  $\mathbf{G}_a^{PN}$  failed to capture major aspects in the first three sentences, and the questions generated by  $\mathbf{G}_a^{PN}+$  aspect are not as concrete as ours, owing to the insufficient training instances.
Figure 2: Analysis for proposition of instances.
4.5 Analysis on Instances Composition
The training instance set for the generator, i.e., S in Algorithm 1, is initialized with QA set and gradually adapted and augmented. Here, we investigate the effect of composition property of S on the generator performance at different epochs. As shown in Fig 2, two product categories and two metrics are illustrated, with the gradually changed training instance set S. The proportion of review-question  $(qr)$  instances in S starts with 0, and significant performance improvement can be observed while the  $qr$  proportion gradually increases. The results stay stable until the  $qr$  proportion reach  $80\%$ .
We propose a practical task of question generation from reviews, whose major challenge is the lack of training instances. An adaptive instance transfer and augmentation framework is designed for handling the task via an iterative learning algorithm. Unsupervised aspect extraction is integrated for aspect-aware question generation. Experiments on real-world E-commerce data demonstrate the effectiveness of the training instance manipulation in our framework and the potentials of the review-based question generation task.
Lidong Bing, Tak-Lam Wong, and Wai Lam. 2016. Unsupervised extraction of popular product attributes
from e-commerce web sites by considering customer reviews. ACM Transactions on Internet Technology, 16:1-17.
Yllias Chali and Tina Baghaee. 2018. Automatic opinion question generation. In INLG, pages 152-158.
Muthusamy Chelliah and Sudeshna Sarkar. 2017. Product recommendations enhanced with reviews. In ACM Conference on Recommender Systems, RecSys '17, pages 398-399.
Zhiyuan Chen, Arjun Mukherjee, Bing Liu, Meichun Hsu, Malu Castellanos, and Riddhiman Ghosh. 2013. Exploiting domain knowledge in aspect extraction. In EMNLP, pages 1655-1667.
Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang, Ting Liu, and Guoping Hu. 2017. Attention-over-attention neural networks for reading comprehension. In ACL, pages 593-602.
Xiaozheng Dong, Yu Hong, Xin Chen, Weikang Li, Min Zhang, and Qiaoming Zhu. 2018. Neural question generation with semantics of question type. In CCF NLPCC, pages 213-223.
Xinya Du and Claire Cardie. 2017. Identifying where to focus in reading comprehension for neural question generation. In EMNLP, pages 2067-2073.
Xinya Du and Claire Cardie. 2018. Harvesting paragraph-level question-answer pairs from wikipedia. In ACL, pages 1907-1917.
Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to ask: Neural question generation for reading comprehension. In ACL, pages 1342-1352.
Nan Duan, Duyu Tang, Peng Chen, and Ming Zhou. 2017. Question generation for question answering. In EMNLP, pages 866-874.
Yifan Gao, Lidong Bing, Wang Chen, Michael R. Lyu, and Irwin King. 2019. Difficulty controllable generation of reading comprehension questions. In *IJCAI*, pages 4968-4974.
David Golub, Po-Sen Huang, Xiaodong He, and Li Deng. 2017. Two-stage synthesis networks for transfer learning in machine comprehension. In EMNLP, pages 835-844.
Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier. 2017. An unsupervised neural attention model for aspect extraction. In ACL, pages 388-397.
Wenpeng Hu, Bing Liu, Jinwen Ma, Dongyan Zhao, and Rui Yan. 2018. Aspect-based question generation. In ICLR Workshop track.
Sosuke Kobayashi. 2018. Contextual augmentation: Data augmentation by words with paradigmatic relations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 452-457.
Sang-Woo Lee, Yu-Jung Heo, and Byoung-Tak Zhang. 2018. Answerer in questioner's mind: Information theoretic approach to goal-oriented visual dialog. In NeurIPS, pages 2579-2589.
Jingjing Li, Yifan Gao, Lidong Bing, Irwin King, and Michael R. Lyu. 2019. Improving question generation with to the point context. In EMNLP, pages 3214-3224.
Juntao Li, Ruidan He, Hai Ye, Hwee Tou Ng, Lidong Bing, and Rui Yan. 2020. Unsupervised domain adaptation of a pretrained cross-lingual language model. In *IJCAI*.
Xin Li, Lidong Bing, Wai Lam, and Bei Shi. 2018. Transformation networks for target-oriented sentiment classification. In ACL, pages 946-956.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. 2017. Deep transfer learning with joint adaptation networks. In ICML, pages 2208-2217.
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. 2015. Image-based recommendations on styles and substitutes. In SIGIR, pages 43-52.
Sinno Jialin Pan and Qiang Yang. 2009. A survey on transfer learning. IEEE Transactions on TKDE, 22(10):1345-1359.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543.
Rajat Raina, Alexis Battle, Honglak Lee, Benjamin Packer, and Andrew Y Ng. 2007. Self-taught learning: transfer learning from unlabeled data. In ICML, pages 759-766. ACM.
Xingwu Sun, Jing Liu, Yajuan Lyu, Wei He, Yanjun Ma, and Shi Wang. 2018. Answer-focused and position-aware neural question generation. In EMNLP, pages 3930-3939.
Ben Tan, Yu Zhang, Sinno Jialin Pan, and Qiang Yang. 2017. Distant domain transfer learning. In AAAI, pages 2604-2610.
Duyu Tang, Nan Duan, Tao Qin, Zhao Yan, and Ming Zhou. 2017. Question answering and question generation as dual tasks. arXiv preprint arXiv:1706.02027.
Duyu Tang, Nan Duan, Zhao Yan, Zhirui Zhang, Yibo Sun, Shujie Liu, Yuanhua Lv, and Ming Zhou. 2018. Learning to collaborate for question answering and asking. In NAACL-HLT, pages 1564-1574.
Mengting Wan and Julian McAuley. 2016. Modeling ambiguity, subjectivity, and diverging viewpoints in opinion question answering systems. In ICDM, pages 489-498.
Linlin Wang, Kang Liu, Zhu Cao, Jun Zhao, and Gerard De Melo. 2015. Sentiment-aspect extraction based on restricted boltzmann machines. In ACL, pages 616-625.
Siyuan Wang, Zhongyu Wei, Zihao Fan, Yang Liu, and Xuanjing Huang. 2019. A multi-agent communication framework for question-worthy phrase extraction and question generation. In AAAI, pages 7168-7175.
Zichao Wang, Andrew S Lan, Weili Nie, Andrew E Waters, Phillip J Grimaldi, and Richard G Baraniuk. 2018. QG-Net: a data-driven question generation model for educational content. In Annual ACM Conference on Learning at Scale, page 7.
Han Xiao, Feng Wang, Yanjian Feng, and Jingyao Zheng. 2018. Dual ask-answer network for machine reading comprehension. arXiv preprint arXiv:1809.01997.
Zhilin Yang, Junjie Hu, Ruslan Salakhutdinov, and William Cohen. 2017. Semi-supervised qa with gen
erative domain-adaptive nets. In ACL, pages 1040-1050.
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. 2018. QANet: Combining local convolution with global self-attention for reading comprehension. In ICLR.
Wei Zhao, Ziyu Guan, Long Chen, Xiaofei He, Deng Cai, Beidou Wang, and Quan Wang. 2018a. Weakly-supervised deep embedding for product review sentiment analysis. IEEE Transactions on TKDE, 30(1):185-197.
Yao Zhao, Xiaochuan Ni, Yuanyuan Ding, and Qifa Ke. 2018b. Paragraph-level neural question generation with maxout pointer and gated self-attention networks. In EMNLP, pages 3901-3910.
Qingyu Zhou, Nan Yang, Furu Wei, Chuanqi Tan, Hangbo Bao, and Ming Zhou. 2017. Neural question generation from text: A preliminary study. In CCF NLPCC, pages 662-671.