SlowGuess/ACL_2020_OCR huggingface dataset:
Review-based Question Generation with Adaptive Instance Transfer and Augmentation*
Qian Yu $^{1,2\dagger}$  Lidong Bing $^{2}$  Qiong Zhang $^{2}$  Wai Lam $^{1}$  Luo Si $^{2}$
<sup>1</sup> The Chinese University of Hong Kong
2 DAMO Academy, Alibaba Group
$^{1}\{$  yuqian, wlam\}@se.cuhk.edu.hk
2{1.bing, qz.zhang, luo.si}@alibaba-inc.com
While online reviews of products and services become an important information source, it remains inefficient for potential consumers to exploit verbose reviews for fulfilling their information need. We propose to explore question generation as a new way of review information exploitation, namely generating questions that can be answered by the corresponding review sentences. One major challenge of this generation task is the lack of training data, i.e. explicit mapping relation between the user-posed questions and review sentences. To obtain proper training instances for the generation model, we propose an iterative learning framework with adaptive instance transfer and augmentation. To generate to the point questions about the major aspects in reviews, related features extracted in an unsupervised manner are incorporated without the burden of aspect annotation. Experiments on data from various categories of a popular E-commerce site demonstrate the effectiveness of the framework, as well as the potentials of the proposed review-based question generation task.
The user-written reviews for products or service have become an important information source and there are a few research areas analyzing such data, including aspect extraction (Bing et al., 2016; Chen et al., 2013), product recommendation (Chelliah and Sarkar, 2017), and sentiment analysis (Li et al., 2018; Zhao et al., 2018a). Reviews reflect certain concerns or experiences of users on products or services, and such information is valuable for other
potential consumers. However, there are few mechanisms assisting users for efficient review digestion. It is time-consuming for users to locate critical review parts that they care about, particularly in long reviews.
We propose to utilize question generation (QG) (Du et al., 2017) as a new means to overcome this problem. Specifically, given a review sentence, the generated question is expected to ask about the concerned aspect of this product, from the perspective of the review writer. Such question can be regarded as a reading anchor of the review sentence, and it is easier to view and conceive due to its concise form. As an example, the review for a battery case product in Table 1 is too long to find sentences that can answer a user question such as "How long will the battery last?" Given the generated questions in the right column, it would be much easier to find out the helpful part of the review. Recently, as a topic attracting significant research attention, question generation is regarded as a dual task of reading comprehension in most works, namely generating a question from a sentence with a fixed text segment in the sentence designated as the answer (Duan et al., 2017; Sun et al., 2018).
Two unique characteristics of our review-based question generation task differentiate it from the previous question generation works. First, there is no review-question pairs available for training, thus a simple Seq2Seq-based question generation model for learning the mapping from the input (i.e. review) to the output (i.e. question) cannot be applied. Even though we can easily obtain large volumes of user-posed review sets and question sets, they are just separate datasets and cannot provide any supervision of input-output mapping (i.e. review-question pair). The second one is that different from the traditional question generation, the generated question from a review sentence will not simply take a fixed text segment in the review as its
*The work described in this paper is partially supported by a grant from the Research Grant Council of the Hong Kong Special Administrative Region, China (Project Code: 14204418).
†The work was done when Qian Yu was an intern at Alibaba.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 280-290 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Review	Question
It doesn’t heat up like most of the other ones, and I was completely fascinated by the ultra light and sleek design for the case. Before I was using the Mophie case but I couldn’t wear it often because it was like having a hot brick in your pocket, hence I had to always leave it at home. On the contrary, with PowerBear, I never take it off because I can’t even tell the difference. Also it is build in a super STRONG manner and even though I dropped my phone a few times, its shock resistant technology won’t let a single thing happen to the case or the phone. The PowerBear case became an extension to my phone that I never have to take off because when I charge it at night, it charges both my phone and the case. I have battery life for more than two days for normal use, i.e. not power-consuming gaming.	Does this make the phone warm during charging?
Have any of you that own this had a Mophie?
Does this give protection to the phone?
Can this charge the phone and the extra battery at the same time?
How many days it can last?
Table 1: A product review and the example questions.
answer. The reason is that some reviews describing user experiences are highly context-sensitive. For the example in Table 1, for the review "I have battery life for more than two days for normal use, i.e. not power-consuming gaming." and its corresponding example question "How many days it can last?", obviously the text segment "more than two days" is a less precise answer, while the whole review sentence is much more informative. In some other case, even such less precise answer span cannot be extracted from the review sentence, e.g. for the example question "Does this give protection to the phone?" and the review sentence "Also it is ... even though I dropped my phone ..., its shock resistant technology won't let a single thing happen to the case or the phone". Of course here, a simple "Yes" or "No" answer does not make much sense as well, while the whole review sentence is a vivid and informative answer.
The above two unique characteristics raise two challenges for our task. The first challenge, namely lacking review-question pairs as training instances, appears to be intractable, particularly given that the current end-to-end models are very data-hungry. One instant idea is to utilize user-posed (question, answer) pairs as substitute for training. However, several instance-related defects hinder the learned generation model from being competent for the review-based question generation. Some answers are very short, e.g. "more than two days", therefore, without necessary context, they are not helpful to generate good questions. The second challenge, namely the issue that some verbose answers contain irrelevant content especially for subjective questions. To handle this challenge, we propose a learning framework with adaptive instance transfer and augmentation.
Firstly, a pre-trained generation model based on user-posed answer-question pairs is utilized as an initial question generator. A ranker is designed to work together with the generator to improve the
training instance set by distilling it via removing unsuitable answer-question pairs to avoid "negative transfer" (Pan and Yang, 2009), and augmenting (Kobayashi, 2018) it by adding suitable review-question pairs. For selecting suitable reviews for question generation, the ranker considers two factors: the major aspects in a review and the review's suitability for question generation. The two factors are captured via a reconstruction objective and a reinforcement objective with reward given by the generator. Thus, the ranker and the generator are iteratively enhanced, and the adaptively transferred answer-question pairs and the augmented review-question pairs gradually relieve the data lacking problem.
In accordance with the second characteristic of our task, it is plausible to regard a review sentence or clause as the answer to the corresponding question originated from it. Such treatment brings in the second challenge: how can we guarantee that the generated question concentrates on the critical aspect mentioned by the review sentence? For example, a question like "How was the experience for gaming?" is not a favourable generation for "I have battery life for more than two days for normal use, i.e. not power-consuming gaming." To solve this problem, we incorporate aspect-based feature discovering in the ranker, and then we integrate the aspect features and an aspect pointer network in the generator. The incorporation of such aspect-related features and structures helps the generator to focus more on critical product aspects, other than the less important parts, which is complied with the real user-posed questions.
To sum up, our main contributions are threefold. (1) A new practical task, namely question generation from reviews without annotated instance, is proposed and it has good potential for multiple applications. (2) A novel adaptive instance transfer and augmentation framework is proposed for handling the data lacking challenge in the task. (3)
Review-based question generation is conducted on E-commerce data of various product categories.
Question generation (QG) is an emerging research topic due to its wide application scenarios such as education (Wang et al., 2018), goal-oriented dialogue (Lee et al., 2018), and question answering (Duan et al., 2017). The preliminary neural QG models (Du et al., 2017; Zhou et al., 2017; Du and Cardie, 2017) outperform the rule-based methods relying on hand-craft features, and thereafter various models have been proposed to further improve the performance via incorporating question type (Dong et al., 2018), answer position (Sun et al., 2018), long passage modeling (Zhao et al., 2018b), question difficulty (Gao et al., 2019), and to the point context (Li et al., 2019). Some works try to find the possible answer text spans for facilitating the learning (Wang et al., 2019). Question generation models can be combined with its dual task, i.e., reading comprehension or question answering with various motivations, such as improving auxiliary task performance (Duan et al., 2017; Yang et al., 2017; Golub et al., 2017), collaborating QA and QG model (Tang et al., 2018, 2017), and unified learning (Xiao et al., 2018).
Although question generation has been applied on other datasets, e.g., Wikipedia (Du and Cardie, 2018), most of the existing QG works treat it as a dual task of reading comprehension (Yu et al., 2018; Cui et al., 2017), namely generating a question from a piece of text where a certain text span is marked as answer, in spite of several exceptions where only sentences without answer spans are used for generating questions (Du et al., 2017; Chali and Baghaee, 2018). Such generation setting is not suitable for reviews due to the lack of (question, review) pairs and improper assumption of text span answer as aforementioned. There are works training the question generation model with the user-written QA pairs in E-commerce sites (Hu et al., 2018; Chali and Baghaee, 2018), but the practicality is limited since the questions are only generated from answers instead of reviews.
Transfer learning (Pan and Yang, 2009; Tan et al., 2017; Li et al., 2020) refers to a broad scope of methods that exploit knowledge across domains for handling tasks in the target domain. A few terms are used for describing specific methods in this learning paradigm, e.g., self-taught learning (Raina
et al., 2007), domain adaptation (Long et al., 2017), etc. Based on "what to transfer", transfer learning is categorized into four groups (Pan and Yang, 2009), namely instance transfer, feature representation transfer, parameter transfer, and relational knowledge transfer. Our learning framework can be regarded as a case of instance transfer with iterative instance adaptation and augmentation.
3 The Proposed AITA Framework
For handling the aforementioned issues, we propose an Adaptive Instance Transfer and Augmentation (AITA) framework as shown in Figure 1. Since the review-related processing is always sentence-based, we use "review" for short to refer to review sentence in this paper. Its two components, namely ranker and generator, are learned iteratively. Initially, AITA simply transfers all available (question, answer) pairs and trains a generator. Then it will iteratively enhance the generator with the help of the ranker. The ranker takes a (question, answer) pair and a review as its input and calculates a ranking score  $s$ . Thus, it can rank all reviews for a given QA pair. The ranking objective incorporates the reward provided by the generator, which helps find out those suitable reviews to form (review, question) pairs for training (i.e. augmenting the training data). Meanwhile, the reward from the generator also helps remove unsuitable QA pairs for training, so that it makes the transfer more adaptive. Note that the ranker also learns to model two hidden aspect related variables for the review, which are helpful for the generator to ask about the major aspects in review. Such an iterative instance manipulation procedure gradually transfers and augments the training set for handling review-based question generation.
3.1 Review Ranker for Data Augmentation
There are two pieces of input text for ranker. The first one is the concatenation of a (question, answer) pair  $qa$  and the second one is a review sentence  $r$ .  $qa$  and  $r$  are associated with the same product. Since the ranker is responsible for instance augmentation that provides (question, review) pairs, it is trained to learn a score  $s(qa, r)$  which can be used to return suitable  $r$ 's for a given  $qa$ .
Ranking with Partially Shared Encoders. The input  $qa$  and  $r$  are encoded with two Transformer encoders with the same structure and partially shared parameters, to leverage the advantage of
Figure 1: AITA framework.  $\mathbf{M}$  is the shared parameter matrix for QA and review.
multi-head self attention on modeling word associations without considering term position. An input  $(qa$  or  $r)$  is written as a matrix  $\mathrm{E} = [e_1^T,\dots,e_n^T ]^T$  where  $e$  is a word embedding and  $n$  is the text length. The number of heads in the multi-head selfattention is denoted as  $m$ , and the output of the  $j$ -th head is written as:
$$
\mathrm {Q} ^ {j}, \mathrm {K} ^ {j}, \mathrm {V} ^ {j} = \mathrm {E W} _ {Q} ^ {j}, \mathrm {E W} _ {K} ^ {j}, \mathrm {E W} _ {V} ^ {j} \tag {1}
$$
$$
\operatorname {h e a d} ^ {j} (\mathrm {E}) = \operatorname {s o f t m a x} \left(\frac {\mathrm {Q} ^ {j} \mathrm {K} ^ {j T}}{\sqrt {d}}\right) \mathrm {V} ^ {j} \tag {2}
$$
where  $d$  is the dimension of word embedding. The outputs of different heads are concatenated and the encoding for the  $i$ -th word is written as  $\mathbf{h}_i = [head_i^1; \ldots; head_i^m]$ .
To obtain the sentence representation considering the complete semantics, we apply a global attention layer on the output of the Transformer encoder:
$$
\mathbf {h} ^ {\alpha} = \sum_ {i = 1} ^ {n} \alpha_ {i} \mathbf {h} _ {i} \tag {3}
$$
where the attention weight  $\alpha_{i} = \exp (\mathbf{h}_{i}\cdot \mathbf{M}\cdot \overline{\mathbf{h}}) / Z_{\alpha},$ $Z_{\alpha}$  is the normalization, and  $\bar{\mathbf{h}} = \sum \mathbf{h}_i / n$  . The parameter matrix M is shared by encoders for both  $q a$  and  $r$  for capturing the common attention features across them.
After encoding  $qa$  and  $r$  as  $\mathbf{h}^{\alpha}(qa)$  and  $\mathbf{h}^{\alpha}(qa)$ , a vector  $\mathbf{g}(qa,r)$  is assigned with the concatenation of  $\mathbf{h}^{\alpha}(qa)$ ,  $\mathbf{h}^{\alpha}(qa)$  and their difference
$$
\mathbf {g} (q a, r) = \left[ \begin{array}{l} \mathbf {h} ^ {\alpha} (q a), \mathbf {h} ^ {\alpha} (r), | \mathbf {h} ^ {\alpha} (q a) - \mathbf {h} ^ {\alpha} (r) | \end{array} \right]
$$
The review ranking score  $s(qa, r)$  is calculated as:
$$
s (q a, r) = \sigma \left(\mathbf {W} ^ {s} \mathbf {g} (q a, r) + \mathbf {b} ^ {s}\right) \tag {4}
$$
where  $\sigma$  is sigmoid function.
Reinforcement Objective for Ranker Learning. To learn an appropriate  $s(qa, r)$ , we encounter a major challenge, namely lacking ground truth labels for (question, review). Our solution takes the generator in our framework as an agent that can provide reward for guiding the learning of ranker. The generator is initially trained with (question, answer) data, and is gradually updated with adapted and augmented training instances, so that the rewards from the generator can reflect the ability of review for generating the corresponding question.
Specifically, we propose a reinforcement objective that makes use of the reward from the generator, denoted as  $\mathrm{reward}_G(r,q)$ . For each pair of question and review, we take the normalized  $\log \mathrm{ppl}(q|r)$  in the generator as reward:
$$
\operatorname {r e w a r d} _ {G} (r, q) = \frac {\log \operatorname {p p l} (q | r)}{\sum_ {r ^ {*} \in R _ {q a}} \log \operatorname {p p l} (q | r ^ {*})} \tag {5}
$$
where  $\mathbf{R}_{qa}$  is the reviews under the same product as  $qa$ , and  $\log \mathrm{ppl}(q|r)$  is the log perplexity of generating a question  $q$  from a review  $r$ :
$$
\log \operatorname {p p l} (q | r) = - \frac {1}{| q |} \sum_ {t \in [ 1, | q | ]} p _ {G} \left(q _ {t} | r, q _ {1} \dots q _ {t - 1}\right)
$$
The reinforcement objective for the ranker is to maximize the average reward for all the reviews given a question. The sampling probabilities for reviews are obtained via normalized ranking score, namely  $p(r|qa) = s(qa,r) / \mathbf{Z}_{qa}$ , where  $\mathbf{Z}_{qa} = \sum_{r*\in \mathbb{R}_{qa}}s(qa,r*)$ . The loss function is:
$$
L ^ {g} (q a, r) = \mathrm {E} _ {r \sim p (r | q a)} \operatorname {r e w a r d} _ {G} (r, q) \tag {6}
$$
The gradient calculation for the above objective is an intractable problem. As an approximated method which performs well in the iterative algorithm, the normalization term  $\mathbf{Z}_{qa}$  is fixed during
the calculation of the policy gradient:
$$
\Delta L ^ {g} (q a, r) = \sum_ {r} \Delta s (q a, r) \operatorname {r e w a r d} _ {G} (r, q) / \mathrm {Z} _ {q a}
$$
Regularization with Unsupervised Aspect Extraction. Product aspects usually play a major role in all of product questions, answers and reviews, since they are the discussion focus of such text content. Thus, such aspects can act as connections in modeling input pairs of  $qa$  and  $r$  via the partially shared structure. To help the semantic vector  $\mathbf{h}^{\alpha}$  in Eqn 3 capture salient aspects of reviews, an autoencoder module is connected to the encoding layer for reconstructing  $\mathbf{h}^{\alpha}$ . Together with the matrix M, the autoencoder can be used to extract salient aspects from reviews. Note that this combined structure is similar to the ABAE model (He et al., 2017), which has been shown effective for unsupervised aspect extraction. Compared with supervised aspect detection methods, such a unsupervised module avoid the burden of aspect annotation for different product categories, and our experiments demonstrate that regularization based on this module is effective.
Specifically,  $\mathbf{h}^{\alpha}$  is mapped to an aspect distribution  $\mathbf{p}^{\alpha}$  and then reconstructed:
$$
\begin{array}{l} \mathbf {p} ^ {\alpha} = \operatorname {s o f t m a x} \left(\mathbf {W} ^ {p} \cdot \mathbf {h} ^ {\alpha} + \mathbf {b} ^ {p}\right) (7) \\ \mathbf {h} ^ {\alpha \prime} = \mathbf {p} ^ {\alpha} \cdot \mathbf {A} (8) \\ \end{array}
$$
where each dimension in  $p^\alpha$  stands for the probability that the review contains the corresponding aspect, and  $h^{\alpha \prime}$  is the reconstruction of review representation, and A is a learnable parameter matrix. Note that we define "aspects" as implicit aspect categories, namely clusters of associated attributes of product, which is commonly used in unsupervised aspect extraction (Wang et al., 2015; He et al., 2017). The reconstruction objective is written as:
$$
L ^ {\alpha} (q a, r) = \left[ \mathbf {h} ^ {\alpha} (r) - \mathbf {h} ^ {\alpha^ {\prime}} (r) \right] ^ {2} / 2. \tag {9}
$$
Only the reconstruction of review representations is considered since we focus on discovering aspects in reviews. In this way, the aspect-based reconstruction will force  $\mathbf{h}^{\alpha}$  to focus on salient aspects that facilitate the reconstruction. The final loss function of the ranker is regularized to:
$$
L (q a, r) = L ^ {g} (q a, r) - \lambda L ^ {\alpha} (q a, r) \tag {10}
$$
3.2 Question Generator in Transfer Learning
We adapt the Seq2Seq model for the aspect-focused generation model, which is updated gradually via the transferred and augmented instances. With the help of aspect-based variables learned in ranker, the generator can generate questions reflecting the major aspect in the review.
Aspect-enhanced Encoding. To emphasize the words related to salient aspects, the attention weight  $\alpha_{i}$  obtained in the ranker is incorporated into the word embedding. Given an input review sentence, we obtain the extended word embedding  $\tilde{\mathbf{e}}_i$  at position  $i$ :
$$
\tilde {\mathbf {e}} _ {i} = \left[ \mathbf {e} _ {i}, \mathbf {e} _ {i} ^ {P O S}, \mathbf {e} _ {i} ^ {N E R}, \alpha_ {i} \right] \tag {11}
$$
where  $\mathbf{e}_i$  is the pre-trained word embedding,  $\mathbf{e}_i^{POS}$  is the one-hot POS tag of  $i$ -th word,  $\mathbf{e}_i^{NER}$  is a BIO feature for indicating whether the  $i$ -th word is a named entity, and  $\alpha_i$  indicates the aspect-based weight for the  $i$ -th word. Bi-LSTM is adopted as the basic encoder of generator, encoding the  $i$ -th word as the concatenation of hidden states with both directions:  $\mathbf{h}_i^g = [\overrightarrow{\mathbf{h}}_i, \overleftarrow{\mathbf{h}}_i]$ .
Decoding with Aspect-aware Pointer Network. Pointer network, i.e., copy mechanism, can significantly improve the performance of text generation. In our task, in addition to the word-level hidden state in the decoder, the overall aspect distribution of the review can also provide clues for how likely the generator should copy corresponding review aspect words into the generated question.
The question is generated with an LSTM decoder. The word probability for the current time step is formulated as:
$$
p _ {0} \left(q _ {t}\right) = \operatorname {s o f t m a x} \left(\mathbf {W} _ {2} \tau + \mathbf {b} _ {2}\right)
$$
and related variables are calculated as:
$$
\begin{array}{l} \tau = \sigma \left(\mathrm {W} _ {1} \left[ \mathbf {s} _ {t}, \mathbf {c} _ {t} \right] + \mathbf {b} _ {1}\right), \quad \mathbf {s} _ {t} = \operatorname {L S T M} \left(y _ {t}, \mathbf {s} _ {t - 1}\right), \\ \mathbf {c} _ {t} = \sum_ {j} \mathbf {z} _ {t j} \mathbf {h} _ {j} ^ {g}, \quad \mathbf {z} _ {t j} = \operatorname {s o f t m a x} \left(\mathbf {h} _ {j} ^ {g} \mathbf {W} _ {h} \mathbf {s} _ {t}\right) \\ \end{array}
$$
where  $\mathbf{s}_t$  is the hidden state for the  $t$ -th word in question and  $\mathbf{c}_t$  is the context encoding based on attention weight  $\mathbf{z}_{tj}$ .
In the pointer network, for a particular position  $t$  in the generated text, the word may be copied from a distribution based on the attention weight  $\mathbf{z}_t = \{z_{tj}\}$ , where the copy probability is assigned according to the current hidden state  $\mathbf{s}_t$ . We also
<sup>1</sup>We simplified the objective in AEAB model by eliminating the additional regularization term which is not necessary when combining  $L^{\alpha}(qa,r)$  and  $L^{g}(qa,r)$ .
Data: QA set  $\mathbf{S}_{qa} = \{(q,a)\}$ ; review set  $\mathbf{S}_r = \{r\}$ ;
Result: S; generator trained with S
Prepare pairs of  $(qa, r)$  under each product Initialize the training set  $\mathbf{S} = \mathbf{S}_{qa}$
For each epoch Do