SlowGuess/ACL_2020_OCR huggingface dataset:
Zhiliang Tian,\*1,4 Wei Bi,\*2 Dongkyu Lee,\*1 Lanqing Xue,\*1 Yiping Song,\*3 Xiaojiang Liu,\*2 Nevin L. Zhang\*1,4
$^{1}$ Department of Computer Science and Engineering,
The Hong Kong University of Science and Technology, Hong Kong SAR, China
$^{3}$ Department of Computer Science School of EECS, Peking University, Beijing, China
$^{4}$ HKUST Xiao-i Robot Joint Lab, Hong Kong SAR, China
{ztianac,dlear,lxueaa,lzhang}@cse.ust.hk
{victoriabi,kieranliu}@tencent.com songyiping@pku.edu.cn
Neural conversation models are known to generate appropriate but non-informative responses in general. A scenario where informativeness can be significantly enhanced is Conversing by Reading (CbR), where conversations take place with respect to a given external document. In previous work, the external document is utilized by (1) creating a context-aware document memory that integrates information from the document and the conversational context, and then (2) generating responses referring to the memory. In this paper, we propose to create the document memory with some anticipated responses in mind. This is achieved using a teacher-student framework. The teacher is given the external document, the context, and the ground-truth response, and learns how to build a response-aware document memory from three sources of information. The student learns to construct a response-anticipated document memory from the first two sources, and the teacher's insight on memory creation. Empirical results show that our model outperforms the previous state-of-the-art for the CbR task.
Neural conversation models have achieved promising performance in response generation. However, it is widely observed that the generated responses lack sufficient content and information (Li et al., 2016a). One way to address this issue is to integrate various external information into conversation models. Examples of external information include document topics (Xing et al., 2017), commonsense knowledge graphs (Zhou et al., 2018), and domain-specific knowledge bases (Yang et al., 2019). Conversing by reading (CbR) (Qin et al.,
Figure 1: A motivating example of constructing a response-anticipated document memory for response generation. Details are provided in the introduction.
2019) is a recently proposed scenario where external information can be ingested to conversations. In CbR, conversations take place with reference to a document. The key problem in CbR is to learn how to integrate information from the external document into response generation on demand.
To exploit knowledge from documents for conversations, a conventional way is to extend the sequence-to-sequence (Seq2Seq) model (Sutskever et al., 2014) with Memory Networks (Sukhbaatar et al., 2015), which store knowledge representations accessible to their decoder (Ghazvininejad et al., 2018; Parthasarathi and Pineau, 2018). Dinan et al. (2018) propose to encode the dialogue context as well as a set of retrieved knowledge by Transformer (Vaswani et al., 2017) to construct the memory. However, these methods only use sentence-level representations of the documents in the memory, which cannot pinpoint accurate token-level document information.
To discover token-level document information, researchers borrow models from other generation tasks, which are adept at extracting segments of sentences for given questions. Moghe et al. (2018) explore the pointer generator network (See et al., 2017) for abstractive summarization and the bidirectional attention flow model (Seo et al., 2017), which is a QA model to predict a span of the
*This work was partially done when Zhiliang Tian was an intern at Tencent AI Lab.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 650-659 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
document to be contained in the response. Qin et al. (2019) follow the stochastic answer network (SAN) (Liu et al., 2018) in machine reading comprehension (MRC), integrating both context and document information to form the context-aware document memory. This approach obtains the state-of-the-art performance on the CbR task.
However, we should notice the difference between existing generation tasks and CbR. For summarization, QA, and MRC, they require models to extract exact answers from documents, where documents cover all requisite knowledge. Meanwhile, CbR expects to output a general utterance relevant to both context and document. As the example in Fig. 1, the document refers to actor, films, fans, wealthy and the context mentions disease. Document and context discuss the same person but have no topic overlap; thus we cannot pinpoint document information from the context. If we use SAN as in Qin et al. (2019), SAN can hardly acquire helpful information from context-document interaction. To ingest useful knowledge for response generation, we argue that processing documents should consider not only the interaction between context and document but also the target response. As in the example, the document should attend more on fans, wealthy by considering the response.
In this work, we propose a method to construct a response-anticipated memory to contain document information that is potentially more important in generating responses. Particularly, we construct a teacher-student framework based on Qin et al. (2019). The teacher model accesses the ground-truth response, context, and document. It learns to construct a weight matrix that contains information about the importance of tokens in the document to the response. The student model learns to mimic the weight matrix constructed by the teacher without access to the response. That is, the teacher learns to build a response-aware memory, while the student learns to build a response-anticipated memory. During inference on testing data, the student will be applied. Our experiments show our model exceeds all competing methods.
Most neural conversation models in open domain chit-chat scenarios are based on the Seq2Seq model (Sutskever et al., 2014; Shang et al., 2015). A critical issue of these models is the safe response problem, i.e., generated responses often
lack enough content and information. To address this issue, previous work encourages response diversity and informativeness by introducing new training objectives (Li et al., 2016b; Zhao et al., 2017), refining beam search strategies (Li et al., 2016a; Vijayakumar et al., 2018; Song et al., 2017), exploiting information from conversational contexts (Serban et al., 2016, 2017; Tian et al., 2017), or incorporating with retrieval-based conversation systems (Song et al., 2018; Wu et al., 2019b; Tian et al., 2019).
Some researchers augment information in generating responses by external resources. Zhou et al. (2018) utilize the commonsense knowledge graph by their designed graph attention. Agarwal et al. (2018) propose a knowledge encoder to encode query-entity pairs from the knowledge base. Wu et al. (2019a) enrich response generation with knowledge triplets. These work all uses knowledge information in structured formats.
External unstructured text information has also been investigated to improve conversation models. Some researchers directly build "document memory" by using distributed representations of the knowledge sentences into conversation models (Ghazvininejad et al., 2018; Parthasarathi and Pineau, 2018). Dinan et al. (2018) make use of the Transformer (Vaswani et al., 2017) to encode the knowledge sentences as well as the dialogue context. Ren et al. (2020) design a knowledge selector to construct the document memory on selective knowledge information. As stated in the introduction, some other researchers borrow models from other generation tasks, including abstractive summarization models (Moghe et al., 2018), QA models (Moghe et al., 2018) and MRC models (Meng et al., 2020; Qin et al., 2019). Especially, Qin et al. (2019) get the state-of-the-art performance. However, they all construct the document memory relying on connections between context and document without consideration of the response. If context or document contains a lot of noise tokens irrelevant to the response, which is indeed the case in CbR, the constructed memory may be misled by these noise information (as the case in Fig. 1). Therefore, we propose to involve the consideration of responses in the memory construction, which can benefit generating a more desired response.
In this section, we will first give an overall description of the proposed teacher-student architecture for CbR, then briefly describe the base model. The detailed teacher model and student model are presented in Sec 3.3 and 3.4. Lastly, we summarize the training updates of the two models in Sec 3.5.
The CbR task provides a conversation context  $X$  and a document  $D$  as inputs, requiring the model to generate a response  $R$  to  $X$  by referring to  $D$ . In the rest of the paper, we use  $|X|, |D|$ , and  $|R|$  to denote the number of tokens in  $X$ ,  $D$ , and  $R$  respectively. To pinpoint accurate document information for response generation, we design a teacher-student framework to construct document memory as follows:
- The teacher model learns a response-aware document memory  $\mathbf{M}$  used in our base conversation model. Specifically, we construct a response-aware weight matrix  $\mathbf{G} \in \mathbb{R}^{|D| \times |D|}$ , which considers the correlation between context-aware document representations and response representations, and then impose  $\mathbf{G}$  on the memory matrix  $\mathbf{M}$ . The teacher model is optimized to reconstruct the response with the use of response-aware memory  $\mathbf{M}$ .
- The student model learns to construct a response-anticipated weight matrix to estimate  $\mathbf{G}$  used in the teacher model but without access to the response. It is a feed-forward neural network with document and context as its input.
The teacher model and the student model are jointly optimized with training data, while only the student model is applied to testing data.
Following Qin et al. (2019), we use SAN (Liu et al., 2018) as our base model, which mainly consists of three components:
- Input encoder: We use two bi-directional LSTM encoders to extract token-level representations of the document  $D$  and the context  $X$ .
- Memory construction: We build the document memory  $\mathbf{M} \in \mathbb{R}^{|D| \times k}$  ( $k$  is the hidden size of the memory) which will be used in the decoder. A cross-attention layer is first applied to the outputs of the two encoders to integrate information from the context to the document. Then, we obtain a set of context-aware document representation
$\mathbf{D} = [\mathbf{d}_1,\dots ,\mathbf{d}_{|D|}]$  . Since each  $\mathbf{d}_i$  corresponds to a document token, we treat it as the contextaware token representation of the  $i$  -th token. Next, a self-attention layer is employed to ingest salient information of the context-aware document representations:
$$
\mathbf {M} = \operatorname {S e l f A t t n} (\mathbf {D}) = \mathbf {A D} ^ {T}, \mathbf {A} = \operatorname {s o f t m a x} (\mathbf {D} ^ {T} \mathbf {D}) \tag {1}
$$
where the softmax conducts the normalization over each row of the matrix.
- Output decoder: We use an attentional recurrent decoder to generate response tokens by attending to the memory  $\mathbf{M}$ . The initial hidden state is set as the summation of token-level context representations. For each decoding step  $t$ , we get a hidden state  $\mathbf{h}_t$ :
$$
\mathbf {z} _ {t} = \operatorname {G R U} \left(\mathbf {e} _ {t - 1}, \mathbf {h} _ {t - 1}\right), \tag {2}
$$
$$
\mathbf {h} _ {t} = \mathbf {W} _ {\mathbf {1}} \left[ \mathbf {z} _ {t}; \text {C r o s s A t t n} \left(\mathbf {z} _ {t}, \mathbf {M}\right) \right] \tag {3}
$$
where  $[;]$  indicates concatenation, and the crossattention layer here integrates information from the memory to the recurrent outputs.  $\mathbf{e}_{t - 1}$  is the word-embedding at step  $t - 1$ . Finally, we generate a token  $y_{t}$  by a softmax on  $\mathbf{h}_t$ .
Our model modifies the memory construction by refining its self-attention layer so that the memory represents more accurate and on-demand knowledge that helps generating the response.
3.3 Teacher Model
To ingest accurate memory information for response generation under the aforementioned base model, our teacher model builds a response-aware weight matrix  $\mathbf{G} \in \mathbb{R}^{|D| \times |D|}$  given the context-aware document representation  $\mathbf{D}$  and the response  $R$ , then refines the document memory  $\mathbf{M}$  with  $\mathbf{G}$ . Elements in  $\mathbf{G}$ 's indicate the importance of tokens or token pairs in the document, with consideration of the response information.
First, we describe how to modify the memory matrix  $\mathbf{M}$  when  $\mathbf{G}$  is given. The original memory  $\mathbf{M}$  is constructed by a self-attention operation as Eq. 1. To facilitate response awareness, we update the attention weight matrix  $\mathbf{A}$  by element-wise multiplying  $\mathbf{G}$ , and then get the refined memory  $\widetilde{\mathbf{M}}$  as
$$
\mathbf {A} = \operatorname {s o f t m a x} \left(\mathbf {D} ^ {T} \mathbf {D}\right), \widetilde {\mathbf {M}} = (\mathbf {G} \odot \mathbf {A}) \mathbf {D} ^ {T}. \tag {4}
$$
In the following, we describe two methods to construct the response-aware weight matrix  $\mathbf{G}$ : (1)
Figure 2: The architecture of our model. Blocks and lines in gray color compose the base model. Blue and gray parts compose the teacher model, while purple parts compose the student model. All components work for training, while only the student model and the decoder works for inference. In the response-aware/anticipated weight matrix, darker grids indicate higher weights.  $(\otimes)$  : matrix multiplication;  $\odot$  : element-wise matrix multiplication.)
We measure the response-aware token importance (RTI) considering the ground-truth response to construct  $\mathbf{G}$ . (2) We measure the response-aware pairwise importance (RPI) of each token pair  $(i,j)$ , which can be directly assigned to the element  $G_{ij}$  in  $\mathbf{G}$ . For both methods, matrix elements can be either continuous or binary.
Response-Aware Token Importance (RTI)
We denote the response-aware token importance of document tokens as  $\beta \in \mathbb{R}^{|D|}$ , and measure it by response  $R$  and context-aware token representation  $\mathbf{D}$ . To obtain  $\beta$ , we first apply an encoder to obtain the token-level representations of the response as  $[\mathbf{r}_1, \dots, \mathbf{r}_{|R|}]$  and use its last hidden state  $\mathbf{r}_{|R|}$  as the sentence-level response representation. The response-aware token importance of token  $i$  is defined as the similarity between its context-aware token representation  $\mathbf{d}_i$  and the response representation  $\mathbf{r}_{|R|}$ . Next, we adjust each attention distribution (i.e., each column of  $\mathbf{A}$ ) with each of its attention weight multiplied by the token importance  $\beta_i$ . Therefore, the resulting  $\mathbf{G}$  can be obtained as:
$$
\beta_ {i} = \mathbf {d} _ {i} ^ {T} \mathbf {r} _ {| R |}, \quad \mathbf {G} = \mathbf {1} \boldsymbol {\beta} ^ {T}, \tag {5}
$$
where  $\mathbf{1} \in \mathbb{R}^{|D|}$  represents an identity vector with all elements as 1. By plugging the above  $\mathbf{G}$  in Eq. 5, we can construct a memory matrix with plagiarized signals from the response. In this way, the self-attention distributions can adjust to emphasize important tokens, and their corresponding context-aware document token representations be
come more important in the memory matrix.
Recall that the document contains a large amount of noise information in CbR. Thus the attention distributions may become long-tailed due to the existence of many redundant document tokens. Hence, we can further construct a binary weighting vector based on  $\beta$ . We keep the weight of each element as 1 with the probability of  $\beta_{i}$  calculated in Eq. 5. If the weight of a token turns to 0, this token is deactivated in calculating the attention distributions. However, the binary weight sampled from the Bernoulli distribution is not differentiable. To enable back-propagation of our model, we apply the Gumbel-Softmax (Jang et al., 2016) to approximate the Bernoulli distribution in the training phase, and sample the binary value from the Bernoulli distribution in the prediction phase as:
$$
\mathbf {G} = \mathbf {1} g (\boldsymbol {\beta}) ^ {T}, \tag {6}
$$
where  $g(\beta)$  is defined as:
$$
\left\{ \begin{array}{l l} g \left(\beta_ {i}\right) = \text {G u m b e l S o f t m a x} \left(\beta_ {i}\right) & \text {T r a i n i n g}, \\ g \left(\beta_ {i}\right) \sim \text {B e r n o u l l i} \left(\beta_ {i}\right) & \text {P r e d i c t i o n}. \end{array} \right. \tag {7}
$$
The objective function of the teacher model is to maximize the log-likelihood of responses generated by the response-aware memory constructed with  $\beta$ :
$$
\beta = f _ {\theta_ {t}} ^ {t} (D, X, R), \mathcal {J} _ {t} = \underset {D, X, R \sim \mathcal {D}} {\mathbb {E}} \log P _ {\phi} (R | D, X, \beta), \tag {8}
$$
where  $f^t$  denotes operations in Eq. 5 and its pre-order operations.  $\theta_t$  consists of all parameters in the layers of  $f^t$ .  $\phi$  denotes parameters in Eq. 1 to Eq. 3. Both  $\phi$  and  $\theta_t$  are learning parameters for  $\mathcal{I}_t$ .
Response-Aware Pairwise Importance (RPI)
Instead of using token importance, we can construct  $\mathbf{G}$  by the pairwise importance of token pairs. After obtaining the token representations  $[\mathbf{r}_1,\dots ,\mathbf{r}_{|R|}]$  from the response encoder similarly as in RTI, we can calculate the similarity of each  $\mathbf{d}_i$  towards all  $\mathbf{r}_j$ 's, denoted as  $\mathbf{n}_i\in \mathbb{R}^{|R|}$ . Each element in  $\mathbf{G}$  can be associated with a weight  $\mathbf{B}_{ij}$  defined as the inner-product between  $\mathbf{n}_i$  and  $\mathbf{n}_j$ . Thus, we can treat  $\mathbf{B}$  as the response-aware pairwise importance, and directly set each element in  $\mathbf{G}$  as  $\mathbf{B}_{ij}$ :
$$
\mathbf {n} _ {i} = \left[ \mathbf {r} _ {1}, \dots , \mathbf {r} _ {| R |} \right] ^ {T} \mathbf {d} _ {i}, \mathbf {B} _ {i j} = \mathbf {n} _ {i} ^ {T} \mathbf {n} _ {j}, \mathbf {G} = \mathbf {B}. \tag {9}
$$
Compared with response-aware token importance in which the designed  $\mathbf{G}$  has identical column values, response-aware pairwise importance allows different values of different index  $(i,j)$ 's in  $\mathbf{G}$  (but  $(i,j)$  and  $(j,i)$  have the same value since  $\mathbf{G}$  is symmetric). Thus, the space of  $\mathbf{G}$  is larger.
Notice that, the aforementioned binary processing with each  $\beta_{i}$  can also be applied on each  $\mathbf{B}_{ij}$  here and the resulting  $\mathbf{G}$  is binary. By using a binary  $\mathbf{G}$  in our model, the memory construction can be considered as passing through a Graph Attention Network (GAT) (Velicković et al., 2018), which also constructs a graph and updates its representations relying on the information from itself and neighbors on the graph. However, our neighborhood matrix (i.e.  $\mathbf{G}$  in our model) is not predefined as in GAT but dependant on the inputs  $\mathbf{d}_i$ 's and  $\mathbf{r}_j$ 's, which involve parameters to be estimated.
The objective of the teacher model for RPI can be modified from Eq. 8 by replacing  $\beta$  with  $\mathbf{B}$  obtained in Eq. 9.
3.4 Student Model
The student model learns to construct a response-anticipated weight matrix to estimate the weight matrix  $\mathbf{G}$  in the teacher model without access to the ground-truth  $R$ . If we employ RTI, the estimated target of the student model is  $\beta$  in Eq. 5. For RPI, the estimated target is  $\mathbf{B}$  in Eq. 9.
Given  $\mathbf{D}$  and  $\mathbf{X}$  as inputs, we apply a bilinear attention layer to obtain a hidden representation matrix  $\mathbf{H}$ . We apply a two-layer multi-layer per
ceptron (MLP) with ReLU activation to estimate  $\beta$ ; we combine two attention outputs by  $\mathbf{W}_{\mathbf{a}}$  to estimate  $\mathbf{B}$  in the RPI:
$$
\mathbf {H} = \operatorname {s o f t m a x} \left(\mathbf {D} ^ {T} \mathbf {W} \mathbf {X}\right) \mathbf {X} ^ {T}, \tag {10}
$$
$$
\left\{ \begin{array}{l l} \hat {\boldsymbol {\beta}} = \operatorname {M L P} (\mathbf {H}) & \text {f o r R T I}, \\ \hat {\mathbf {B}} = \mathbf {H} \mathbf {W} _ {\mathbf {a}} \mathbf {H} ^ {T} & \text {f o r R P I}. \end{array} \right. \tag {11}
$$
The objective function of the student model is to maximize the log-likelihood of generating responses based on the estimated  $\hat{\beta}$  or  $\hat{\mathbf{B}}$ , and diminish the gap of the weighting vector or matrix between the student model and the teacher model by a mean square loss. Taking the RTI strategy as an example, we optimize the following objective:
$$
\hat {\beta} = f _ {\theta_ {s}} ^ {s} (D, X), \tag {12}
$$
$$
\mathcal{J}_{s}\underset {D,X,R\sim \mathcal{D}}{= \mathbb{E}\log P_{\phi}(R|D,X,\hat{\boldsymbol{\beta}}) - \lambda   \mathcal{L}_{\mathrm{MSE}}(\boldsymbol {\beta},\hat{\boldsymbol{\beta}}),}
$$
where  $f^s$  denotes the operation in Eq. 11 and its preorder operations.  $\theta_s$  consists of the layer parameters in  $f^s$ .  $\lambda$  balances the two loss terms. For RPI, we replace to optimize with  $\mathbf{B}$  and  $\hat{\mathbf{B}}$ .
We first train the teacher model until it converges, and then train the student model with the use of  $\beta$  or  $\mathbf{B}$  from the converged teacher model. Next, we repeat the above processes iteratively. In the training of the teacher model, we fix parameters in  $\theta_{s}$  (except parameters shared with  $\theta_{t}$ ) and train the model subject to  $\mathcal{I}_t$ ; for the student model, we fix  $\phi$  and  $\theta_{t}$  (except parameters shared with  $\theta_{s}$ ) and train the model subject to  $\mathcal{I}_s$ . For inference, only the student model will be used to infer the response-anticipated weight matrix and the decoder applies it for generating the output response.
As stated in RPI, it has better model capacity by allowing a larger space of  $\mathbf{G}$  with the use of the weight matrix  $\mathbf{B}$  instead of the token importance vector  $\beta$  in RTI. In terms of optimization, we need to estimate more parameters by using RPI, which requires higher training difficulty.
4 Experiment Setting
We use the dataset for the CbR task released by Qin et al. (2019). The dataset contains crawled articles and discussions about these articles from Reddit. The articles act as the documents, while the discussions serve as conversational contexts and responses. In total, we have  $2.3\mathrm{M} / 13\mathrm{k} / 1.5\mathrm{k}$  samples for training/testing/validation.
Appropriateness	Grounding	Informativeness	Len
NIST	BLEU	Meteor	P	R	F1	PGT	RGT	F1GT	Ent4	Dist1	Dist2
Human	2.650	3.13%	8.31%	2.89%	0.45%	0.78%	0.44%	0.09%	0.14%	10.445	0.167	0.670	18.8
Seq2Seq	2.223	1.09%	7.34%	1.20%	0.05%	0.10%	0.89%	0.05%	0.09%	9.745	0.023	0.174	15.9
MemNet	2.185	1.10%	7.31%	1.25%	0.06%	0.12%	0.91%	0.05%	0.10%	9.821	0.035	0.226	15.5
GLKS	2.413	1.34%	7.61%	2.47%	0.13%	0.24%	0.84%	0.05%	0.10%	9.715	0.034	0.213	15.3
CMR	2.238	1.38%	7.46%	3.39%	0.20%	0.38%	0.91%	0.05%	0.10%	9.887	0.052	0.283	15.2
CMR+Copy	2.155	1.41%	7.39%	5.37%	0.28%	0.54%	0.92%	0.06%	0.11%	9.798	0.044	0.266	14.4
RAM_T	2.510	1.43%	7.74%	4.46%	0.26%	0.49%	1.04%	0.08%	0.15%	9.900	0.053	0.290	15.1
RAM_P	2.353	1.40%	7.59%	3.89%	0.21%	0.41%	0.97%	0.07%	0.13%	9.891	0.049	0.279	14.9
RAM_T+Copy	2.467	1.41%	7.64%	6.14%	0.32%	0.61%	0.65%	0.04%	0.08%	9.813	0.045	0.265	14.9
RAM_P+Copy	2.342	1.41%	7.51%	5.83%	0.30%	0.57%	0.84%	0.06%	0.10%	9.798	0.045	0.267	14.6
Table 1: Automatic evaluation results on all competing methods. Len denotes the length of the generated responses.