SlowGuess/NAACL_2022_OCR huggingface dataset:
Quanbin Wang $^{1,2*}$ , Xiexiong Lin $^{3}$ , Feng Wang $^{3}$ $^{1}$ Alibaba Group
$^{2}$ College of Computer Science and Technology, Zhejiang University, China  $^{3}$ Ant Group
{quanbin.wqb,xiexiong.lxx,zifan.wf}@antgroup.com
The title generation task that summarizes article content in recapitulatory words relies heavily on utilizing the corresponding key context. To generate a title with appropriate information in the content and avoid repetition, we propose a title generation framework with two complementary components in this paper. First, we propose a Timestep aware Sentence Embedding (TSE) mechanism, which updates the sentences' representations by re-locating the critical words in the corresponding sentence for each decoding step. Then, we present an Acme Coverage (AC) mechanism to solve the repetition problem and preserve the remaining valuable keywords after each decoding step according to the final vocabulary distribution. We conduct comprehensive experiments on various title generation tasks with different backbones, the evaluation scores of ROUGE and METEOR in varying degrees are significantly outperforming most of the existing state-of-the-art approaches. The experimental results demonstrate the effectiveness and generality of our novel generation framework TSE-AC.
On account of the existence of many articles and news, automated title generation which summarizes the source content into a succinct title with recapitulatory words can significantly reduce the cost of obtaining information and improve the efficiency of information transmission.
As the example shown in Table 1, the subject is composed of different critical words from the email's first two sentences. When generating the word "birthday", the model needs to focus on the phrase "a year old" to form the first sentence's representation. But for the second target word "party", the model has to update the first sentence's embedding by re-locating to the word "party". Therefore,
email content	We are planning a party it has been a while since the group has had a party and my daughter is going to be a year old. So we are planning a party for November 14th and Vandhana and I would like to invite everyone in research and their family. As yet we do not have the ...
email subject	birthday party invitation
Table 1: An example of email and its subject.
it is necessary for a title generation method to refresh the sentences' embeddings with corresponding key words in different decoding timestep.
End-to-end neural generation models have achieved impressive performance in title generation via sequence-to-sequence (seq2seq) framework (Bahdanau et al., 2014; Sutskever et al., 2014). However, most of them suffered from the difficulty of integrating key information from different parts of the source content effectively. Some other works have realized the importance of the key components from different pieces of the article content (Gehrmann et al., 2018; Li et al., 2018, 2020a; Tan et al., 2017; Cohan et al., 2018), nevertheless, they extract all potentially useful parts as another static input of the model instead of dynamically locating the critical words in combination with the decoding states to update the sentence level representations.
To overcome the shortness of previous works that freeze the sentences' embeddings without considering the changes of the decoding states, we propose a novel title generation architecture with a TSE mechanism, which finely adopts different related words of each sentence to update the sentence's embedding vector in each decoding step. Furthermore, the architecture with TSE incorporates the commonly used hierarchical encoder (Yang et al., 2016; Tan et al., 2017; Cohan et al.,
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1906 - 1918 July 10-15, 2022 ©2022 Association for Computational Linguistics
2018), part of speech (POS) information (Liu et al., 2019), and graph structure (Zhang et al., 2018; Yu et al., 2020) to take full advantages of inner relations among words within each sentence as well as among sentences through the article content, so as to re-locate the related key words for each target word more accurately. This encoding method gives insight into the correlation of each keyword and then integrates them well to summarize the article. To the best of our knowledge, this is the first work which finely updates the embedding of each sentence via the target words for each decoding step.
Besides, repetition is a common problem in generation tasks, which especially stands out in the title generation scenario due to its succinctness. The coverage mechanism is widely used to address this problem and achieves impressive results (Tu et al., 2016; See et al., 2017). Previous coverage mechanisms focus on maintaining a coverage vector which is the sum of attention distributions over all previous decoding timesteps and make the next attention weights as different from the coverage vector as possible. However, in title generation, the penalization with this kind of coverage vector will penalize all attended keywords, disturbing the attention mechanism in the following decoding steps. And the generated titles may lose some keywords because of the inappropriate penalization. Moreover, since our TSE relies heavily on the attention mechanism, the superimposed negative influence will seriously impair the accuracy of the generated title. In this paper, we leverage an AC mechanism that only prevents repeatedly attending to the word which has been generated actually to avoid producing repetitive text but without missing other valuable keywords.
- We propose an end-to-end title generation framework with timestep aware sentence embedding, which is effective for the model to dynamically encode each sentence with critical words and the latest valuable information in the corresponding decoding timestep.
- We present an acme coverage mechanism that solves the repetition problem but avoids unreasonable penalization, which obtains significant outperformance on our novel architecture and other seq2seq models.
- Our model achieves significant improvements over several strong baselines on email subject gen
eration and news headline generation tasks. The detailed experimental results demonstrate that our method is effective and general for different kinds of title generation scenarios.
Title generation has been investigated for a long time, some classical works (Kennedy and Hauptmann, 2000; Jin and Hauptmann, 2001, 2002; Jin et al., 2020) presented various approaches for different kinds of title generation tasks. Rush et al. (2015) first adopted the attention mechanism to the abstractive title generation task, which is a commonly used method for diverse text generation scenarios (Paulus et al., 2018a; Song et al., 2019; Bi et al., 2020a; Lewis et al., 2020). Zhang et al. (2020) pre-trained a model with objectives tailored for abstractive text summarization, they achieved excellent performances on various tasks. However, it remains a major challenge for seq2seq models to tackle document inputs since more information in long documents will probably confuse the model and result in degraded performance. Some previous works (Zhang and Tetreault, 2019; Tan et al., 2017) chose the approach with two separate stages to avoid long inputs, they generated the title via some selected sentences. These kinds of methods may lose necessary information in the generated titles once some useful sentences are ignored.
Some other works applied hierarchical frameworks to encode document level inputs. Tan et al. (2017); Cohan et al. (2018) utilized a hierarchical encoder to model the discourse structure of long documents, but the sentences' embeddings are static during all decoding steps in their works, we update the embedding of each sentence for every targeted word dynamically based on the corresponding valuable words and the decoding state. Li et al. (2020b) presented a hierarchical model to generate summarization for multiple documents. The main differences between their work and ours are twofold. First is the granularity of the encoder, we pay more emphasize on tokens instead of paragraphs since every word is crucial for title generation. The second but critical difference is the related information locating mechanism used in decoder. They first extracted a central location for the key information and select several paragraphs around it. Our model proposes a TSE mechanism which will focus on all related keywords in each sentence to update their embeddings dynamically
Figure 1: The model architecture of our title generation method
for every decoding step.
Besides, repetition is a common problem in generation tasks, there are several efforts made to address this issue. Temporal attention is a technique that has been applied to neural machine translation (NMT) (Sankaran et al., 2016) and summarization (Nallapati et al., 2016), where each attention distribution is divided by the sum of the previous, which effectively avoids repeated attention but distorts the signal from the attention mechanism and then reduces the performance. N-gram blocking (Paulus et al., 2018b) is another technique proposed to discard the n-gram if it appeared already. Coverage mechanism is a widely used technique to dampen repetition, which was first applied to NMT by maintaining a coverage vector (Tu et al., 2016; Mi et al., 2016). A simple approach that upholds the coverage vector with the sum of the attention distributions was introduced in summarization task and achieved impressive results (See et al., 2017). While this approach penalizes the previous attention distribution indiscriminately, which disturbed the generation of later target words and resulted in degraded performance. We leverage an AC mechanism that maintains a coverage vector based on the attention weights with the highest probability of the final vocabulary, and outperforms difference evaluation metrics compared to the vanilla coverage.
The problem we focused on in this paper is giving an article  $A$  as the content of email, news, story etc., which consists of  $|A|$  sentences:  $A = [S_{1}, S_{2}, \dots, S_{|A|}]$ , we aim to generate a succinct but informative title of the source content, denoted as a token sequence  $Y = [y_{1}, y_{2}, \dots, y_{n}]$ . The architecture of our approach (as shown in Figure1) is comprised of two essential parts: 1) Timestep aware Sentence Embedding; 2) Acme Coverage.
3.2 Timestep aware Sentence Embedding
The TSE mechanism (Figure 1) is motivated by the fact that each word in the title reflects different parts of the source content (as shown in Table 1). We use the hidden state of the decoder to update the representations of all sentences by re-locating the valuable words for each decoding step. Besides, the roles of words, relations within the same sentence and across different sentences are meaningful for the TSE to identify really useful words. We encode the source article with the POS information, dependency parsing information within each sentence and the lexical relations among all sentences explicitly. The details of the TSE are described in the following.
The left part of Figure 1 is a hierarchical encoder which encodes the source content into context vectors by Gate Recurrent Unit (GRU) (Cho et al., 2014) or Transformer (Vaswani et al., 2017) (we take GRU as example in the following) and Graph Convolutional Network (GCN) (Kipf and Welling, 2017; Fu et al., 2019). More precisely, for the word encoder, we use GRU to capture the sequence dependency among words in the same sentence. As words with various POS play different roles in the sentence, we add the POS embedding to the corresponding word's embedding as Equation 1.
Moreover, with the aim of obtaining the inner relationships between different words in a sentence, we apply GCN to encode each word in every sentence separately with the dependency parsing results  $(Adj_{dep})$ , if a relation exists, the edge value between the two words is set to 1 in the adjacent matrix, 0 vice versa, the calculation process is shown in Equations 2-4.
$$
I _ {i, t} = \operatorname {E m b e d} \left(w _ {i, t}\right) + \operatorname {E m b e d} \left(P O S _ {i, t}\right) \tag {1}
$$
$$
o _ {i _ {G R U}}, h _ {i _ {G R U}} = G R U \left(I _ {i}\right) \tag {2}
$$
$$
o _ {i _ {G C N}} = G C N \left(o _ {i _ {G R U}}, A d j _ {d e p}\right). \tag {3}
$$
$$
o _ {i}, h _ {i} = G R U \left(o _ {i _ {G C N}}\right) \tag {4}
$$
We use the similar modules for the sentence encoder compared to the word encoder while the weighted sum  $h_t^*$  of the re-located useful words' embedding vectors are used as the first GRU layer's inputs (described in 3.2.2). And we form the adjacent matrix for sentence GCN layer by the cosine similarity between any two sentences' TF-IDF vectors ( $Adj_{tf-idf}$ ), the values in the matrix are converted to  $1/0$  by a certain threshold 0.3.
3.2.2 Relocating Critical Words
Different from previous work which used the encoder outputs of the last word as the corresponding sentence's embedding, we apply TSE to locate the corresponding critical words and then update the embedding of each sentence. Concretely, a attention layer is utilized to obtain the encoding input of sentence  $S_{i}$  at decoding step  $t$ . First, we use the hidden states  $h_i$  of the word-level encoder within sentence  $S_{i}$  and decoder hidden state  $s_t$  to calculate the similarity score  $e_w^t$ . Secondly, the softmax function is used to calculate the attention weight  $a_w^t$ . The calculation process is shown in Equations
$$
e _ {i} ^ {t} = v _ {w} ^ {t} \tanh  \left(W _ {h, w} h _ {i} + W _ {s, w} s _ {t} + b _ {a, w}\right) \tag {5}
$$
$$
a ^ {t} = \operatorname {s o f t m a x} \left(e ^ {t}\right) \tag {6}
$$
$$
h _ {t} ^ {*} = \sum_ {i} a _ {i} ^ {t} o _ {i} \tag {7}
$$
where  $v_{w}^{t}, W_{h,w}, W_{s,w}, b_{attn,w}$  are learnable parameters.
The weighted sum  $h_{t,S_i}^*$  of all sentences is transformed by GRU and GCN layers via Equations 2-4, the output vectors  $o_{S_i}$  are used as the representation of sentence  $S_i$  at decoding timestep  $t$ .
Similar to Equations 5-7, we adopt another attention layer to calculate the context vector  $h_{t,s}^{*}$  based on dynamically updated sentence embeddings  $O_S$ . The context vector is applied to calculate the generation probability  $p_g$  and vocabulary distribution  $P_v$  by Equation 8 and Equation 9.
$$
p _ {g} ^ {t} = \sigma \left(W _ {h ^ {*}} ^ {\prime} h _ {t, s} ^ {*} + W _ {s} ^ {\prime} s _ {t} + W _ {y} ^ {\prime} y _ {t - 1} + b ^ {\prime}\right) \tag {8}
$$
$$
p _ {v} ^ {t} = \operatorname {s o f t m a x} \left(W _ {\text {o u t}} \left[ s _ {t}, h _ {t, s} ^ {*} \right] + b\right) \tag {9}
$$
Where  $W_{h^*}^{\prime}, W_s^{\prime}, W_y^{\prime}, b^{\prime}, W_{out}, b$  are learnable parameters,  $y_{t-1}$  is the generated word at step  $t-1$ .
3.2.3 Decoder
For each decoding step, the key information from different parts of the source content which were picked by the TSE are utilized to generate the target word. In addition, the copy mechanism (Vinyals et al., 2015; Gu et al., 2016) is also adopted since some keywords in the title appeared in the source content. Different from the traditional copy mechanism with a pure sequence encoder, a hierarchical copy approach is proposed based on the hierarchical encoder framework. The attention weights over words within the same sentence and the sentence level attention scores are multiplied to calculate the copy probabilities for all words throughout the article content as shown in Equation 10. The final generation probability for word  $w$  at decoding step  $t$  over the extended vocabulary is calculated by Equation 11.
$$
p _ {c} ^ {t} = a _ {w} ^ {t} * a _ {s} ^ {t} \tag {10}
$$
$$
p ^ {t} (w) = p _ {g} ^ {t} p _ {v} ^ {t} (w) + \left(1 - p _ {g} ^ {t}\right) \sum_ {i: w _ {i} = w} p _ {c, i} ^ {t} \tag {11}
$$
3.3 Acme Coverage
As mentioned above, seq2seq models with copy mechanisms usually suffer from the repetition problem. Especially in the title generation tasks, the repeated words are conspicuous in a concise title and
resulting in poor readability. The commonly used Vanilla Coverage (VC) mechanism maintains a coverage vector  $cov^t$  to calculate the sum of attention distribution over all previous decoder steps. But for the title generation scenario, the copied word may appear in different positions with different attention weights, the vanilla coverage mechanism only penalizes the model to avoid repeatedly attending to the same locations but not the same word. Besides, since the attention weights in our model are in a hierarchical mode, the weights' values of closely located words may be similar sometimes. The vanilla coverage mechanism may wrongly penalize words that have not been generated actually but are essential in the following steps.
With the aim of making up for the two shortcomings of the vanilla coverage mechanism, we proposed a novel mechanism named Acme Coverage, which only sums over all the attention weights for the words generated at each decoding step if they appeared in the content actually. The final generation probability  $p^t$  over the extended vocabulary is used to select the truly generated word  $w^t$ , and all the attention weights corresponding to  $w^t$  in  $a_w^t$  will be added together for loss penalization. The calculation is shown in Equations 12 and 13, where  $I_{a_w^t}$  is an indicator function.
$$
c o v ^ {t} = \sum_ {t = 0} ^ {t - 1} I _ {a _ {w} ^ {t}} * a _ {w} ^ {t} \tag {12}
$$
$$
I _ {a _ {w} ^ {t}} = \left\{ \begin{array}{l l} 1 & i = = \operatorname {a r g m a x} \left(p ^ {t}\right) \\ 0 & \text {o t h e r s} \end{array} \right. \tag {13}
$$
Considering the AC mechanism, the calculation of Equation 5 of word-level attention weight will be changed to Equation 14.
$$
e _ {i} ^ {t} = v ^ {t} \tanh  \left(W _ {h} h _ {i} + W _ {s} s _ {t} + W _ {c} c o v _ {i} ^ {t} + b _ {a}\right) \tag {14}
$$
Similar to the work in (See et al., 2017), we also add the coverage loss in the final loss function to enhance the model's ability to accurately copy while avoiding duplication. The composite loss function is shown in Equation 15
$$
L _ {t} = - \log (p ^ {t} (w)) + \lambda \sum_ {i} \min  \left(a _ {w, i} ^ {t}, c o v _ {i} ^ {t}\right) \tag {15}
$$
We conduct sufficient experiments over two publicly available title generation datasets with brief
but informative titles to verify the effectiveness of our method, AESLC (Zhang and Tetreault, 2019), and Chinese Gigaword (Parker et al., 2011).
AESLC The AESLC is an annotated email subject line corpus which is a collection of email messages of employees in Enron Corporation. The average title length in AESLC is 4 words, much shorter than other summary generation tasks. There are 18k samples in AESLC, with train/val/test: 14,436/1,960/1,906. Notably, AESLC has two different targets in validation and test set, one is the original subject of the email, another is annotated subjects from three annotators according to the content.
Chinese Gigaword This dataset is a collection of Chinese news articles, it contains paragraphs from the Chinese Gigaword Fifth Edition release. The average lengths of the source articles and target titles are 665.5 and 16.3. In total there are more than 5M articles, and we sample 74689 for training and 9400/9345 for validation and testing.
To evaluate the performance of our proposed model and compare it with other baselines, we use the automatic metrics from text summarization and machine translation: ROUGE 1/2/L (Lin and Och, 2004) and METEOR (Denkowski and Lavie, 2014) to measure the quality of the generated titles.
Besides, human evaluation is also adopted to evaluate the quality of the generated news headlines in three dimensions. The first one is the fluency which indicates whether the title is grammatical correct and in high readability. The second one is to measure the relevance between the generated title and the input article. The last but most important one is the usability of the generated title which means whether the brief title can be used as a formal one in practical scenarios. The scores for fluency and relevance are between 1-5. A higher score means the quality is better. The usability is a simple 0/1 judgment. The average score of fluency, relevance and the available ratio are used to compare the quality of the generated titles by different settings of the coverage mechanism.
Our model is with 2 layers of bidirectional GRU (or Transformer) and GCN for word and sentence encoders respectively. The POS of each word and dependency parsing information are obtained by the
methods	DEV	TEST
R-1	R-2	R-L	METEOR	R-1	R-2	R-L	METEOR
PG-net (See et al., 2017)	18.02	5.73	16.63	10.83	17.02	5.45	15.78	10.31
Zhang and Tetreault (2019)	25.41	11.34	25.07	9.83	23.67	10.29	23.44	9.37
T5 (Raffel et al., 2020)	23.74	11.73	23.43	8.67	23.68	11.97	23.27	8.92
SimCLS (Liu and Liu, 2021)	25.67	12.36	25.42	9.72	24.52	12.35	24.03	9.63
PEGASUS-base (Zhang et al., 2020)	-	-	-	-	34.85	18.94	34.10	-
PEGASUS-large (Zhang et al., 2020)	-	-	-	-	37.69	21.85	36.84	-
Human Annotation	23.43	9.71	22.17	10.87	23.90	10.09	22.75	11.04
TSE-AC	26.28	11.53	25.73	11.17	24.91	10.91	24.27	11.09
TSE-AC-Trans	26.35	12.07	25.99	11.08	25.18	11.86	24.59	11.25
Table 2: The performance against the original subject of AESLC. The top two lines are results referred from (Zhang and Tetreault, 2019). The T5 and SimCLS are results conducted by ourselves with models proposed recently. Human Annotation means using annotated subjects from the third annotator as predict results. TSE-AC is our model with GRU layers while TSE-AC-Trans use Transformer layers instead.
methods	DEV	TEST
R-1	R-2	R-L	METEOR	R-1	R-2	R-L	METEOR
PG-net	23.37	7.36	20.99	16.27	23.31	7.28	20.83	15.68
Zhang and Tetreault (2019)	25.39	10.94	24.72	13.04	26.11	11.43	25.64	13.52
Original Subject	24.38	10.15	23.00	16.49	25.47	10.40	23.15	14.08
T5 (Raffel et al., 2020)	21.31	10.26	20.88	11.84	21.76	10.80	21.33	11.92
SimCLS (Liu and Liu, 2021)	22.14	10.03	21.25	12.33	22.08	10.82	21.57	12.40
TSE-AC	29.55	12.52	28.09	15.87	29.44	12.41	28.20	15.53
TSE-AC-Trans	30.01	13.29	29.13	16.55	30.20	13.32	29.07	16.17
Human Annotation	35.93	17.76	33.55	21.74	36.19	17.75	33.50	21.42
Table 3: The performance against two human annotations as references of AESLC. Original Subject means using the original subjects as predict results.
Spacy toolkit (Honnibal et al., 2020). The threshold of TF-IDF cosine similarity used in sentence-level adjacent matrix is 0.3. The decoder includes two layers of GRU (or Transformer) and one fully connected layer with vocabulary size 35000. The embedding size of words and POS are 256 as same as the hidden size. The Adam (Kingma and Ba, 2014) optimizer is used to train the model with learning rate and dropout ratio set to 0.0005 and 0.5. The coverage loss is added after 2 epochs of training steps with weight set to 1.0. The training is converged in 6 epochs for AESLC dataset and 20 epochs for Gigaword with one NVIDIA P100 GPU and batch size set to 8, the maximum value of gradient clip is 5.0. In the model prediction stage, we use beam search with size 4 to generate the titles for samples in the test sets. The code is available at https://github.com/alipay/Timestep-aware-SentenceEmbedding-and-AcmeCoverage.
Automatic Evaluation The results of experiments over AESLC are given in Table 2 and Table 3. In Table 2, where the original subjects are ground-truth, our model achieves the best results on all automatic evaluation metrics for both validation and test sets except Rouge-2 and the PEGA