SlowGuess/ACL_2020_OCR huggingface dataset:
Jie Lei $^{1*}$  Liwei Wang $^{2}$  Yelong Shen $^{3*}$  Dong Yu $^{2}$  Tamara L. Berg $^{1}$  Mohit Bansal $^{1}$
$^{1}$ UNC Chapel Hill  $^{2}$ Tencent AI Lab Seattle USA  $^{3}$ Microsoft Dynamics 365 AI
{jielei, tlberg, mbansal}@cs.unc.edu
{liweiwang, dyu}@tencent.com,{yeshe}@microsoft.com
Generating multi-sentence descriptions for videos is one of the most challenging captioning tasks due to its high requirements for not only visual relevance but also discourse-based coherence across the sentences in the paragraph. Towards this goal, we propose a new approach called Memory-Augmented Recurrent Transformer (MART), which uses a memory module to augment the transformer architecture. The memory module generates a highly summarized memory state from the video segments and the sentence history so as to help better prediction of the next sentence (w.r.t. coreference and repetition aspects), thus encouraging coherent paragraph generation. Extensive experiments, human evaluations, and qualitative analyses on two popular datasets ActivityNet Captions and YouCookII show that MART generates more coherent and less repetitive paragraph captions than baseline methods, while maintaining relevance to the input video events. $^{1}$
In video captioning, the task is to generate a natural language description capturing the content of a video. Recently, dense video captioning (Krishna et al., 2017) has emerged as an important task in this field, where systems first generate a list of temporal event segments from a video, then decode a coherent paragraph (multi-sentence) description from the generated segments. Park et al. (2019) simplifies this task as generating a coherent paragraph from a provided list of segments, removing the requirements for generating the event segments, and focusing on decoding better paragraph captions from the segments. As noted by Xiong et al.
(2018); Park et al. (2019), generating paragraph descriptions for videos can be very challenging due to the difficulties of having relevant, less redundant, as well as coherent generated sentences.
Towards this goal, Xiong et al. (2018) proposed a variant of the LSTM network (Hochreiter and Schmidhuber, 1997) that generates a new sentence conditioned on previously generated sentences by passing the LSTM hidden states throughout the entire decoding process. Park et al. (2019) further augmented the above LSTM caption generator with a set of three discriminators that score generated sentences based on defined metrics, i.e., relevance, linguistic diversity, and inter-sentence coherence. Though different, both these methods use LSTMs as the language decoder.
Recently, transformers (Vaswani et al., 2017) have proven to be more effective than RNNs (e.g., LSTM (Hochreiter and Schmidhuber, 1997), GRU (Chung et al., 2014), etc.), demonstrating superior performance in many sequential modeling tasks (Vaswani et al., 2017; Zhou et al., 2018; Devlin et al., 2019; Dai et al., 2019; Yang et al., 2019). Zhou et al. (2018) first introduced the transformer model to the video paragraph captioning task, with a transformer captioning module decoding natural language sentences from encoded video segment representations. This transformer captioning model is essentially the same as the original transformer (Vaswani et al., 2017) for machine translation, except that it takes a video representation rather than a source sentence representation as its encoder input. However, in such design, each video segment caption is decoded individually without knowing the context (i.e., previous video segments and the captions that have already been generated), thus often leading to inconsistent and redundant sentences w.r.t. previously generated sentences (see Figure 3 for examples). Dai et al. (2019) recognize this problem as context fragmentation in
* Work done while Jie Lei was an intern and Yelong Shen was an employee at Tencent AI Lab.
1All code is available open-source at https://github. com/jayleicn/recurrent-transformer
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2603-2614 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
the task of language modeling, where the transformers are operating on separated fixed-length segments, without any information flow across segments. Therefore, to generate more coherent video paragraphs, it is imperative to build a model that can span over multiple video segments and capture longer range dependencies.
Hence, in this work, we propose the Memory-Augmented Recurrent Transformer (MART) model (see Section 3 for details), a transformer-based model that uses a shared encoder-decoder architecture augmented with an external memory module to enable the modeling of the previous history of video segments and sentences. Compared to the vanilla transformer video paragraph captioning model (Zhou et al., 2018), our first architecture change is the unified encoder-decoder design, i.e., the encoder and decoder in MART use shared transformer layers rather than separated as in Zhou et al. (2018); Vaswani et al. (2017). This unified encoder-decoder design is inspired by recent transformer language models (Devlin et al., 2019; Dai et al., 2019; Sun et al., 2019) to prevent overfitting and reduce memory usage. Additionally, the memory module works as a memory updater that updates its memory state using both the current inputs and previous memory state. The memory state can be interpreted as a container of the highly summarized video segments and caption history information. At the encoding stage, the current video segment representation is enhanced with the memory state from the previous step using cross-attention (Vaswani et al., 2017). Hence, when generating a new sentence, MART is aware of the previous contextual information and can generate paragraph captions with higher coherence and lower repetition.
Transformer-XL (Dai et al., 2019) is a recently proposed transformer language model that also uses recurrence, and is able to resolve context fragmentation for language modeling (Dai et al., 2019). Different from MART that uses a highly-summarized memory to remember history information, Transformer-XL directly uses hidden states from previous segments. We modify the Transformer-XL framework for video paragraph captioning and present it as an additional comparison. We benchmark MART on two standard datasets: ActivityNet Captions (Krishna et al., 2017) and YouCookII (Zhou et al., 2017). Both automatic evaluation and human evaluation show that MART generates more satisfying results than
previous LSTM-based approaches (Xiong et al., 2018; Zhou et al., 2019; Zhang et al., 2018) and transformer-based approaches (Zhou et al., 2018; Dai et al., 2019). In particular, MART can generate more coherent (e.g., coreference and order), less redundant paragraphs without losing paragraph accuracy (visual relevance).
Video Captioning Recently, video captioning has attracted much attention from both the computer vision and the natural language processing community. Methods for the task share the same intrinsic nature of taking a video as the input and outputting a language description that can best describe the content, though they differ from each other on whether a single sentence (Wang et al., 2019; Xu et al., 2016; Chen and Dolan, 2011; Pasunuru and Bansal, 2017a) or multiple sentences (Rohrbach et al., 2014; Krishna et al., 2017; Xiong et al., 2018; Zhou et al., 2018; Gella et al., 2018; Park et al., 2019) are generated for the given video. In this paper, our goal falls into the category of generating a paragraph (multiple sentences) conditioned on an input video with several pre-defined event segments.
One line of work (Zhou et al., 2018, 2019) addresses the video paragraph captioning task by decoding each video event segment separately into a sentence. The final paragraph description is obtained by concatenating the generated single sentence descriptions. Though individual sentences may precisely describe the corresponding event segments, when put together the sentences often become inconsistent and redundant. Another line of works (Xiong et al., 2018; Gella et al., 2018) use the LSTM decoder's last (word) hidden state from the previous sentence as the initial hidden state for the next sentence decoding, thus enabling information flow from previous sentences to subsequent sentences. While these methods have shown better performance than their single sentence counterpart, they are still undesirable as the sentence-level recurrence is achieved at word-level, and the context history information quickly decays due to vanishing gradients (Pascanu et al., 2013) problem. Additionally, these designs also have difficulty modeling long-term dependencies (Hochreiter et al., 2001). In comparison, the recurrence in MART resides in the sentence or segment level and is thus more robust to the aforementioned problems. AdvInf (Park
et al., 2019) augments the above LSTM word-level recurrence methods with adversarial inference, using a set of separately trained discriminators to re-rank the generated sentences. The techniques in AdvInf can be viewed as an orthogonal way of generating captions with better quality.
Transformers Transformer (Vaswani et al., 2017) is used as the basis of our approach. Different from RNNs (e.g., LSTM (Hochreiter and Schmidhuber, 1997), GRU (Chung et al., 2014), etc) that use recurrent structure to model long-term dependencies, transformer relies on self-attention to learn the dependencies between input words. Transformers have proven to be more efficient and powerful than RNNs, with superior performance in many sequential modeling tasks, including machine translation (Vaswani et al., 2017), language modeling/pre-training (Devlin et al., 2019; Dai et al., 2019; Yang et al., 2019) and multi-modal representation learning (Tan and Bansal, 2019; Chen et al., 2019; Sun et al., 2019). Additionally, Zhou et al. (2018) have shown that a transformer model can generate better captions than the LSTM model.
However, transformer architectures are still unable to model history information well. This problem is identified in the task of language modeling as context fragmentation (Dai et al., 2019), i.e., each language segment is modeled individually without knowing its surrounding context, leading to inefficient optimization and inferior performance. To resolve this issue, Transformer-XL (Dai et al., 2019) introduces the idea of recurrence to the transformer language model. Specifically, the modeling of a new language segment in Transformer-XL is conditioned on hidden states from previous language segments. Experimental results show Transformer-XL has stronger language modeling capability than the non-recurrent transformer. Transformer-XL directly uses all the hidden states from the previous segment to enable recurrence. In comparison, our MART uses highly summarized memory states, making it more efficient in passing useful semantic or linguistic cues to future sentences.
Though our method provides a general temporal multi-modal learning framework, we focus on the video paragraph captioning task in this paper. Given a video  $V$ , with several temporally ordered event segments  $[e_1,e_2,\dots ,e_T]$ , the task is to generate a coherent paragraph consisting of multiple sen
Figure 1: Vanilla transformer video captioning model (Zhou et al., 2018).  $PE$  denotes Positional Encoding,  $TE$  denotes token Type Embedding.
tences  $[s_1, s_2, \dots, s_T]$  to describe the whole video, where sentence  $s_t$  should describe the content in the segment  $e_t$ . In the following, we first describe the baseline transformer that generates sentences without recurrent architecture, then introduce our approach - Memory-Augmented Recurrent Transformer (MART). Besides, we also compare MART with the recently proposed Transformer-XL (Dai et al., 2019) in detail.
3.1 Background: Vanilla Transformer
We start by introducing the vanilla transformer video paragraph captioning model proposed by Zhou et al. (2018), which is an application of the original transformer (Vaswani et al., 2017) model for video paragraph captioning. An overview of the model is shown in Figure 1. The core of the architecture is the scaled dot-product attention. Given query matrix  $Q \in \mathbb{R}^{T_q \times d_k}$ , key matrix  $K \in \mathbb{R}^{T_v \times d_k}$  and value matrix  $V \in \mathbb{R}^{T_v \times d_v}$ , the attentional output is computed as:
$$
A (Q, K, V) = \operatorname {s o f t m a x} \left(\frac {Q K ^ {\top}}{\sqrt {d _ {k}}}, \dim = 1\right) V,
$$
where  $\mathrm{softmax}(\cdot ,\mathrm{dim} = 1)$  denotes performing softmax at the second dimension of the input. Combining  $h$  paralleled scaled dot-product attention,
Figure 2: Left: Our proposed Memory-Augmented Recurrent Transformer (MART) for video paragraph captioning. Right: Transformer-XL (Dai et al., 2019) model for video paragraph captioning. Relative PE denotes Relative Positional Encoding (Dai et al., 2019).  $SG(\cdot)$  denotes stop-gradient,  $\odot$  denotes Hadamard product.
we obtain the multi-head attention (Vaswani et al., 2017), we denote it as MultiHeadAtt  $(Q, K, V)$ . The attention formulation discussed above is quite general. It can be used for various purposes, such as self-attention (Vaswani et al., 2017) where query, key, and value matrix are all the same, and cross-attention (Vaswani et al., 2017) where the query matrix is different from the key and value matrix. In this paper, we also use multi-head attention for memory aggregation and update, as discussed later.
The vanilla transformer video paragraph captioning model has  $N$  encoder layers and  $N$  decoder layers. At the  $l$ -th encoder layer, the multi-head attention module takes the last layer's hidden states  $H^{l-1}$  as inputs and performs self-attention. The attentional outputs are then projected by a feedforward layer. At the  $l$ -th decoder layer, the model first encodes the last decoder layer's hidden states using masked multi-head attention. It then uses multi-head attention, with the masked outputs as query matrix, and the hidden states  $H^l$  from  $l$ -th encoder layer as key and value matrix to gather
information from the encoder side. Similarly, a feed-forward layer is used to encode the sentences further. Residual connection (He et al., 2016) and layer-normalization (Ba et al., 2016) are applied for each layer, for both encoder and decoder.
3.2 Memory-Augmented Recurrent Transformer
The vanilla transformer captioning model follows the classical encoder-decoder architecture, where the encoder and decoder network are separated. In comparison, the encoder and decoder are shared in MART, as shown in Figure 2 (left). The video and text inputs are firstly separately encoded and normalized. We denote the encoded video and text embeddings as  $H_{video}^{0} \in \mathbb{R}^{T_{video} \times d}$  and  $H_{text}^{0} \in \mathbb{R}^{T_{text} \times d}$ , where  $T_{video}$  and  $T_{text}$  are the lengths of video and text, respectively.  $d$  denotes the hidden size. We then concatenate these two embeddings as input to the transformer layers:  $H^{0} = [H_{video}^{0}; H_{text}^{0}] \in \mathbb{R}^{T_{c} \times d}$ , where  $[]$  denotes concatenation,  $T_{c} = T_{video} + T_{text}$ . This unified encoder-decoder design is inspired by recent works on multi-modal representation learning (Chen et al., 2019; Sun et al., 2019). We also use two trainable
Memory-Augmented Recurrent Transformer (at step  $t$ )
Transformer-XL (at step  $t$ )
${}^{2}$  masked multi-head attention is used to prevent the model from seeing future words (Vaswani et al., 2017).
token type embedding vectors to indicate whether an input token is from video or text, similar to Devlin et al. (2019) where the token type embeddings are added to indicate different input sequences. We ignore the video token positions and only consider the text token positions when calculating loss and generating words.
While the aforementioned vanilla transformer is a powerful method, it is less suitable for video paragraph captioning due to its inability to utilize video segments and sentences history information. Thus, given the unified encoder-decoder transformer, we augment it with an external memory module, which helps it to utilize video segments and the corresponding caption history to generate the next sentence. An overview of the memory module is shown in Figure 2 (left). At step  $t$ , i.e., decoding the  $t$ -th video segment, the  $l$ -th layer aggregates the information from both its intermediate hidden states  $\bar{H}_t^l \in \mathbb{R}^{T_c \times d}$  and the memory states  $M_{t-1}^l \in \mathbb{R}^{T_m \times d}$  ( $T_m$  denotes memory state length or equivalently #slots in the memory) from the last step, using a multi-head attention. The input query matrix of the multi-head attention  $Q = \bar{H}_t^l$ , key and value matrices are  $K, V = [M_{t-1}^l; \bar{H}_t^l] \in \mathbb{R}^{(T_m + T_c) \times d}$ . The memory augmented hidden states are further encoded using a feed forward layer and then merged with the intermediate hidden states  $\bar{H}_t^l$  using a residual connection and layer norm to form the hidden states output  $H_t^l \in \mathbb{R}^{T_c \times d}$ . The memory state  $M_{t-1}^l$  is updated as  $M_t^l$ , using the intermediate hidden states  $\bar{H}_t^l$ . This process is conducted in the Memory Updater module, illustrated in Figure 2. We summarize the procedure below:
$$
S _ {t} ^ {l} = \operatorname {M u l t i H e a d A t t} \left(M _ {t - 1} ^ {l}, \bar {H} _ {t} ^ {l}, \bar {H} _ {t} ^ {l}\right),
$$
$$
C _ {t} ^ {l} = \tanh (W _ {m c} ^ {l} M _ {t - 1} ^ {l} + W _ {s c} ^ {l} S _ {t} ^ {l} + b _ {c} ^ {l}),
$$
$$
Z _ {t} ^ {l} = \operatorname {s i g m o i d} \left(W _ {m z} ^ {l} M _ {t - 1} ^ {l} + W _ {s z} ^ {l} S _ {t} ^ {l} + b _ {z} ^ {l}\right),
$$
$$
M _ {t} ^ {l} = \left(1 - Z _ {t} ^ {l}\right) \odot C _ {t} ^ {l} + Z _ {t} ^ {l} \odot M _ {t - 1} ^ {l},
$$
where  $\odot$  denotes Hadamard product,  $W_{mc}^{l}$ ,  $W_{sc}^{l}$ ,  $W_{mz}^{l}$ , and  $W_{sz}^{l}$  are trainable weights,  $b_{c}^{l}$  and  $b_{z}^{l}$  are trainable bias.  $C_t^l\in \mathbb{R}^{T_m\times d}$  is the internal cell state.  $Z_{t}^{l}\in \mathbb{R}^{T_{m}\times d}$  is the update gate that controls which information to retain from the previous memory state, and thus reducing redundancy and maintaining coherence in the generated paragraphs.
This update strategy is conceptually similar to LSTM (Hochreiter and Schmidhuber, 1997) and GRU (Chung et al., 2014). It differs in that multi
head attention is used to encode the memory state and thus multiple memory slots are supported instead of a single one in LSTM and GRU, which gives it a higher capacity of modeling complex relations. Recent works (Sukhbaatar et al., 2015; Graves et al., 2014; Xiong et al., 2016a) introduce a memory component into neural networks, where the memory is mainly designed to memorize facts in the input context to support downstream tasks, e.g., copy (Graves et al., 2014) or question answering (Sukhbaatar et al., 2015; Xiong et al., 2016a). In comparison, the memory in MART is designed to memorize the sequence generation history to support the coherent generation of the next sequence.
3.3 Comparison with Transformer-XL
Transformer-XL (Dai et al., 2019) is a recently proposed transformer-based language model that uses a segment-level recurrence mechanism to capture the long-term dependency in context. In Figure 2 (right) we show a modified version of Transformer-XL for video paragraph captioning. At step  $t$ , at its  $l$ -th layer, Transformer-XL takes as inputs the last layer's hidden states from both the current step and the last step, which we denote as  $H_{t}^{l-1}$  and  $SG(H_{t-1}^{l-1})$ , where  $SG(\cdot)$  stands for stop-gradient, and is used to save GPU memory and computation (Dai et al., 2019). The input query matrix of the multi-head attention  $Q = H_{t}^{l-1}$ , key and value matrices are  $K, V = [SG(H_{t-1}^{l-1}); H_{t}^{l-1}]$ . Note the multi-head attention here is integrated with relative positional encoding (Dai et al., 2019).
Both designed to leverage the long-term dependency in context, the recurrence in Transformer-XL is between  $H_{t}^{l}$  and  $H_{t - 1}^{l - 1}$ , which shifts one layer downwards per step. This mismatch in representation granularity may potentially be harmful to the learning process and affect the model performance. In contrast, the recurrence in MART is between  $\bar{H}_t^l$  and  $M_{t - 1}^{l}$  (updated using  $\bar{H}_{t - 1}^{l}$ ) of the same layer. Besides, Transformer-XL directly uses all the hidden states from the last step to enable recurrence, which might be less effective as less relevant and repetitive information is also passed along. In comparison, MART achieves recurrence by using memory states that are highly summarized from previous steps, which may help the model to reduce redundancy and only keep important information from previous steps.
We conducted experiments on two popular benchmark datasets, ActivityNet Captions (Krishna et al., 2017) and YouCookII (Zhou et al., 2017). We evaluate our proposed MART and compare it with various baseline approaches.
4.1 Data and Evaluation Metrics
Datasets ActivityNet Captions (Krishna et al., 2017) contains 10,009 videos in train set, 4,917 videos in val set. Each video in train has a single reference paragraph while each video in val has two reference paragraphs. Park et al. (2019) uses the same set of videos (though different segments) in val for both validation and test. To allow better evaluation of the models, we use splits provided by Zhou et al. (2019), where the original val set is split into two subsets: ae-val with 2,460 videos for validation and ae-test with 2,457 videos for test. This setup makes sure the videos used for test will not be seen in validation. YouCookII (Zhou et al., 2017) contains 1,333 training videos and 457 validation videos. Each video has a single reference paragraph. Both datasets come with temporal event segments annotated with human written natural language sentences. On average, there are 3.65 event segments for each video in ActivityNet Captions, 7.7 segments for each video in YouCookII.
Data Preprocessing We use aligned appearance and optical flow features extracted at 2FPS to represent videos, provided by Zhou et al. (2018). Specifically, for appearance, 2048D feature vectors from the 'Flatten-673' layer in ResNet-200 (He et al., 2016) are used; for optical flow, 1024D feature vectors from the 'global pool' layer of BN-Inception (Ioffe and Szegedy, 2015) are used. Both networks are pre-trained on ActivityNet (Caba Heilbron et al., 2015) for action recognition, provided by (Xiong et al., 2016b). We truncate sequences longer than 100 for video and 20 for text and set the maximum number of video segments to 6 for ActivityNet Captions and 12 for YouCookII. Finally, we build vocabularies based on words that occur at least 5 times for ActivityNet Captions and 3 times for YouCookII. The resulting vocabulary contains 3,544 words for ActivityNet Captions and 992 words for YouCookII.
Evaluation Metrics (Automatic and Human) We evaluate the captioning performance at paragraph-level, following (Park et al., 2019; Xiong
et al., 2018), reporting numbers on standard metrics, including BLEU@4 (Papineni et al., 2002), METEOR (Denkowski and Lavie, 2014), CIDER-D (Vedantam et al., 2015). Since these metrics mainly focus on whether the generated paragraph matches the ground-truth paragraph, they fail to evaluate the redundancy of these multi-sentence paragraphs. Thus, we follow previous works (Park et al., 2019; Xiong et al., 2018) to evaluate repetition using R@4. It measures the degree of N-gram  $(N = 4)$  repetition in the descriptions. Besides the automated metrics, we also conduct human evaluations to provide additional comparisons between the methods. We consider two aspects in human evaluation, relevance (i.e., how related is a generated paragraph caption to the content of the given video) and coherence (i.e., whether a generated paragraph caption reads fluently and is linguistically coherent over its multiple sentences).