SlowGuess/ACL_2020_OCR huggingface dataset:
MART is implemented in PyTorch (Paszke et al., 2017). We set the hidden size to 768, the number of transformer layers to 2, and the number of attention heads to 12. For positional encoding, we follow Vaswani et al. (2017) to use the fixed scheme. For memory module, we set the length of recurrent memory state to 1, i.e.,  $T_{m} = 1$ . We optimize the model following the strategy used by Devlin et al. (2019). Specifically, we use Adam (Kingma and Ba, 2014) with an initial learning rate of 1e-4,  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$ , L2 weight decay of 0.01, and learning rate warmup over the first 5 epochs. We train the model for at most 50 epochs with early stopping using CIDEr-D and batch size 16. We use greedy decoding as we did not observe better performance using beam search.
Vanilla Transformer This model originates from the transformer (Vaswani et al., 2017), proposed by Zhou et al. (2018) (more details in Section 3.1). It takes a single video segment as input and independently generates a single sentence describing the given segment. Note that Zhou et al. (2018) also have a separate proposal generation module, but here we only focus on its captioning module. To obtain paragraph-level captions, the independently generated single sentence captions are concatenated as the output paragraph.
Model	Re.	ActivityNet Captions (ae-test)	YouCookII (val)
B@4	M	C	R@4 ↓	B@4	M	C	R@4 ↓
VTransformer (2018)	X	9.31	15.54	21.33	7.45	7.62	15.65	32.26	7.83
Transformer-XL (2019)	✓	10.25	14.91	21.71	8.79	6.56	14.76	26.35	6.30
Transformer-XLRG	✓	10.07	14.58	20.34	9.37	6.63	14.74	25.93	6.03
MART	✓	9.78	15.57	22.16	5.44	8.00	15.9	35.74	4.39
Human	-	-	-	-	0.98	-	-	-	1.27
Table 1: Comparison with transformer baselines on ActivityNet Captions ae-test split and YouCookII val split. Re. indicates whether sentence-level recurrence is used. We report BLEU@4 (B@4), METEOR (M), CIDEr-D (C) and Repetition (R@4). VTransformer denotes vanilla transformer.
Det.	Re.	B@4	M	C	R@4 ↓
LSTM based methods
MFT (2018)	X	✓	10.29	14.73	19.12	17.71
HSE (2018)	X	✓	9.84	13.78	18.78	13.22
LSTM based methods with detection feature
GVD (2019)	✓	X	11.04	15.71	21.95	8.76
GVDsup (2019)	✓	X	11.30	16.41	22.94	7.04
AdvInf (2019)	✓	✓	10.04	16.60	20.97	5.76
Transformer based methods
VTransformer (2018)	X	X	9.75	15.64	22.16	7.79
Transformer-XL (2019)	X	✓	10.39	15.09	21.67	8.54
Transformer-XLRG (Ours) MART	X	✓	10.17	14.77	20.40	8.85
X	✓	10.33	15.68	23.42	5.18
Human	-	-	-	-	-	0.98
Table 2: Comparison with baselines on ActivityNet Captions ae-val split. Det. indicates whether the model uses detection feature. Models that use detection features are shown in gray background to indicate they are not in fair comparison with the others. Re. indicates whether sentence-level recurrence is used. VTransformer denotes vanilla transformer.
Transformer-XL Transformer-XL is proposed by Dai et al. (2019) for modeling long-term dependency in natural language. Here we adapt it for video paragraph captioning (more details in Section 3.3). The original design of Transformer-XL stops gradients from passing between different recurrent steps to save GPU memory and computation. To enable a more fair comparison with our model, we implemented a version that allows gradient flow through different steps, calling this Transformer-XLRG (Transformer-XL with Recurrent Gradient).
AdvInf AdvInf (Park et al., 2019) uses a set of three discriminators to do adversarial inference on a strong LSTM captioning model. The input features of the LSTM model are the concatenation of image recognition, action recognition, and object detection features. To encourage temporal coherence between consecutive sentences, the last hidden state from the previous sentence is used as input to the
decoder (Xiong et al., 2018; Gella et al., 2018). The three discriminators are trained adversarially and are specifically designed to reduce repetition and encourage fluency and relevance in the generated paragraph.
GVD An LSTM based model for grounded video description (Zhou et al., 2019). It uses densely detected object regions as inputs, with a grounding module that grounds generated words to the regions. Additionally, we also consider a GVD variant (GVDsup) that uses grounding supervision from Zhou et al. (2019).
MFT MFT (Xiong et al., 2018) uses an LSTM model with a similar sentence-level recurrence as in AdvInf (Park et al., 2019).
HSE HSE (Zhang et al., 2018) is a hierarchical model designed to learn both clip-sentence and paragraph-video correspondences. Given the learned contextualized video embedding, HSE uses a 2-layer LSTM to generate captions.
For AdvInf, MFT, HSE, GVD, and GVDsup, we obtain generated sentences from the authors. We only report their performance on ActivityNet Captions ae-val split to enable a fair comparison, as (i) AdvInf, MFT and HSE have different settings as ours, where ae-test videos are included as part of their validation set; (ii) we do not have access to the ae-test predictions of GVD and GVD-sup. For vanilla transformer, Transformer-XL and Transformer-XLRG, we borrow/modify the model implementations from the original authors and train them under the same settings as MART.
Automatic Evaluation Table 1 shows the results of MART and several transformer baseline methods. We observe stronger or comparable performance for the language metrics (B@4, M, C) for
MART wins (%)	VTransformer wins (%)	Delta
relevance	37	29.5	+7.5
coherence	42.8	26.3	+16.5
MART wins (%)	Transformer-XL wins (%)	Delta
relevance	40.0	39.5	+0.5
coherence	39.2	36.2	+3.0
both ActivityNet Captions and YouCookII datasets. For R@4, MART produces significantly better results compared to the three transformer baselines, showing its effectiveness in reducing redundancy in the generated paragraphs. Table 2 shows the comparison of MART with state-of-the-art models on ActivityNet Captions. MART achieves the best scores for both CIDEr-D and R@4 and has a comparable performance for B@4 and METEOR. Note that the best B@4 model, GVDsup (Zhou et al., 2019), and the best METEOR model, AdvInf (Park et al., 2019), both use strong detection features, and GVDsup has also used grounding supervision. Regarding the repetition score R@4, MART has the highest score. It outperforms the strong adversarial model AvdInf (Park et al., 2019) even in an unfair comparison where AdvInf uses extra detection features. Additionally, AdvInf has a time-consuming adversarial training and decoding process where a set of discriminator models are trained and used to re-rank candidate sentences, while MART can do much faster inference with only greedy decoding and no further post-processing. The comparisons in Table 1 and Table 2 show that MART is able to generate less redundant (thus more coherent) paragraphs while maintaining relevance to the videos.
Human Evaluation In addition to the automatic metrics, we also run human evaluation on Amazon Mechanical Turk (AMT) with 200 randomly sampled videos from ActivityNet Captions ae-test split, where each video was judged by three different AMT workers. We design a set of pairwise experiments (Pasunuru and Bansal, 2017b; Park et al., 2019), where we compare two models at a time. AMT workers are instructed to choose which caption is better or the two captions are not distinguishable based on relevance and coherence, respectively. The models are anonymized, and the predictions are shuffled. In total, we have 54 work
Table 3: Human evaluation on ActivityNet Captions aetest set w.r.t. relevance and coherence. Top: MART vs. vanilla transformer (VTransformer). Bottom: MART vs. Transformer-XL.
#hidden layers	mem. len.	Re.	B@4	M	C	R@4 ↓
#hidden layers
MART	1	1	✓	10.42	16.01	22.87	6.70
MART	5	1	✓	10.48	16.03	24.33	6.74
mem. len.
MART	2	2	✓	10.30	15.66	22.93	5.94
MART	2	5	✓	10.12	15.48	22.89	6.83
recurrence
MART w/o re.	2	-	X	9.91	15.83	22.78	7.56
MART	2	1	✓	10.33	15.68	23.42	5.18
Table 4: Model ablation on ActivityNet Captions ae-val split. Re. indicates whether sentence-level recurrence is used. mem. len. indicates the length of the memory state. MART w/o re. denotes a MART variant without recurrence. Top two scores are highlighted.
ers participated the MART vs. vanilla transformer experiments, 47 workers participated the MART vs. Transformer-XL experiments. In Table 3 we show human evaluation results, where the scores are calculated as the percentage of workers that have voted a certain option. With its sentence-level recurrence mechanism, MART is substantially better than the vanilla transformer model for both relevance and coherence. Compared to the strong baseline approach Transformer-XL, MART has similar performance in terms of relevance, but still reasonably better performance in terms of coherence.
Model Ablation We show model ablation in Table 4. MART models with recurrence have better overall performance than the variant without, suggesting the effectiveness of our recurrent memory design. We choose to use the model with 2 hidden layers and memory state length 1 as it shows a good balance between performance and computation.
Qualitative Examples In Figure 3, we show paragraph captions generated by vanilla transformer, Transformer-XL, and our method MART. Compared to the two baselines, MART produces more coherent and less redundant paragraphs. In particular, we noticed that vanilla transformer often uses incoherent pronouns/person mentions, while MART and Transformer-XL is able to use suitable pronouns/person mentions across the sentences and thus improve the coherence of the paragraph. Compare with Transformer-XL, we found that the paragraphs generated by MART have much less cross-sentence repetitions. We attribute MART's success to its recurrence design - the previous memory states are highly summarized, in which redundant information is removed. While there is less redu
Vanilla Transformer
He is sitting down in a chair. He continues playing the harmonica and ends by looking off into the distance. He continues playing the harmonica and looking off into the distance. He stops playing and looks at the camera.
A man is seen speaking to the camera while holding a harmonica. He continues playing the harmonica while looking at the camera. He continues playing the instrument and looking off into the distance. He continues playing and stops playing.
MART (ours)
A man is sitting down talking to the camera while holding a camera. He takes a harmonica and begins playing his harmonica. He continues playing the harmonica as he continues playing. He stops and looks at the camera.
A young man wearing a Cuervo black shirt stares and speaks to the camera as he sits on his chair. He puts a harmonica to his mouth and begins playing. He plays on for about a minute and is very into his song. He then puts the harmonica down and looks into the camera as the video comes to an end.
A girl is seen climbing across a set of monkey bars and leads into her climbing across a set of. He jumps off the monkey bars and lands on a bridge.
A young child is seen climbing across a set of monkey bars and climbing across a set of monkey bars. The boy jumps down and jumps down and jumps down.
A girl is seen speaking to the camera and leads into her climbing across a set of monkey bars. She jumps off the bar and walks back to the camera.
A little girl climbs the monkey bars of a play ground. Then, the little girl jumps to the ground and extend her arms.
Figure 3: Qualitative examples. Red/bold indicates pronoun errors (inappropriate use of pronouns), blue/italic indicates repetitive patterns, underline indicates content errors. Compared to baselines, our model generates more coherent, less repeated paragraphs while maintaining relevance.
A man on a diving board walks to the end. The man bounces on the board two times then dives into the water...
A girl is giving a small dog a bath. She has an orange bottle in her hand...
Figure 4: Nearest neighbors retrieved using memory states. Top row shows the query, the 3 rows below it are the top-3 nearest neighbors.
A young girl is seen walking to the end of a diving board with several other people around her...
A little girl stands on a diving board. Then the little girl jumps, flip and dives in the swimming pool...
dancy between sentences generated by MART, in Figure 3 (left), we noticed that repetition still exists within a single sentence, suggesting further efforts on reducing the repetition in single sentence generation. More examples are in the appendix.
Memory Ablation To explore whether the learned memory state could store useful information about the videos and captions, we conducted a video retrieval experiment on ActivityNet Captions train split with 10K videos, where we extract the
last step memory state in the first layer of a trained MART model for each video as its representation to perform nearest neighbor search with cosine similarity. Though not explicitly trained for the retrieval task, we observe some positive examples in the experiments. We show an example in Figure 4, the neighbors mostly show related activities.
In this work, we present a new approach – Memory-Augmented Recurrent Transformer (MART) for video paragraph captioning, where we designed an auxiliary memory module to enable recurrence in transformers. Experimental results on two standard datasets show that MART has better overall performance than the baseline methods. In particular, MART can generate more coherent, less redundant paragraphs without any degradation in relevance.
We thank the anonymous reviewers for their helpful comments and discussions. This work was performed while Jie Lei was an intern at Tencent AI Lab, Seattle, USA. It was later partially supported by NSF Awards CAREER-1846185, 1562098, DARPA KAIROS Grant FA8750-19-2-1004, and ARO-YIP Award W911NF-18-1-0336. The views contained in this article are those of the authors and not of the funding agency.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. Advances in
NeurIPS 2016 Deep Learning Symposium.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. 2015. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR.
David L Chen and William B Dolan. 2011. Collecting highly parallel data for paraphrase evaluation. In ACL.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2019. Uniter: Learning universal image-text representations. arXiv preprint arXiv:1909.11740.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NIPS 2014 Workshop on Deep Learning.
Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. In ACL.
Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the ninth workshop on statistical machine translation.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL*.
Spandana Gella, Mike Lewis, and Marcus Rohrbach. 2018. A dataset for telling the stories of social media videos. In EMNLP.
Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural tuning machines. arXiv preprint arXiv:1410.5401.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR.
Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, Jürgen Schmidhuber, et al. 2001. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In ICLR.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. 2017. Dense-captioning events in videos. In ICCV.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL.
Jae Sung Park, Marcus Rohrbach, Trevor Darrell, and Anna Rohrbach. 2019. Adversarial inference for multi-sentence video description. In CVPR.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In ICML.
Ramakanth Pasunuru and Mohit Bansal. 2017a. Multi-task video captioning with video and entailment generation. In ACL.
Ramakanth Pasunuru and Mohit Bansal. 2017b. Reinforced video captioning with entailment rewards. In EMNLP.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NeurIPS Autodiff Workshop.
Anna Rohrbach, Marcus Rohrbach, Wei Qiu, Annemarie Friedrich, Manfred Pinkal, and Bernt Schiele. 2014. Coherent multi-sentence video description with variable level of detail. In  $GCPR$ .
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In NeurIPS.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019. Videobert: A joint model for video and language representation learning. In ICCV.
Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from transformers. In EMNLP.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In CVPR.
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. 2019. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In ICCV.
Caiming Xiong, Stephen Merity, and Richard Socher. 2016a. Dynamic memory networks for visual and textual question answering. In ICML.
Yilei Xiong, Bo Dai, and Dahua Lin. 2018. Move forward and tell: A progressive generator of video descriptions. In ECCV.
Yuanjun Xiong, Limin Wang, Zhe Wang, Bowen Zhang, Hang Song, Wei Li, Dahua Lin, Yu Qiao, Luc Van Gool, and Xiaou Tang. 2016b. Cuhk & ethz & siat submission to activitynet challenge 2016. arXiv preprint arXiv:1608.00797.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. Msrvtt: A large video description dataset for bridging video and language. In CVPR.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurIPS.
Bowen Zhang, Hexiang Hu, and Fei Sha. 2018. Cross-modal and hierarchical modeling of video and text. In ECCV.
Luowei Zhou, Yannis Kalantidis, Xinlei Chen, Jason J. Corso, and Marcus Rohrbach. 2019. Grounded video description. In CVPR.
Luowei Zhou, Chenliang Xu, and Jason J. Corso. 2017. Towards automatic learning of procedures from web instructional videos. In AAAI.
Luowei Zhou, Yingbo Zhou, Jason J Corso, Richard Socher, and Caiming Xiong. 2018. End-to-end dense video captioning with masked transformer. In CVPR.
A.1 Additional Qualitative Examples
We show more caption examples in Figure 5. Overall, we see captions generated by models with sentence-level recurrence, i.e., MART and Transformer-XL, tend to be more coherent. Comparing with Transformer-XL, captions generated by MART are usually less repetitive. However, as shown in the two examples at the last row of Figure 5, all three models suffer from the content error, where the models are not able to recognize and describe the fine-grained details in the videos, e.g., gender and fine-grained objects/actions.
He is skateboarding down a road. He goes through the streets and goes. He is skateboarding down a road.
A man is riding a skateboard down a road. He is skateboarding down a road. He is skateboarding down a road.
A man is seen riding down a road with a person walking into frame and speaking to the camera. The man continues riding down the road while looking around to the camera and showing off his movements. The man continues to ride around while looking to the camera.
A camera pans all around an area and leads into a man speaking to the camera. Several shots of the area are shown as well as dogs and leads into a man riding a hill. The man rides a skateboard continuously around the area and ends by m up with the first man.
Figure 5: Additional qualitative examples. Red/bold indicates pronoun errors (inappropriate use of pronouns or person mentions), blue/italic indicates repetitive patterns, underline indicates content errors. Compared to baselines, our model generates more coherent, less repeated paragraphs while maintaining relevance.
She continues moving around the room and leads into her speaking to the camera. She continues moving around on the step and ends by speaking to the camera.
A woman is standing in a gym. She begins to do a step.
A woman is standing in a room talking. She starts working out on the equipment.
A woman is seen speaking to the camera and leads into her walking up and down the board. She then stands on top of the beam while speaking to the camera continuously.
The man then holds up a bottle of mouthwash and talks to the camera. The man then puts lotion on her face and begins rubbing it down. The man then begins to blow dry her face and shows off the camera.
A man is seen speaking to the camera while holding up a brush. He then rubs lotion all over his face and begins brushing his face. He then puts the lotion on the face and rubs it on the wall.
A man is seen speaking to the camera and leads into him holding up a bottle of water. The man then holds up a can and begins to shave his face. He finishes putting the paper into the mirror and smiles to the camera.
A girl's face is shown in front of the camera. She showed an orange bottle, read the label and squirt the orange content on her palm, showed the cream on the camera, then rub the cream all over her face. She bend down and rinse her face, when her face is visible on the camera her face is clear.
He continues speaking while holding the violin and showing how to play his hands. He continues playing the instrument while looking down at the camera. He continues playing the violin and then stops to speak to the camera.
A man is seen speaking to the camera while holding a violin. The man continues playing the instrument while moving his hands up and down. The man continues playing the instrument and ends by looking back to the camera.
A man is seen speaking to the camera while holding a violin and begins playing the instrument. The man continues to play the instrument while moving his hands up and down. He continues to play and ends by moving his hands up and down.
A man is seen looking to the camera while holding a violin. The man then begins playing the instrument while the camera zooms in on his fingers. The man continues to play and stops to speak to the camera.
Several shots are shown of people riding on the surf board and the people riding along the water. Several shots are shown of people riding around on a surf board and leads into several clips of people riding.
A large wave is seen followed by several shots of people riding on a surf board and riding along the. The people continue riding along the water while the camera pans around the area and leads into several more shots.
A man is seen riding on a surfboard and surfing on the waves. The man continues surfing while the camera captures him from several angles.
A man is seen moving along the water on a surf board while another person watches on the side. The person continues riding around and slowing down to demonstrate how to play.
A young girl is seen climbing across a set of monkey bars. A young child is seen climbing across a set of monkey bars. A little girl is standing on a platform in a playground.
A young child is seen standing before a set of monkey bars and begins climbing across monkey bars. The girl then climbs back and fourth on the bars.