dataesr/scientific-paragraphs-categorization huggingface dataset:
With the broad success of transformer architectures  [4]  in the Natural Language Processing (NLP) domain, recent years have featured a trend toward the use of attention mechanisms in both encoder and decoder. Most models  [5] [6]  [7]  rely on large datasets to learn more discriminative multimodal features, with instructional datasets such as HowTo100M  [10]  and Cooking312k  [11]  being advantageous due to their aligned speech and transcripts available. Several approaches  [5] [6]  [12]   [13]  pre-train sequence-to-sequence models on such unlabeled data using denoising auto-encoders. In this case, the input is purposely modified by masking random words/frames to create artificial noise. This process enables the model to learn robust representations by reconstructing the original input from the corrupted version.
The proposed framework is illustrated in Figure  1. It includes a modality-attention module that uses cross-correlation to simultaneously learn text-guided video features and video-guided textual features. Furthermore, we introduce an auxiliary task based on contrastive learning between videotranscript and caption embeddings, which improves the representation of inter-modal relationships (Figure  1 ). The objective of the contrastive loss function is to align the visualtranscript representation with the correct caption representation, while simultaneously distinguishing them from other captions within the batch. The alignment ensures that the model focuses on capturing relevant features by discerning meaningful multimodal interactions from irrelevant or unrelated information. Furthermore, the integration of a contrastive loss enhances the model's generalization capability while mitigating overfitting.
It is important to note that the caption information is strictly used during the training phase. It serves as a form of supervision to guide the model in learning grounded multimodal representations. During inference, the model just uses the knowledge and patterns learned during the training phase to generate the predicted caption. In view of the success of attention mechanisms, we leverage the transformer architecture for both the encoder and decoder. To summarize, the main contributions of the paper are the following: 1) A novel multimodal video captioning framework that incorporates various attention mechanisms to learn inter-modal and intra-modal representations. 2) A modality-attention strategy that employs crosscorrelation to reduce the gap between the visual and textual modality 3) The use of a contrastive loss between the videotranscript multimodal representation and the caption embedding to improve the understanding of inter-modal relationships and underlying semantics.
VOLUME XX, 2017 4) A comprehensive experimental evaluation on the challenging MSRVTT dataset, which shows that the proposed method outperforms state-of-the-art approaches, reaching a 0. 4408 BLEU4 score. 5) A fine-level investigation of the impact of each modality in the framework. The rest of the paper is organized as follows. In Section II, we present the state-of-the-art video captioning methods. Two families of approaches are here identified, including videobased techniques that rely solely on visual cues and multimodal methods that integrate multiple modalities. Section III provides a comprehensive overview of the proposed methodology, detailing the key steps involved. In section IV, we provide the different training strategies used to optimize the model. Section V presents the experimental setup, dataset, and ablation studies conducted.
The model utilizes a soft attention mechanism and leverages InceptionV4  [24]  to extract semantic features from the video. To capture temporal dynamics within the video, the 2D-CNN architecture has been later extended to 3D-CNN  [1]. Xu et al. [25]  introduce a two-module model for video captioning. A proposal module extracts features using 3D convolutional layers (C3D), while a so-called segment proposal network (SPN) is used for obtaining temporal segments. The model maps the visual representation onto a common vector space, while the syntactic representation relies on the Part-of-Speech (POS) tagging structures of the video description. Hemalatha and Sekhar  [28]  introduce a video captioning approach that incorporates domain-specific decoders through the use of a domain classifier. The model utilizes ResNet152 for extracting 2D-CNN features and a 3D-CNN for extracting temporal features.
To obtain a video representation, both the 2D-CNN and 3D-CNN features are aggregated using VLAD  [29]. For sentence generation, many existing approaches rely on recurrent neural networks (RNNs) such as LSTM  [26]  and GRU  [27]  to generate the caption. Yao et al. [30] , Donahue et al. [31]  and Venugopalan et al. [32]  use the LSTM architecture for yielding variable-length video descriptions. Guo et al. [33]  further incorporate attention mechanisms within the LSTM model to refine the captions. Similarly, Zhang et al. [34]  introduce a hierarchical decoder with temporal or spatial attention. The model implements a teacher-recommended learning system to leverage external language models and incorporate linguistic information. Overall, visual-based approaches primarily focus on leveraging visual cues to generate accurate and descriptive captions.
VOLUME XX, 2017 However, multimodal video captioning also presents several challenges. One major challenge concerns the alignment between different modalities, as the content of the visual and textual channels may not always be perfectly synchronized  [10] [37]  [40]. Furthermore, the size and complexity of multimodal datasets can raise challenges for training models that are both accurate and efficient  [38]   [39]. To tackle such issues, several works  [11] [13]  [35] [40] use instructional videos  [10]   [37] , where the synchronization between visual content and subtitles is more favorable for video captioning task. While such videos are useful for training, they have a specific structure that may not be representative of real-world scenarios  [14]. This makes it difficult to generalize the model on unseen data.
Furthermore, real-world speech tends to be less structured, with key actions or events in the video not always corresponding to the same segments in the input transcript. To address the visually misaligned narrations, various approaches have employed contrastive learning between video and transcript. For instance, MIL-NCE  [9]   Traditionally, most existing methods have applied the contrastive loss to the outputs of visual and text encoders, typically before the multimodal fusion stage  [42]. The primary aim of this loss is to establish alignment between the video and transcript during the pre-training phase. In our approach, we tackle the alignment challenge differently by incorporating the modality attention module. This module is specifically designed to bridge the gap between video and text modalities before feeding them into a transformer encoder.
Utilizing cross-correlation, the modality attention module generates text-conditioned visual features and videoconditioned textual features, facilitating more effective alignment. Our model is extensively evaluated on a diverse range of YouTube videos using the MSRVTT dataset. Conversely, the contrastive loss serves the purpose of aligning the multimodal representation of the input with its corresponding caption. In contrast to previous state-of-the-art models, we apply this loss to the output of the multimodal transformer.
2) TRAINING STRATEGIES In recent years, vision-language pre-training has gained considerable popularity within the research community VOLUME XX, 2017  [55]   [56][57][58]. This approach involves an initial phase where multimodal models are pre-trained on extensive datasets in an unsupervised manner, followed by subsequent fine-tuning for specific downstream tasks (e. g. video captioning, action recognition, video question answering). Typically, the considered datasets comprise videos along with their associated transcripts, a resource that is abundantly available. These methods learn multimodal representations by formulating proxy tasks such as masked language modeling  [5] [6], or vision-language matching  [55]   [56]. The paradigm of pre-training followed by fine-tuning for multimodal models is undeniably effective and has yielded remarkable results across various applications  [42]   [59].
In our case, due to hardware constraints, we opt for an alternative strategy by forgoing pretraining altogether. Despite this, we demonstrate that competitive results can still be achieved. The proposed approach leverages the available resources efficiently, focusing on task-specific training without the need for massive pre-training datasets or extensive computational power. This resource-aware approach not only makes multimodal modeling accessible to a wider range of users and applications but also highlights the potential for effective multimodal model development in resource-constrained environments.
A. FEATURE EXTRACTION The feature extraction process concerns the two components involved, which are the visual and textual (with both transcript and caption) data. In order to acquire the visual representations, we utilize a uniform sampling approach to divide the video into N fixedlength, non-overlapping clips of 16 frames each. The clips are then processed with the help of the S3D network  [1] , which is designed to learn robust video representations. Prior to use, the S3D model has been pre-trained on HowTo100M  [10]  with the MIL-NCE technique  [9]. This technique is widely adopted in multimodal learning  [6] [41]  [60]  for its robust handling of diverse and noisy data sources. The feature activations before the final fully connected layer are extracted and we apply average pooling to generate a dv = 1024dimensional vector (the v subscript stands here for visual).
Subsequently, a feed forward network that includes a linear projection, followed by the GeLU  [51]  activation function and layer normalization, is used to yield the final feature vector (of the same size dv). The resulting visual features are represented as a 𝑁 × 𝑑 𝑣 matrix, denoted by 𝑉. Let us underline that the S3D model is used only as a backbone for feature extraction and its weights are frozen. A video transformer is further employed to effectively capture the dependencies between frames in video clips and learn the inherent temporal dynamics of video objects, actions, and scenes. This approach enables us to learn grounded visual features that are specifically optimized for the task of video captioning, without being restricted to preextracted features from external models. In addition, using a pre-extracted feature-based model with a transformer architecture can significantly reduce the computational cost of training, as the S3D model can be pre-trained on largescale video datasets and the transformer can be fine-tuned on a smaller dataset dedicated to video captioning.
To take into account the dynamic dependencies between clips, we employ temporal attention on the feature vector V. Our approach is motivated by the observation that video data often contains redundant information, and only a limited number of clips contain discriminative information that is relevant to the video captioning task. For this reason, a multihead temporal attention mechanism is applied on the visual descriptor V. For each attention head ℎ ∈ {1, … , 𝐻 𝑣 } (where 𝐻 𝑣 denotes the number of visual attention heads), we first compute the associated 𝑄𝑢𝑒𝑟𝑦 ℎ 𝑉 , 𝐾𝑒𝑦 ℎ 𝑉 𝑎𝑛𝑑 𝑉𝑎𝑙𝑢𝑒 ℎ 𝑉 components defined as: 𝑄𝑢𝑒𝑟𝑦 ℎ 𝑣 = 𝑉𝑊 𝑞𝑢𝑒𝑟𝑦,ℎ 𝑣 ; 𝐾𝑒𝑦 ℎ 𝑣 = 𝑉𝑊 𝑘𝑒𝑦,ℎ 𝑣 (1) 𝑉𝑎𝑙𝑢𝑒 ℎ 𝑣 = 𝑉𝑊 𝑣𝑎𝑙𝑢𝑒,ℎ 𝑣 (2) where ). The visual temporal attention for a given attention head is computed as: 𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛 ℎ 𝑣 = 𝑠𝑜𝑓𝑡𝑚𝑎𝑥( 𝑄𝑢𝑒𝑟𝑦 ℎ 𝑣 (𝐾𝑒𝑦 ℎ 𝑣 ) 𝑇 √𝑑 𝑣 /𝐻 𝑣 )𝑉𝑎𝑙𝑢𝑒 ℎ 𝑣 (3) where superscript T denotes the matrix transpose operator.
The attention heads are then concatenated under the form of a (𝑁 × 𝑑 𝑣 ) matrix, denoted by 𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛 𝑣 and globally gathering the visual representation. An additional projection is considered in order to obtain the final visual representation, denoted by 𝐹 𝑣 and defined as: where 𝑊 𝑐𝑜𝑛𝑣𝑒𝑟𝑡 𝑣 is a learnable matrix of size (𝑑 𝑣 × 𝑑). This final operation performs the dimensionality conversion of the visual feature to a common dimension d that will also be used for the textual representation. 𝐹 𝑣 = 𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛 𝑣 𝑊 𝑐𝑜𝑛𝑣𝑒𝑟𝑡 𝑣 (4) Concerning the textual data, we consider the audio transcript (if the audio channel includes speech) as well as the video captions. To obtain the audio transcript from the input video, we utilize the Whisper model  [52] , which is an ASR algorithm that exhibits human-level robustness in English speech recognition, even in the presence of background noise and reverberation.
Whatever the source (audio transcript or caption), the textual data undergo a tokenization process using WordPieces  [53] , which segments the text into sub-words using a vocabulary of 𝑆 𝑣𝑜𝑐 = 30,000 tokens. The tokenized sequences are fed into the BERT-based uncased model  [54] , following previous state-of-the-art methods  [59] , to perform the embedding. As recommended in  [51] , the first token in the input sequence is represented as a dedicated [CLS] token, and the final one is represented by a so-called [SEP] token. To achieve equal length for all the tokenized text sequences, we expand the sentence using padding, with the help of a dedicated [PAD] token. Let us denote by M the length of the padded tokenized sequences, which correspond to the maximal number of tokens that are allowed to appear in a given sentence.
Let us also mention that a random masking of the tokens can also be considered. In this case, the input token is replaced by a dedicated token, denoted by  [MASK]. The BERT approach also employs a self-attention mechanism, yielding in output a (𝑀 × 𝑑 𝐵𝐸𝑅𝑇 ) feature matrix, with 𝑑 𝐵𝐸𝑅𝑇 = 768, corresponding to the activations of the last BERT layer. The text embedding approach is applied to both transcript and caption data. Similarly to the visual component, the transcript feature matrix is finally converted into a (𝑀 × 𝑑) matrix denoted by 𝐹 𝑡 , with d being the common dimension considered also for the visual representation. The caption feature matrix, denoted by 𝐹 𝑐 , does not require projection onto a space of common dimension (see its utilization in section IV) and thus remains of size (𝑀 × 𝑑 𝐵𝐸𝑅𝑇 ).
B. MODALITY ATTENTION MODULE Modeling visual and textual dynamics within and across modalities is a highly intricate task. To overcome such a challenge, we have developed a modality-attention module (Figure  3 ) that effectively captures both intra and inter-modal relationships between the visual and audio transcript modalities. It is designed to bridge the gap between features 𝐹 𝑣 and 𝐹 𝑡 , which are generated from separate models trained on different tasks. As we use real-life videos as input, in a majority of cases the feature vectors are not well-aligned. Figure  4  illustrates some video examples with their respective transcripts. We observe that people tend to speak in a disorganized manner, and the key actions or events in the video do not necessarily correspond to the same segment of the input text.
To avoid information loss during the cross-correlation phase, we have considered a dense skip connection technique. This means that we add the reweighted features 𝐹 𝑡-𝑣 and 𝐹 𝑣-𝑡 to the original representation of each modality, and regularize the result with the help of a tanh function: 𝐹 ̂𝑡-𝑣 = 𝑡𝑎𝑛ℎ (𝐹 𝑡-𝑣 + 𝐹 𝑡 ) (8) 𝐹 ̂𝑣-𝑡 = 𝑡𝑎𝑛ℎ (𝐹 𝑣-𝑡 + 𝐹 𝑣 ) The modality attention module addresses the alignment issue between modalities. In (5), the cross-correlation matrix encodes the relationships between video and text features learned by the model through the trainable parameter 𝑊 𝑡,𝑣. Applying softmax to the matrix 𝑍 𝑡,𝑣 enhances the discriminative power of the features. The model assigns higher weights to visual features when they exhibit strong correlations with textual features, and vice versa.
A. MASKED LANGUAGE MODELING Similarly to BERT, we also randomly replace 15% of the tokens in the sentence with the special token [MASK] and then generate the masked tokens given the known tokens and video input. The Masked Language Modeling (MLM) loss function is defined as the cross-entropy loss between the predicted probability distribution over the vocabulary and the true distribution for each masked token as seen in (  31 ): ℒ 𝑀𝐿𝑀 = -∑ ∑ 𝑦 𝑖𝑗 log (𝑝 𝑖𝑗 ) 𝑆 𝑣𝑜𝑐 𝑗=1 𝑆 𝑚𝑎𝑠𝑘 𝑖=1 (31) Here, 𝑆 𝑚𝑎𝑠𝑘 is the number of masked tokens, 𝑆 𝑣𝑜𝑐 is the size of the vocabulary, 𝑦 𝑖𝑗 is the true probability of the j-th token for the i-th masked position and 𝑝 𝑖𝑗 is the predicted probability of the j-th token for the i-th masked position.
A. DATASET The MSRVTT (Microsoft Research Video to Text) dataset is widely used for benchmarking video captioning methods. It spans over 20 domains, including sports, news, education, and how-to videos. The dataset comprises 10,000 video clips, with an average length of 20 seconds, and 200,000 natural language descriptions, which have been collected from crowd-workers, ensuring diverse and human-like language expressions. The videos have been crawled from YouTube, contributed by internet users, and thus correspond to real-life situations. The MSRVTT dataset raises several challenges, such as recognizing objects, actions, and scenes, as well as understanding the context and generating semantically meaningful captions. Additionally, it is worth noting that the MSRVTT dataset comprises videos with both visual and audio modalities, which adds an extra level of complexity to the task of generating captions.
Nevertheless, around 20% of the videos in the dataset have no audio channel, while others have non-English audio, making the task even more challenging with sparse modalities. In order to study the effect of each modality on the performance of the model, we have manually annotated two distinct subsets. The first subset, labeled as "vision and text" (534 samples), encompasses videos where both the visual VOLUME XX, 2017 features and transcript information contribute to the video captioning task. For example in Figure  8. (a), the transcript helps identify specific ingredients such as oil type, difficult to infer solely from visual modality. The second subset, called "vision only" (663 samples), comprises videos where the task can be accomplished solely through visual cues. Some of these videos include silent or non-English speaking videos, where the transcript modality cannot be provided.
C. ABLATION STUDY We have conducted an ablation study in order to determine the significance of each component within the framework. The study compares several combinations to evaluate their relative performances. The following methods are considered for evaluation:  text only, which used only text as input, trained with the transformer encoder and decoder.  video only, which used only video as input, also trained with the transformer encoder and decoder.  video-text, which used both video and text as input but did not employ the modality-attention module.  MAM, which adds the modality attention module (MAM) to the former.  MAM+init, which uses the initialization of the encoder and decoder weights from the model in  [6].  MAM+Cont, which is trained with all objectives from scratch on MSRVTT, including the contrastive loss with the caption as input.
 MAM+Cont+init, which initializes the encoder and decoder weights using those of  [6]  and includes both modality-attention and contrastive loss objective techniques. We denote this complete architecture by CapVT. The following evaluation metrics are retained to evaluate the performance of the models: BLEU (1-4)  [62] , METEOR  [63]  and ROUGE  [64]. BLEU evaluates the quality of generated text based on the n-gram (1 to 4) overlap with the reference text. ROUGE measures the overlap of n-grams and word sequences between the generated and reference captions. METEOR considers both n-gram overlap and semantic similarity between the generated and reference text. All scores range between 0 and 1, with higher values indicating better performances. The results obtained, summarized in Pre-training the model on external large datasets can be beneficial, but this process is often computationally expensive and requires significant hardware resources.
To address this issue, we have used transfer-learning techniques to initialize the weights of our transformer encoder and decoder, which allowed us to leverage the knowledge learned from a larger dataset while reducing the computational load. This is observed with an improvement in performance of 1. 2% and 2. 17% for BLEU4 when comparing MAM to MAM+Init and MAM+Cont to CapVT, respectively. The study also highlights the importance of the contrastive loss objective in improving the caption quality, as removing it leads to a significant drop in performance (2. 19% in terms of BLEU4). Additionally, incorporating textual information is crucial for generating accurate captions, as evidenced by the lower score achieved when only the video modality is considered. Finally, the results indicate that the visual modality is more informative than the textual one as we achieve better results when feeding only the visual modality as compared to feeding only the textual modality.
As part of our study, we have also investigated how the quality of the generated captions is affected by different input modalities. For this purpose, we selected the first three baselines: text-only model, video-only model and video-text model. We have deliberately excluded the other models that employ additional strategies such as modality attention or contrastive loss. Our primary objective here is to solely examine the impact of the input modality on the model's performance. We have assessed the performance of each baseline on videos that require only the visual modality to generate captions and those that require both visual and textual modalities. However, it was not feasible to label videos that require only textual modality in the MSRVTT dataset as certain information, such as key objects/persons can only be perceived through visual cues and not through text.
We have randomly selected 1179 test samples and manually labeled them as either "vision-only" (663 samples) or "vision and text" (534 samples) to evaluate the performance of each baseline model on these different types of videos. TABLE  II  shows the performance comparison in terms of BLEU4 of the three models trained with different input modalities. The evaluation has been performed on the two subsets of samples that require either only visual modality or both visual and textual modalities to generate captions. The following conclusions can be drawn: (1) The effectiveness of video captioning models is heavily influenced by the input modalities. The different performances obtained on the two subsets underscore the significance of the dataset's modality composition. (  2 ) The text-only model struggles to generate captions from visual content alone with a low score of 0.
D. COMPARISON WITH STATE OF THE ART To facilitate a meaningful comparison between our work and previous state-of-the-art models in the context of video captioning, we have examined key statistics pertaining to these models. Specifically, we have compiled comprehensive data encompassing model size (number of parameters), the scale of pre-training samples, hardware infrastructure employed (GPU/TPU), and the training duration. The detailed findings of this analysis are presented in  For a fair comparison, we have retained models that are comparable to ours, including OA-BTG  [21] , VideoAsMT  [46] , SwinBert  [7] , and OpenBook  [3]. Additionally, we have retained the UniVL model  [6] , as we leveraged its weights to initialize the encoder and decoder parameters in our own approach. This approach aims to deliver a comprehensive and equitable evaluation of our method in relation to its peers, thus establishing a clear understanding of its performance within a defined resource context.