# Multi-Task Supervised Alignment Pre-Training for Few-Shot Multimodal Sentiment Analysis
###### Abstract
By doing so, the pre-trained model can acquire robust sentiment analysis proficiency through a limited number of training samples in data-scarce scenarios. Current alignment pre-training approaches exhibit deficiencies in rich semantic information and modal sharing, which are crucial elements for few-shot multimodal sentiment analysis tasks. Consequently, our research focuses on addressing these limitations to improve model performance in few-shot multimodal sentiment analysis contexts. By refining the alignment pre-training process, we aim to bridge the gap between visual and textual modalities, enabling more effective knowledge transfer and generalization in low-resource settings. This approach has the potential to significantly advance the field of multimodal sentiment analysis, particularly in scenarios where labeled data are scarce. To address the above issues, we propose Multi-task Supervised Alignment Pre-training (MSAP) for the FMSA task. For issues with missing rich semantic information, since more and more large-scale labeled datasets designed for tasks such as image classification or image captioning have become easily accessible [11], our method combines these datasets for supervised pre-training [12], which provide explicit semantic signals like image categories, styles, or descriptions. This makes it possible to align visual and language modalities by incorporating stronger semantic information into the visual representation. Regarding the issue of missing modal shared information, we recognize that relying on a single pre-training task during the alignment stage may lead to potential overfitting and hinder the model's ability to learn generalized multimodal representations. This limitation could result in the model becoming overly biased, thereby compromising its effectiveness in capturing shared information across different modalities. To address this issue, we employ multi-task learning [13], a paradigm that enables the model to synthesize knowledge acquired from diverse tasks, thereby mitigating the risk of overfitting to any single task [14]. Moreover, multi-task learning facilitates the extraction of shared information across modalities, enhancing the model's ability to leverage cross-modal relationships. This approach not only promotes a more balanced and generalizable learning process but also fosters the development of robust representations that capture the intricate interplay between different modalities. Specifically, we use a vision encoder to extract visual features and use a simple single-layer MLP as the alignment layer to align visual features with language features. Then, we feed the aligned visual tokens and language tokens into the pre-trained language model. During the Multi-task Supervised Alignment Pre-training (MSAP) stage, we first select three large image datasets, each annotated for different tasks. Combining these datasets, we created a massive comprehensive dataset containing approximately 900,000 images. As shown in Figure 1, our strategy is to fix the pre-trained language model (PLM) and update the weights of the visual encoder and alignment layer. Through the transformation of diverse visual tasks--encompassing image classification, image style recognition, and image captioning--into a unified Masked Language Modeling (MLM) paradigm, we achieve a synergistic integration of supervised pre-training and multi-task learning methodologies. Doing so not only improves the overall learning effect through the complementarity of each task but also allows the visual features to obtain richer semantic information. In the prompt fine-tuning stage, we fix the visual encoder and fine-tune the PLM with prompts to complete the FMSA task. Specifically, we make full use of the visual features with rich semantic information and modality sharing information obtained in the first stage through pre-training techniques. We use these features for lightweight and efficient parameter fine-tuning, successfully increasing the training efficiency while ensuring the predictive effect.