# Multi-Task Supervised Alignment Pre-Training for Few-Shot Multimodal Sentiment Analysis
###### Abstract  
Few-shot multimodal sentiment analysis (FMSA) has garnered substantial attention due to the proliferation of multimedia applications, especially given the frequent difficulty in obtaining large quantities of training samples. Previous works have directly incorporated vision modality into the pre-trained language model (PLM) and then leveraged prompt learning, showing effectiveness in few-shot scenarios. However, these methods encounter challenges in aligning the high-level semantics of different modalities due to their inherent heterogeneity, which impacts the performance of sentiment analysis. In this paper, we propose a novel framework called Multi-task Supervised Alignment Pre-training (MSAP) to enhance modality alignment and consequently improve the performance of multimodal sentiment analysis. Our approach uses a multi-task training method--incorporating image classification, image style recognition, and image captioning--to extract modal-shared information and stronger semantics to improve visual representation. We employ task-specific prompts to unify these diverse objectives into a single Masked Language Model (MLM), which serves as the foundation for our Multi-task Supervised Alignment Pre-training (MSAP) framework to enhance the alignment of visual and textual modalities. Extensive experiments on three datasets demonstrate that our method achieves a new state-of-the-art for the FMSA task. Keywords:few-shot multimodal sentiment analysis; multi-task learning; supervised pre-training +
Footnote †: journal:  
scenarios and directly incorporates vision modality into the PLM. However, the challenges brought by heterogeneity cannot be ignored. This fundamental difference between visual and language modalities impedes PLM from directly parsing and comprehending visual information. Faced with this situation, a research direction of few-shot multimodal sentiment analysis (FMSA) is how to effectively align visual and language modalities to obtain consistent visual-language representations. Inspired by recent studies on the Multimodal Large Language Models (MLLM) [6; 7; 8], we recognize that alignment pre-training is a potent way to tackle this issue. Implementing this strategy would aid in enhancing the PLM's flexible adaptability to the visual modality. UP-MPF [9], the previous state-of-the-art method for FMSA, leverages alignment pre-training to address the challenge of modality alignment in FMSA. To achieve alignment pre-training, UP-MPF freezes the PLM and feeds a substantial quantity of images along with text prompts into the PLM to predict the direction of image rotation in textual format, thereby enabling the model to learn the relationship between low-level visual features and language features, effectively enhancing the alignment between these modalities. However, UP-MPF exposes two main issues in practice: (1) The task of predicting the direction of image rotation, a well-known self-supervised pretext task in computer vision, primarily models low-level visual features and lacks the complex semantics that include human emotions [10]. These supervision signals, which are absent in this task, are crucial for aligning the learned visual features with strongly semantic language features. (2) UP-MPF only uses a single task during the pre-training stage. Focusing solely on one pre-training task can limit a model's visual understanding by exposing it to only a narrow range of contexts and patterns, making it less adept at comprehensively analyzing diverse visual information. This could hinder the model's effective learning of shared information between visual and language modalities, thereby presenting an obstacle to effective modality alignment. In summary, the enhancement of few-shot multimodal sentiment analysis performance hinges on the optimization of the alignment pre-training stage. This optimization aims to bolster the model's visual grounding capabilities and foster a more comprehensive multimodal understanding, thereby empowering downstream sentiment analysis tasks.