# Multi-Task Supervised Alignment Pre-Training for Few-Shot Multimodal Sentiment Analysis
## 3 Methods
### Model Architecture  
In this section, we will introduce our model in detail. Formally, given a template (e.g., Image: \(H_{V}\). Text: \(T\). It is [MASK].) that contains a text-image set and a designed text prompt with [MASK] token, we align the semantic information in the image to text through the alignment layer, then predict what may appear in [MASK], with an aim to help the model learn the multimodal sentiment information in both text and image forms.  
As illustrated in Figure 1, the overall model architecture comprises a vision encoder \(f_{V}\), a pre-trained foundation language model \(f_{L}\), and a single-layer MLP as the alignment layer \(f_{A}\). We employ the vision encoder and alignment layer to extract semantic information from the image and transform it into corresponding image tokens to align with the text \(T\). For the image modality, we first employ NF-ResNet [29] as the vision encoder \(f_{V}\) to extract the vision representation \(X_{V}\) corresponding to the input image \(I\). The visual representation \(X_{V}\) extracted at this stage forms a set of critical features that capture the essence of the visual elements presented in the image. Subsequently, we utilize the alignment layer \(f_{A}\), a simple single-layer MLP, as a bridge that connects two different spaces: the vision embedding space and the language embedding space. The alignment layer \(f_{A}\) plays an instrumental role in connecting visual representation with linguistic representation. To achieve this, we employ a lightweight linear network as the alignment layer to obtain the vision-language representation of \(X_{V}\), denoted as \(H_{V}\), which can also be viewed as a sequence of image tokens. Formally, to obtain the image tokens \(H_{V}\) of the input image \(I\),  
\[X_{V}=f_{V}(I) \tag{1}\]  
\[H_{V}=f_{A}(X_{V}) \tag{2}\]  
where \(H_{V}\in\mathbb{R}^{K\times d_{L}}\); \(d_{L}\) is the dimension of the language embedding space in the pre-trained language model; and \(K\) serves as a hyper-parameter, representing the length of the image token sequence and quantifying the extent of visual information encompassed within these tokens.  
Furthermore, as a key component integrating visual and language understanding, the alignment layer \(f_{A}\) not only fulfills the crucial purpose of aligning modalities but also plays a pivotal role in effectively filtering out fine-grained visual information. It essentially performs a downsampling process, dimensionally reducing the larger image features produced within the vision encoder \(f_{V}\) and leading to a substantial reduction in computational complexity. Once we have obtained \(H_{V}\), we combine the text \(T\), the designed text prompt with [_MASK_] token (e.g., It is [_MASK_]), and the image tokens \(H_{V}\) to create a multimodal prompt template \(T\). An example of \(T\) is provided below:  
\[Image:H_{V}.Text:T.\textit{ It is}[MASK] \tag{3}\]  
We then feed the embedding of \(T\) into the pre-trained language model M in order to predict the token with the highest probability at the [_MASK_] position. Additionally, we define a verbalizer, denoted as \(v\), to convert sentiments into indices that point towards specific vocabulary. The process can be described as \(Y\longrightarrow V\), where \(V\) is the vocabulary of M. For each label \(y\in Y\), \(v(y)\) maps the label to its corresponding token in \(V\). Formally, obtaining the predicted label \(\hat{y}\) given the image-text pair \(x\) can be illustrated as  
\[\hat{y}=\arg\max\ p(\text{[MASK]}=v(\hat{y})\mid T(x)) \tag{4}\]