## 3 Materials and Methods
### Model Architecture  
#### 3.2.1 Dual-Tower Encoding with CLIP  
The backbone of our framework is the Contrastive Language-Image Pre-training (CLIP) model [7], which employs a dual-tower encoder architecture to independently process text and image inputs. Each tower maps its modality into a shared semantic embedding space through a modality-specific Transformer, trained jointly via contrastive learning on paired data. As illustrated in Figure 6, the text encoder processes natural language inputs through three stages: (1) Byte Pair Encoding (BPE) tokenization using a 49,408-token vocabulary; (2) 768-dimensional token and positional embeddings; and (3) a  
\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & \multicolumn{2}{c}{**Train (70\%)**} & \multicolumn{2}{c}{**Dev (15\%)**} & \multicolumn{2}{c}{**Test (15\%)**} & \multicolumn{2}{c}{**Total**} \\ \cline{2-9}  & **Text** & **Image** & **Text** & **Image** & **Text** & **Image** & **Text** & **Image** \\ \hline other\_relevant\_information & 1222 & 1269 & 235 & 239 & 244 & 245 & 1222 & 1269 \\ rescue\_volunteering\_or\_donation\_effort & 749 & 827 & 183 & 188 & 168 & 172 & 749 & 827 \\ affected\_individuals & 310 & 329 & 67 & 70 & 67 & 71 & 444 & 470 \\ infrastructure\_and\_utility\_damage & 478 & 539 & 99 & 108 & 122 & 126 & 478 & 539 \\ not\_humanitarian & 2666 & 2957 & 644 & 660 & 642 & 660 & 2666 & 2957 \\ \hline Total & 5425 & 5921 & 1228 & 1265 & 1243 & 1274 & 7896 & 8460 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Distribution of categories of humanitarian tasks in the CrisisMMD benchmark train/dev/test set.  
Figure 5: Balanced sample distribution.  
12-layer Transformer with multi-head self-attention (12 heads). The resulting [CLS] token representation is projected to a 512-dimensional text feature vector (Equation (1)):  
\[v_{t}=Proj_{text}([CLS]_{text}) \tag{1}\]  
In parallel, the image encoder processes \(224\times 224\) pixel RGB images. The image is divided into 32 \(\times\) 32 pixel non-overlapping patches, each patch linearly projected into a 768-dimensional patch embedding. These are then fed into a 12-layer Vision Transformer (ViT-B/32), where the [CLS] token output is projected into a 512-dimensional image feature vector (Equation (2)):  
\[v_{i}=Proj_{image}\Big{(}[CLS]_{image}\Big{)} \tag{2}\]  
During contrastive pre-training, the model learns to pull the embeddings of aligned image-text pairs closer together while pushing apart mismatched pairs. This is achieved using a symmetric InfoNCE-based contrastive loss computed across all image-text pairs in a batch, defined as (Equation (3)):  
\[\mathcal{L}=-\frac{1}{N}\sum_{i=1}^{N}log\frac{exp\big{(}sim\big{(}v_{t}^{i},v _{i}^{i}\big{)}/\tau\big{)}}{\sum_{j=1}^{N}exp\Big{(}sim\big{(}v_{t}^{i},v_{i} ^{j}\big{)}/\tau\Big{)}} \tag{3}\]  
Here, \(N\) is the batch size, \(sim(\cdot,\cdot)\) denotes cosine similarity, and \(\tau\) is the temperature parameter (typically 0.07 in CLIP [7]). This loss enforces cross-modal semantic alignment, crucial for addressing the 40% cross-modal misalignment in CrisisMMD.  
The output embeddings, \(v_{t}\) and \(v_{i}\), are used as modality-specific semantic features for downstream multimodal classification. Their dimension definitions, tokenization pipelines, and projection components are summarized in Table 2.  
Figure 6: Architecture of the CLIP dual-tower encoder.