# A Unified Visual and Linguistic Semantics Method for Enhanced Image Captioning
## 4 Methodology
### SegmentCLIP
Our observations suggest that hard assignment is more effective for feature grouping compared to soft assignment, as evidenced by the results presented in Table 1.  
Analogous to a single cycle of the previously established Slot Attention framework, the Segmenting Block executes a similar function. However, where Slot Attention acquires instance-level feature clustering autonomously, our Segmenting Block arranges akin semantic segments employing minimal text-based guidance. For instance, as depicted in Figure 2's second row, the algorithm has clustered the pair of women effectively. To enhance SegmentCLIP's capacity for hierarchical clustering, we employ carefully designed contrastive losses between image-text pairs. We harness these losses to refine visual representations via textual guidance, adopting a methodology inspired by prior research to train a dual-encoder framework. This framework encompasses an image encoder and a text encoder, leveraging a contrastive loss strategy. For the Image-Text Contrastive Loss, we utilize the final image embedding derived from the SegmentCLIP and the concluding text embedding from the text encoder, representing the last output token. Our objective is to converge the representations of correlated image-text pairs and concurrently diverge the representations of non-correlated pairs. Formally, we define a batch of B image-text pairs \(\{(x^{I}_{i},x^{T}_{i})\}_{i=1}^{B}\), where \(x^{I}_{i}\) and \(x^{T}_{i}\) represent the image and textual components, respectively. Encoder into embeddings \(z^{I}_{i}\) and \(z^{T}_{i}\) through their respective encoders, we then calculate the dot product to assess their similarity. The comprehensive loss for image-text contrastive learning is articulated as a sum of image-to-text and text-to-image contrastive losses:  
\[\mathcal{L}_{I\leftrightarrow T}=\mathcal{L}_{I\to T}+\mathcal{L}_{T \to I}, \tag{13}\]  
Image-to-Text Contrastive Loss:  
\[\{\mathbf{g}_{i}^{l}\},\{\mathbf{s}_{i}^{l}\}=\text{Transformer}((\{\mathbf{g }_{i}^{l}\};\{\mathbf{s}_{i}^{l}\})), \tag{6}\]0  
Text-to-Image Contrastive Loss:  
\[\{\mathbf{g}_{i}^{l}\},\{\mathbf{s}_{i}^{l}\}=\text{Transformer}((\{\mathbf{g }_{i}^{l}\};\{\mathbf{s}_{i}^{l}\})), \tag{6}\]1 \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Dataset** & **Lang.** & **Images** & **Captions** & **Caption Vocabulary** \\ \hline AIC-ICC & zh & 300 K & 1500 K & 7654 \\ Flickr8k-CN & zh & 8000 & 40,000 & 1447 \\ COCO-CN & zh & 2041 & 101,705 & 2069 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Statistics of the datasets used in our experiments. Here, \(\tau\) is a tunable temperature parameter to modulate the logits. To facilitate enhanced visual categorization, we put forth a Multi-Label Image-Text Contrastive Loss in addition to the foundational image-text loss outlined in Equation (6). This multi-faceted contrastive loss is enriched through the use of 'prompt-engineered' textual labels, which are derived from the initial image descriptions as delineated in [31]. Specifically, we elect \(K\) nouns from the narrative \(x_{i}^{T}\) and synthesize these with pre-determined sentence frameworks, resulting in prompts akin to \"An image depicting a noun\". Such an approach is rooted in the rationale that nouns typically correspond to discernible objects within images. In parallel to the training with original pairs, we introduce contrastive losses for the newly created 'image-prompted text' pairings \(\{(x_{i}^{I},x_{i}^{T_{1}})\}_{i=1}^{B},\{(x_{i}^{I},x_{i}^{T_{2}})\}_{i=1}^{B},\dots,\{(x_{i}^{I},x_{i}^{T_{k}})\}_{i=1}^{B}\), where \(\{x_{i}^{T_{k}}\}_{k=1}^{K}\) are the sentences crafted from the nouns isolated from \(x_{i}^{T}\). Referencing Figure 3, this enriched method diverges from the standard contrastive loss as each image \(x_{i}^{I}\) correlates with \(K\) positive text pairings and \(K(B-1)\) negative associations.