## 3 Approach
### Food Image Classification  
Contrastive learning [23; 24; 25; 26] has increasingly become a mainstream method in deep learning in recent years, and it is used for learning meaningful representations. The core idea of this learning framework is that semantically related concepts should have similar representations, while non-related concepts should have different representations. Contrastive learning was initially used mainly for self-supervised image representation learning [23; 26] and has gradually expanded to the linguistic fields [24; 25]. Recent studies have further leveraged contrastive training to integrate different communication modalities, such as vision and language, audio and language, etc. These models learn the concepts of other modalities and optimize their proximity in a shared abstract space. CLIP [5] is a multimodal neural network that combines vision and language, and it is trained through a contrastive learning method specifically designed to associate visual concepts with corresponding textual descriptions. The model includes an image encoder and a text encoder, which map image and text representations into the same vector space, training the model to bring related images and descriptions closer together in this space. However, although CLIP performs well in general tasks, it requires further fine tuning and optimization for specific domain applications, such as food classification. This study aims to develop and evaluate a pre-trained model specifically designed for the food domain--NutritionCLIP--through contrastive learning. This section focuses on improving the accuracy of the food classification model by fine tuning the CLIP model. This process involves specific training of the image and text encoder to optimize their ability to process data in the food domain. During the dataset processing stage, as described in Section 4.2.1, we divide the datasets into three parts: training set, validation set, and testing set. Each image is converted into a fixed-size tensor and standardized to improve data consistency. Next, using ViT [21] as the image encoder, these standardized images are transformed into feature vectors \(I\). For text processing, considering the specific requirements of regional languages, Chinese-RoBERTa-wwm-large [53; 54] is chosen as the text encoder to handle Chinese inputs and convert the text into feature vectors \(T\). After data feature processing, contrastive learning is used to learn the associations of image-text pairs in different batches. Since the embedded vectors in CLIP are high-dimensional and focus on the similarity between vectors, cosine similarity [55] is selected as the measure. The primary calculation method is as follows:  
\[similarity(I,\,T)=\frac{I\cdot T}{\parallel I\parallel\parallel T\parallel} \tag{1}\]  
where \(I\) and \(T\) are the image and text embedding vectors, respectively, and \(\parallel I\parallel\) and \(\parallel T\parallel\) represent their norms. For each batch of \(n\) pairs of images and texts \((I_{n},T_{n})\), CLIP calculates a similarity matrix of size \(n\times n\), where each element's similarity \(\left(I_{i},T_{j}\right)\) represents the cosine similarity between the \(i_{th}\) image embedding vector and the \(j_{th}\) text embedding vector. Each row ofthis similarity matrix represents the similarities between an image embedding vector and all text embedding vectors, and each column represents the similarities between a text embedding vector and all image embedding vectors, as shown in Figure 3.  
Next, the model is trained based on contrastive loss, maximizing the similarity between matching image and text pairs and minimizing the similarity between non-matching pairs. The calculation method is as follows:  
\[L_{\textit{contrastive}}=\sum_{i=1}^{n}[\textit{pair}_{i}(1-similarity(I_{i},T_{i}))+(1-\textit{pair}_{i})\textit{max}(0,similarity(I_{i},T_{i})- \alpha)] \tag{2}\]  
where \(n\) is the number of samples and \(\alpha\) is a margin hyperparameter. \(\textit{pair}_{i}\) indicates whether it is a matching pair.