# A Framework for Agricultural Intelligent Analysis Based on a Visual Language Large Model
## 2 Datasets and Methods
### Methodology of Research
#### 2.2.2 Visual Language Large Model  
The visual language model used in AgriVLM mainly consists of three parts: image encoder, pre-trained large language model, and Q-former [26]. The image encoder used by AgriVLM to extract information from agricultural images is ViT [23]. ViT firstly divides the input image into patches of the same size and then maps each patch to a fixed-dimensional feature space through a linear projection layer, thus ensuring that each patch can be effectively understood and processed by the model. At the same time, to ensure that the model can make full use of the spatial relationships of each patch, ViT uses positional encoding to provide the model with the position information of each patch in the original image, which enables the model to understand the structure of the image more accurately. After the above processing, each patch is converted into a feature vector. Finally, these feature vectors are fed into the transformer encoder module which learns the image features by capturing the global dependencies of the image using the self-attention mechanism. The large language model employs ChatGLM [10] to understand agriculture-related knowledge and further analyze agricultural text features. ChatGLM has 6.2 billion parameters and is trained on approximately 1 trillion bilingual corpora, including Chinese and English. With the support of technologies such as supervised fine-tuning, self-feedback, and reinforcement learning, ChatGLM performs excellently in bilingual question-and-answering in Chinese and English, and can understand user intent and generate fluent, natural responses. The most important component of ChatGLM is the GLM pre-training framework based on an autoregressive blank filling task [28]. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Question** & **Answer** & **Availability** \\ \hline How many growth stages is & Learn from all agricultural technical & Not available \\ wheat growth divided into? & experts through the platform! & Not available \\ What are the techniques for & Tomatoes cost five yuan a kilogram at & Not available \\ cultivating tomatoes? & the market. & Not available \\ What are the freeze preventive & Thanks for sharing this with the & Not available \\ measures for wheat? & teachers, it’s great. & Not available \\ What is the best temperature & The best temperature for strawberries & Available \\ for strawberries? & is between 18 and 25 \({}^{\circ}\)C. & Available \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Conversation data filtering. The GLM framework trains the model by autoregressively predicting the blank parts of the text to learn the inner rules and structure of the language. Specifically, for a given input text \(\text{x}=[x_{1},\,x_{2},\,\ldots,\,x_{n}]\), GLM randomly selects multiple consecutive text spans \(\{s_{1},\,s_{2},\,\ldots,\,s_{m}\}\) for masking, where each span \(s_{i}\) contains consecutive tokens of length \(l\) in x and is replaced by a special token [Mask], forming the masked text. GLM predicts the masked tokens \(s_{i}\) through autoregression. When predicting the tokens in the mask, the model can refer to the information of previous spans. Therefore, GLM randomly shuffles the order of each text span to fully capture the interdependencies between the individual spans. AgriVLM uses Q-Former as a bridge connecting the image encoder and the language model [24] to achieve alignment of image and text features. Q-Former is a lightweight transformer with a two-stage pre-training strategy. The first stage is visual language representation learning, which enables Q-Former to learn the visual representation most relevant to the text and achieve semantic fusion of cross-modal representations. Q-Former consists of two modules: an image transformer and a text transformer. The image transformer is used to interact with the frozen image encoder, while the text transformer plays the role of both a text encoder and text decoder.