# A Framework for Agricultural Intelligent Analysis Based on a Visual Language Large Model
## 2 Datasets and Methods
### Methodology of Research
#### 2.2.3 LoRA Parameter Fine-Tuning  
LLMs usually contain hundreds of millions or even billions of parameters. Training an LLM from scratch requires a large amount of computing resources, which makes the calculation cost extremely expensive. Meanwhile, the huge number of parameters may lead to overfitting the training results. In order to solve these problems, AgriVLM uses LoRA technology [27] to fine-tune the model parameters to learn the characteristics of agricultural data. LoRA fine-tuning allows the model to capture the characteristics of a specific task while maintaining pre-trained knowledge, allowing it to quickly adapt to new tasks. LoRA effectively avoids risk of overfitting while lowering the computational cost, providing an efficient and reliable solution for processing agricultural data.  
LoRA effectively adjusts the parameters of the model by adding two trainable matrices with Low-Rank decomposition into the model parameter space. Its overall workflow is shown in Figure 2. In the fine-tuning process, LoRA is mainly used in the transformer layers of the language model. Specifically, a trainable Low-Rank separable matrix is added to each layer of the transformer. Assume that the parameters of the pre-trained model need to be updated during fine-tuning, which is expressed as \(W_{pm}+W_{\text{lora}}\), where \(W_{pm}\in\mathbb{R}^{d\times k}\) are the parameters of the pre-trained model and \(W_{\text{lora}}\) are the parameters that should be updated. The parameters are updated as shown in Equation (1):  
\[W_{pm}+W_{Iora}=W_{pm}+B_{Iora}A_{Iora} \tag{1}\]LoRA freezes the \(W_{pm}\) parameters during the training process and only updates the parameters in \(A_{lora}\) and \(B_{lora}\), where \(B_{lora}\in\mathbb{R}^{d\times r}\), \(A_{lora}\in\mathbb{R}^{r\times k}\), r << min (d, k). Matrix \(A_{lora}\) is initialized using a random Gaussian distribution in order to keep the expressive capability of the model. The random initialization of the Gaussian distribution can help the model to learn the effective feature representation quickly in the early stage of training, which can improve the performance of the model. Matrix \(B_{lora}\) is initialized using a zero matrix in order to reduce the impact on the original model parameters while ensuring the stability of the model during fine-tuning. Finally, r is a hyperparameter which is the rank of the matrix \(A_{lora}\) and \(B_{lora}\), and r << min (d, k) makes the number of parameters in the fine-tuning process significantly reduced, saving computational resources.  
When LoRA is fine-tuning the transformer layer, it is mainly applied in the weight matrices used to learn the \(Q\), \(K\), and \(V\) vector representations in the multi-head self-attention module. To be specific, for the training input data \(x\), \(Q\), \(K\), and \(V\) are calculated as shown in Equation (2):  
\[\begin{split} Q&=x\cdot\left(W_{pm}^{q}+W_{lora}^{q} \right)=x\cdot W_{pm}^{q}+x\cdot W_{lora}^{q}\\ K&=x\cdot\left(W_{pm}^{k}+W_{lora}^{k}\right)=x\cdot W _{pm}^{k}+x\cdot W_{lora}^{k}\\ V&=x\cdot\left(W_{pm}^{v}+W_{lora}^{v}\right)=x\cdot W _{pm}^{v}+x\cdot W_{lora}^{v}\end{split} \tag{2}\]  
where \(W_{pm}^{*}\) represents the pre-trained weight matrix and \(W_{lora}^{*}\) is the trainable weight matrix. After calculating \(Q\), \(K\), and \(V\), the softmax operation is used, and the computational inference for matrices \(Q\) and \(K\) with LoRA layers can be expressed as in the following Equation:  
\[\begin{split} Softmax(Q,K)&=Softmax\bigg{(}x\cdot W _{pm}^{q}\cdot\left(W_{pm}^{k}\right)^{T}\cdot x^{T}+x\cdot W_{pm}^{q}\cdot \left(W_{lora}^{k}\right)^{T}\cdot x^{T}\\ +x\cdot W_{lora}^{q}\cdot\left(W_{pm}^{k}\right)^{T}\cdot x^{T}+x \cdot W_{lora}^{q}\cdot\left(W_{lora}^{k}\right)^{T}\cdot x^{T}\bigg{)}\end{split} \tag{3}\]  
Finally, the calculation of self-attention is shown in Equation (4):  
\[Attention=Softmax(Q,K)V=Softmax(Q,K)\cdot x\cdot W_{pm}^{v}+Softmax(Q,K)\cdot x \cdot W_{lora}^{v} \tag{4}\]