# Feature Normalization Reweighting Regression Network for Sugar Content Measurement of Grapes
## 2 Materials and Methods
### FNRR Network
#### 2.3.3 Transformer Model  
A transformer model is a deep neural network, usually based on a self-attention mechanism, which was initially applied in the field of natural language processing. Inspired by its powerful representation capabilities, it was extended to computer vision tasks [18]. For visual tasks, convolutional neural networks have the advantage of inductive bias, translational equivalence, and locality; however, the model requires more convolutional layers to expand the receptive field. While the self-attention mechanism requires long-range information, it performs well for different visual tasks [19]. In Figure 5, the input grape image is divided into fixed-size patches, and patch embedding is performed using a linear transformation, with patch embedding indicating the image information and position embedding indicating the label information of the image from a sequence of token embeddings, taken as the transformer model's input data [19]. The self-attention mechanism aggregates information by assigning different weights to the input information according to the current query. For a given query vector, k key vectors are matched using inner product calculation. The obtained inner product is then normalized using the softmax function to obtain k weights; the output of the query's attention is the weighted average of the value vectors that correspond to the k key vectors [20]. The equation for this matrix calculation is given by  
\[Attention(Q,K,V)=Softmax\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V \tag{1}\]  
where:  
_Q, K, and V_: The three self-attention mechanism vectors.  
_d\({}_{k}\)_: The dimension of _K_[20].  
The transformer uses multi-head self-attention to define attention heads, i.e., self-attention is applied to the input sequence, which can be split into h sequences, and theoutputs of the h different heads are then concatenated together. The final output is obtained using linear transformation [18]. The formula is defined as follows:  
\[MultiHead(Q,K,V)=Concat(head_{1},\dots,head_{h})W^{O} \tag{2}\]  
\[head_{i}=Attention\Big{(}QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V}\Big{)}, \tag{3}\]  
where:  
_head:_ The attention head obtained from Equation (1) [20].  
\(W_{i}^{O},W_{i}^{K},W_{i}^{V}\): Parameter matrices of the model.  
In this experiment, we use the pre-trained transformer models to extract image features. Since the normalized feature is obtained from the classification model, we propose a multi-label loss function that compares the mean square error and cross-entropy with the loss of the mean square error, as described in Equations (4) and (5):  
\[Loss1=\frac{1}{m}\sum_{i=1}^{m}(y_{i}-\hat{y}_{i})^{2} \tag{4}\]  
\[Loss2=\frac{1}{m}\sum_{i=1}^{m}(y_{i}-\hat{y}_{i})^{2}-\lambda\frac{1}{m}\sum _{i=1}^{m}\sum_{j=1}^{n}p_{ij}\log\hat{p}_{ij}, \tag{5}\]  
where:  
\(y_{i}\): Reference value.  
\(\hat{y}_{i}\): Model estimate.  
\(p_{ij}\): Probability of the jth category reference value.  
\(\hat{p}_{ij}\): Probability distribution of the jth category model estimate.  
\(m\): Number of images in one batch.  
\(n\): Size of the categories.  
In this work, we find that the widely used MSE loss (\(Loss1\)) function can be ineffective in imbalanced regression, so we propose a new balanced loss function from an imbalanced perspective to adapt to the label distribution of the imbalanced training set. The balanced loss function (\(Loss2\)) has two parts: The first part is the standard MSE loss and the second part is a new balanced item. According to the data we constructed, each group of data achieved a balanced effect in quantity after amplification. Based on our proposed FNRR framework, we introduced the balance loss term cross-entropy and hyperparameter \(\lambda=0.05\), which is obtained from multiple experiments.  
Figure 5: Grape feature extraction based on transformer model.