# Second-Order Text Matching Algorithm for Agricultural Text
## 4 Proposed Method
### Rearrangement Phase Based on BERT Model  
BERT has several significant advantages over other text matching algorithms. Firstly, BERT uses a bidirectional Transformer model, which allows it to better understand contextual nuances. Other models such as Word2 Vec and GloVe only consider unidirectional contextual information. Secondly, BERT has several layers of representations, each capturing different levels of semantic information. This allows BERT to understand text comprehensively, improving its performance in text matching tasks. Thirdly, BERT is pre-trained on large amounts of text to learn universal language representations. This imparts superior generalisation capabilities, enabling the model to perform well even withlimited annotated data. Fourthly, BERT can be fine-tuned for specific tasks, adapting to different text matching scenarios. This flexibility increases the applicability of the model to a wide range of contexts. Therefore, BERT model is selected in the rearrangement phase.  
BERT model is trained in two steps. A generic model is firstly trained using a large-scale unlabeled document-level corpus and then supervised training of that model on a task-specific corpus is performed to tune the model's parameters. The pre-training phase requires training on large-scale unlabeled documents and this paper directly uses BERT-Chinese model released by Google as the pre-training model. The model is pre-trained on millions of Chinese Wikipedia corpus, which can be well migrated to various Chinese text tasks.  
After pre-training BERT model, the parameters need to be fine-tuned in the downstream task, otherwise the results will not be satisfying. Firstly, the two sentences that need to be text-matched are combined into a single sentence with special tags. Specifically, a \([CLS]\) tag is inserted at the beginning of the first sentence, and a \([SEP]\) tag is inserted at the end of the first sentence and the second sentence. It is necessary to conduct this for all the sentences. To make sure that the length of all word lists is the same, it is necessary to fill the sentences that do not reach the maximum length with \([PAD]\) tags and mark which positions are filled or not. Finally all markers are mapped to a single independent one. This is the text embedding \(input\_ids\). Then map the words in the first sentence of 0 and those in the second sentence of 1. This is the segment embedding \(segment\_ids\). The \(input\_ids\) and \(sentent\_ids\) are the inputs to BERT fine-tuning process. BERT fine-tuning process is shown in Figure 3, where \"\(\cdot\)\"\(\cdot\)\"\(\cdot\)\"\(\cdot\)\" means \"Wheat, what is it\" and \"\(\cdot\)\"\(\cdot\)\"\(\cdot\)\" means \"What is wheat\". The pseudocode for this phase of the algorithm is given on Algorithm 3.  
```
0: One query \(q_{0}\), candidate set \(S\) from recall phase.
0:
1:Pair user input query \(q_{0}\) with each text \(q_{i}\) in the candidate set \(S\) and convert them into text embeddings \(ie\);
2:Generate a type embedding for each text embedding \(te\);
3:Input \(ie\) and \(te\) into BERT model. Return the query with the highest score \(q^{*}\in S\).
```  
**Algorithm 3** Rearrangement Phase Based on BERT Model