# Second-Order Text Matching Algorithm for Agricultural Text
## 4 Proposed Method
### Recall Phase Based on BM25 Algorithm  
Compared to other traditional text matching algorithms, BM25 algorithm has the following advantages. Firstly, it can dynamically adjust its parameters. The two main parameters (\(k_{1}\) and \(b\)) can be adjusted to adapt to different text collections and query patterns, which makes the algorithm more flexible. Secondly, BM25 takes into account the effect of document length on the weights, which are adjusted by the parameter \(b\). This makes the algorithm more robust to different document lengths than other traditional text matching model. Thirdly, BM25 avoids the overfitting problem that may occur in other traditional text matching algorithm like TF-IDF when dealing with long documents, by suppressing the frequent words in the document. Fourthly, BM25 performs better in handling sparse queries (queries with a small number of keywords) and provides more accurate search results than some other algorithms. Therefore, BM25 algorithm is selected for screening and matching in the recall phase.  
The data required for BM25 algorithm can be pre-processed. In the first step, the frequency of word occurrences in a sentence can be calculated. \(NUM[j][i]\) denotes the number of occurrences of word \(i\) in the \(j\)-th sentence. In the second step, the inverse document frequency \(IDF\) of each word can be calculated and stored in a dictionary. \(IDF[i]\) denotes the \(IDF\) value for word \(i\). In the third step, the average length \(\mathit{avglen}\) of all sentences in the text set can be calculated.  
When the user inputs a query \(q_{0}\), the algorithm utilizes Jieba library to segment words, where each word is denoted by \(qs_{i}\). For each \(qs_{i}\), \(TF(qs_{i})\) and \(IDF(qs_{i})\) are calculated. Then, the similarity scores between \(q_{0}\) and each text \(q_{i}\) are computed and stored in the dictionary _scores_. The dictionary stores two pieces of information, an index and the score of the query corresponding to that index. Finally, the dictionary is sorted in descending order based on the query scores and then returned. In this way, the set of problems corresponding to the first few indices of the _scores_ dictionary is the required candidate set and the recall phase ends. The pseudocode for this phase of the algorithm is given on Algorithm 2.  
```
0: One query \(q_{0}\), text set \(T=\{q_{1},q_{2},\ldots,q_{n}\}\).
0: Candidate query subset \(S\subset T\).
```  
**Procedure:**  
```
1: Calculate the frequency of word occurrences \(TF\) in a sentence. \(NUM[j][i]\) is the number of occurrences of the word \(i\) in the \(j\)-th sentence;
2: Calculate the inverse document frequency \(IDF\) of each word, where \(IDF[i]\) denotes the \(IDF\) value for word \(i\);
3: Calculate the average length \(\mathit{avglen}\) of all sentences in the text set;
4: Utilize the Jieba library to segment the input query \(q_{0}\), where \(qs_{i}\) represents the \(i\)-th word after segmentation;
5: For each word \(qs_{i}\), calculate \(TF(qs_{i})\) and \(IDF(qs_{i})\);
6: Calculate the similarity scores between \(q_{0}\) and each text \(q_{i}\) and store them in a dictionary _scores_, where \(\mathit{scores}[i]\) is the score of the query with index \(i\);
7: Sort scores by values in descending order and return scores. From the dictionary, select the first few as the candidate set \(S\subset T\).
```  
**Algorithm 2** Recall Phase Based on BM25 Algorithm