# Second-Order Text Matching Algorithm for Agricultural Text
## 5 Experiments
### Performance Comparison Analysis
#### 5.3.1 Retrieval Accuracy Comparison Analysis  
The purpose of this experiment is to compare the algorithms and models retrieval accuracies of using SOTM algorithm, using only BM25 algorithm, and using only BERT model. The dataset used in this experiment is the text set, and the three algorithms and models are run on this dataset separately to compare their retrieval accuracies. We use P@1 as the metric of retrieval accuracy. For each query in the test set, SOTM, BM25, and BERT retrieve only one answer. If the retrieved answer matches the given answer, it is considered as a successful match; otherwise, it is considered as a failed match. We calculate the retrieval accuracy of each algorithm or model by dividing the number of successful matches by the size of the test set. Figure 7 compares P@1 of different algorithms or models. Figure 6: Variation of algorithm’s average time with recall number \(s\). Figure 7: Comparison of P@1 of different algorithms or models. It can be observed that P@1 is the lowest when only BM25 algorithm is used. The P@1 value of BM25 is only 74.61%. This is because BM25 algorithm matches through some word-level perspectives such as word frequency, inverse document frequency and document length. It ignores specific contextual information and word order relationships, which makes BM25 algorithm unable to comprehensively consider the semantic relationships between words. At the same time, BM25 algorithm focuses more on dealing with common words, while its performance may be poor when dealing with long-tailed words and domain-specific proper nouns. When only BERT model is used, the P@1 is moderate, reaching 79.53%. BERT has been trained on a large corpus, incorporating extensive, diverse, and general linguistic information. Through fine-tuning, BERT model has learned specific terms and semantic information in the agricultural domain. When matching text, BERT model comprehensively considers semantic information from the context, leading to better choices [42]. It can be seen that SOTM even outperforms BERT model in P@1 when \(s\) equals to 10. The P@1 value of SOTM algorithm is 88.34%, the highest among the three. The reasons we hypothesize for SOTM algorithm performing even better than using only BERT model may lie in two aspects. One is the task difference and the other is the granularity difference. * **Task difference**. During fine-tuning, BERT is trained to judge whether two given sentences are similar (returning 1) or dissimilar (returning 0). However, in this experiment, the task is to provide a query and find another text from the dataset. For a given query, BERT may consider that there are several texts similar to the query in the candidate text set. Although we ultimately obtain BERT's scores for each text through the softmax function, and select text with the highest score, this is not consistent with the task of fine-tuning. BERT has not been fully trained for this fine-tuning task. We have found through experimentation that BERT often gives similar scores to multiple texts. On the contrary, we firstly use BM25 algorithm to narrow down the candidate set of BERT. To some extent, this can reduce the interference of texts that have large word differences but are similar to the query in some expressions, and reduce the situation where BERT gives similar scores to multiple texts. It is therefore helpful for BERT to select the most appropriate answer. * **Granularity difference**. BERT model used in this paper is based on characters, while SOTM algorithm is based on both characters and words. This multi-granularity information fusion enables SOTM algorithm to learn more information. This makes SOTM algorithm easier to have a higher retrieval accuracy. Furthermore, to further explore and compare the retrieval accuracies of various algorithms or models, the P@3 and P@5 values of SOTM, BM25, and BERT are also calculated. Additionally, a comparison is made among P@1, P@3, and P@5 values. Figure 8 compares different algorithms or models in terms of P@1, P@3, and P@5.