## 3 BM-RAGAM Retrieval Scheme in the Vertical Domain of LLMs
### The Fundamental LLM  
The Qwen2 series, which is an open-sourced LLM grounded in the transformer architecture, includes a total of five sizes of pre-training and command fine-tuning models, including Qwen2-0.5B, Qwen2-1.5B, Qwen2-7B, Qwen2-57B-A14B, and Qwen2-72B; the series' information is shown in Table 2. The QWen2 team trained all of their pre-trained models on 32K tokens and found that they also performed well on the PPL of 128K tokens and were enhanced for 27 languages other than Chinese and English. In the evaluation of pre-trained language models, the Qwen2-72B model in the Qwen2 series of models significantly outperformed the current leading models in terms of its understanding, knowledge, code, mathematics, and NLP multi-language capabilities [34], as shown in Table 3.  
Overall, the Qwen2-72B model excelled in a wide range of tasks and demonstrated its powerful capabilities, significantly outperforming comparable models and closely following a range of the strongest closed-source models. In addition, the Qwen2-72B performed particularly well in complex problem-solving tasks, such as scoring 64.6 in the HumanEval test. It scored 91.0 in the C-Eval test, surpassing many similar models. Its competitiveness in multi-language understanding tasks was also evident, with scores of 90.1 on the CMMLU test and 76.6 on the multi-exam test, making Qwen2-72B a strong choice for multiple applications in NLP [35; 36]. Since computers cannot directly understand text content, embeddings were used to represent the text numerically. In order to make the model more suitable for processing Chinese text, this study adopted the text2 vec-large-Chinese text vectorization model, which is part of the text2 vec series, to provide a word representation of Chinese text with vectors. In the field of text vectorization, the text2 vec-large-Chinese model can efficiently support \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Model** & **Parameter Quantity** & \multicolumn{2}{c}{**Number of**} & & & \\ \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & **Non-Embedding** & **GQA** & **Tie Embedding** & **Context Length** \\  & & **Parameters** & & & \\ \hline Qwen2-0.5B & 0.49B & 0.35B & True & True & 32K \\ Qwen2-1.5B & 1.54B & 1.31B & True & True & 32K \\ Qwen2-7B & 7.07B & 5.98B & True & False & 128K \\ Qwen2-57B-A14B & 57.41B & 56.32B & True & False & 64K \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The information of the Qwen2 series. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Task** & **Qwen2-72B** & **Llama3-70B** & **Mixtal-8x22B** & **Qwen1.5-110B** \\ \hline MMLU & 84.2 & 79.5 & 77.8 & 80.4 \\ MMLU-Pro & 55.6 & 52.8 & 49.5 & 49.4 \\ GPQA & 37.9 & 36.3 & 34.3 & 35.9 \\ TheoremQA & 43.1 & 32.3 & 35.9 & 34.9 \\ BBH & 82.4 & 81.0 & 78.9 & 74.8 \\ HumanEval & 64.6 & 48.2 & 46.3 & 54.3 \\ MBPP & 76.9 & 70.4 & 71.7 & 70.9 \\ MultiPL-E & 59.6 & 46.3 & 46.7 & 52.7 \\ GSM8K & 89.5 & 83.0 & 83.7 & 85.4 \\ MATH & 51.1 & 42.5 & 41.7 & 49.6 \\ C-Eval & 91.0 & 65.2 & 54.6 & 89.1 \\ CMMLU & 90.1 & 67.2 & 53.4 & 88.3 \\ Multi-Exam & 76.6 & 63.5 & 63.5 & 75.6 \\ Multi-Understanding & 80.7 & 77.9 & 77.7 & 78.2 \\ Multi-Mathematics & 76.0 & 67.1 & 62.9 & 64.4 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Qwen2-72B performance compared with other current open-sourced LLMs. text-mining tasks such as similarity searches and clustering. The construction of this vector knowledge base includes the following: (1) The digitization of documents: Documents are converted into a \"digital\" form based on the file type (such as: \".txt\", \".pdf\", and \".docx\"). (2) Block processing: The document content is usually very long, so the document is divided into a number of small blocks. Each block contains a specific number of characters, with each block or group in this research consisting of 1000 characters. To ensure the full capture of the context or detailed information from longer texts, an overlap of 200 characters is applied among groups.