# Entity Extraction of Key Elements in 110 Police Reports Based on Large Language Models
## 4 Results and Analysis
### Base Model Fine-Tuning
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{3}{c}{**Original**} & \multicolumn{3}{c}{**After Fine-Tuning**} \\ \cline{2-7}  & **P** & **R** & **F1** & **P** & **R** & **F1** \\ \hline ChatGLM & 0.353 & 0.401 & **0.376** & 0.918 \(\pm\) 0.002 & 0.860 \(\pm\) 0.003 & **0.888 \(\pm\) 0.002** \\ Qwen & 0.171 & 0.252 & 0.204 & 0.890 \(\pm\) 0.003 & 0.842 \(\pm\) 0.004 & 0.866 \(\pm\) 0.003 \\ Yi & 0.231 & 0.257 & 0.243 & 0.898 \(\pm\) 0.003 & 0.833 \(\pm\) 0.004 & 0.864 \(\pm\) 0.003 \\ Llama & 0.146 & 0.365 & 0.209 & 0.508 \(\pm\) 0.020 & 0.455 \(\pm\) 0.017 & 0.480 \(\pm\) 0.020 \\ Baichuan & 0.211 & 0.401 & 0.277 & 0.900 \(\pm\) 0.002 & 0.851 \(\pm\) 0.003 & 0.875 \(\pm\) 0.002 \\ Mistral & 0.276 & 0.401 & 0.327 & 0.894 \(\pm\) 0.008 & 0.838 \(\pm\) 0.012 & 0.865 \(\pm\) 0.011 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Organization element extraction results. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{3}{c}{**Original**} & \multicolumn{3}{c}{**After Fine-Tuning**} \\ \cline{2-7}  & **P** & **R** & **F1** & **P** & **R** & **F1** \\ \hline ChatGLM & 0.592 & 0.584 & **0.583** & 0.874 & 0.862 & 0.868 \\ Qwen & 0.540 & 0.593 & 0.561 & 0.887 & 0.880 & 0.883 \\ Yi & 0.474 & 0.525 & 0.497 & 0.879 & 0.869 & 0.873 \\ Llama & 0.250 & 0.298 & 0.253 & 0.785 & 0.744 & 0.764 \\ Baichuan & 0.434 & 0.552 & 0.475 & 0.897 & 0.887 & **0.891** \\ Mistral & 0.414 & 0.494 & 0.445 & 0.883 & 0.869 & 0.876 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Micro-integrated results. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{3}{c}{**Original**} & \multicolumn{3}{c}{**After Fine-Tuning**} \\ \cline{2-7}  & **P** & **R** & **F1** & **P** & **R** & **F1** \\ \hline ChatGLM & 0.615 & 0.444 & 0.516 & 0.886 \(\pm\) 0.001 & 0.920 \(\pm\) 0.004 & 0.903 \(\pm\) 0.003 \\ Qwen & 0.672 & 0.573 & 0.618 & 0.898 \(\pm\) 0.003 & 0.945 \(\pm\) 0.002 & 0.921 \(\pm\) 0.001 \\ Yi & 0.619 & 0.646 & 0.632 & 0.854 \(\pm\) 0.004 & 0.914 \(\pm\) 0.005 & 0.883 \(\pm\) 0.004 \\ Llama & 0.367 & 0.204 & 0.263 & 0.903 \(\pm\) 0.002 & 0.842 \(\pm\) 0.003 & 0.871 \(\pm\) 0.002 \\ Baichuan & 0.624 & 0.646 & **0.635** & 0.908 \(\pm\) 0.001 & 0.937 \(\pm\) 0.001 & **0.922 \(\pm\) 0.003** \\ Mistral & 0.450 & 0.379 & 0.411 & 0.913 \(\pm\) 0.003 & 0.928 \(\pm\) 0.004 & 0.921 \(\pm\) 0.002 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Key action element extraction results. For different base models, the initial performance on the four elements varies, and due to the lack of police incident data in the pre-training of current open-source large language models, such as definitions of police incident types and elements, the initial models perform poorly on the three types of elements other than the highly general location element, with both accuracy and recall rates below 60% overall. By using the LoRA fine-tuning method and initial training sets combined with police incident-related knowledge, all six base models showed improvement across the four categories of elements after fine-tuning, demonstrating the effectiveness of LoRA fine-tuning and enhancing the models' understanding of police incident content. The performance of the six base models varies, likely due to different pre-training datasets and training strategies used. After fine-tuning, Baichuan shows the best overall performance, and apart from Llama, the other open-source models are not significantly different from Baichuan, with accuracy and recall rate differences within 5%. The Llama model, mainly targeting English, performs poorly in Chinese, but after fine-tuning, both accuracy and recall rates improved by about 50%, showing significant enhancement. Leveraging large language models for entity extraction effectively utilizes the semantic understanding capabilities of the base models.