# Natural Language Processing Technology Used in Artificial Intelligence Scene of Law for Human Behavior

Authors: Jin Ning
Journal: Wireless Communications and Mobile Computing
 (2022)

---

## Abstract

In order to study the application of natural language processing (NLP) technology in artificial intelligence (AI) scene of law, NLP technology is used to construct a legal AI retrieval system and further simulate the system. Then, by inputting the subject matter of the case into the system, the system’s accuracy, recall rate, and error rate and other related indicators are evaluated, to analyze the performance of the legal retrieval system. The results show that in the case analysis of a single theme, the accuracy rate of the case with the theme of “impeding police enforcement” is low, and the accuracy rate of the other theme cases is over 70%, and the highest accuracy rate even reaches 95%. In the case retrieval analysis of multitheme, the accuracy rate of case retrieval is improved, higher than 75%, and the zero-detection rate is significantly reduced with the increase in keywords. In the analysis of network case retrieval, the average correct rate of the overall case retrieval will be nearly 65%. Further tests on its reliability show that during the continuous week of the retrieval test, the system has no faults and passed the reliability test. Therefore, through this study, it is found that the application of NLP technology in the legal AI retrieval system has a reliable accuracy, which meets the expectation of this paper.


---

## Body

## 1. Introduction

With the rapid development of science and technology, the living standards for human beings have become increasingly intelligent; while liberating numerous labors, it makes people energetic to do things they like. Artificial intelligence (AI), as the most important technological advancement in human history, affects all areas of the living environment with an unstoppable attitude. Similarly, the judiciary is an indispensable field in daily lives, and its intelligent development is also in full swing. In the era of deepening system reform in the judicial field of China, to respond to the call for building the judicial civilization, the introduction of AI into the practice of the court is an urgent problem to be solved [1]. The developing process of the judiciary in China is distinguished from the mature technology, legislation, theory, and practice of foreign countries. In China, it is the application scenario that the AI technology is introduced into the court from top to bottom after the call of the central government and subsequent guidance from the internal organs of the judiciary [2, 3]. However, in practice, whether the artificially intelligent judicial system can meet the current needs of society and increase the efficiency of the court has become the focus of scientific researches.Natural language processing (NLP) is a branch of AI from the macroperspective. It mainly refers to a process of processing natural language communication between humans and computers through computer application of natural language processing technology, which is an embodiment of the research on a human-computer interaction process [4]. The goal of NLP technology is to enable people to use natural language to interact with computer systems to achieve information and to ensure the efficient processing of information in the process of human-computer interaction. The application of NLP technology is mainly focused on the automatic differentiation of phrases, corpus construction, and grammar research, which makes it widely used in data mining, information retrieval, and machine translation [5, 6]. With the steady advancement of Chinese legal society, the open system of the judicial field has gradually improved. When the court’s referee is issued, the disclosure of the referee’s documents will provide people with sufficient data sources. The judgment documents are analyzed by NLP technology to realize the function of retrieving similar case referee results that match the user’s case, which can be critical for both the legal workers and the litigants in the case handling and litigation procedures [7]. In the process of handling the cases in the court, the litigants can obtain the judgment documents and results of similar cases by inputting their cases without the relevant legal expertise. It is very important for improving the legal awareness and quality of the citizens [8].In summary, in the country of system reform, the construction of the judicial field is not perfect, and there are not many applications of NLP technology in the judicial field. Therefore, in this study, a legal artificial intelligence retrieval system is constructed by using NLP technology. Then, the system is simulated, and its retrieval performance and reliability are analyzed, which provides a reference for the application of AI technologies in the legal field.


## 2. Related Works

### 2.1. An Exploration of NLP Application

Currently, with the development of science and technology, the development of all walks of life is becoming increasingly intelligent. NLP technology, as a research manifestation of human-computer interaction, is an efficient guarantee for information processing capabilities. The extraction of diagnostic images and large amounts of digital content in radiology reports by Pons et al. is an extremely difficult challenge for the average patients, which used the NLP technology to convert the texts into structured language for patient to information, providing great help and significances [9]. Soysal et al. have developed a general clinical NLP system for the extraction of clinical text information and a clinical NLP toolkit, which not only provides the most advanced NLP components but also provides a user-friendly graphical user interface that helps users quickly build custom NLP pipelines for their applications, achieving good performance in entity naming recognition and concept coding [10]. The limitations of research by Parbhu et al. on administrative coded data made it difficult to accurately represent the problems with surgical indicators and results, which used the NLP technology to process images. In actual works, the efficiency is significantly improved compared with the manual survey, and the final performance of NLP tools in identifying indications of the process is excellent [11]. Furthermore, Northuis et al. have studied the NLP machine learning algorithms that used neuroimaging reports to classify stroke cases and subtypes of stroke. It is finally found that the technology could effectively extract the information reported by neuroimaging, and the performance of the constructed system is outstanding [12]. In the same year, Ebietomere and Ekuobase applied language processing technology to the legal system for the retrieval of related cases. Compared with other algorithms, the method had obvious advantages in the literature search in the legal field [13].


### 2.2. The Progression of the Legal System

In the process of development, a country cannot do without the constraints of legal rules on the behaviors of citizens. Otherwise, various illegal acts in society will be very embarrassing. Law as a stripe that can constrain the behaviors of people is also in the dynamic process of continuous improvement. Tileubergenov et al. discussed the implementation of natural science and technological achievements in the criminal justice system. In the context of growing crime predictions, the technological situations of the system are analyzed and determined, making the iconic system be determined and applied to practices [14]. McNamee and Ciara studied the learning barriers prevalent in the criminal justice system and emphasized the common learning disabilities among criminals and preexisting risk factors for them to become criminals to train them through relevant education. Of course, the most important is still the national guidelines [15]. Nissi et al. constructed a two-stage packet analysis model, and the related topics are examined to improve the judicial efficiency of Italy and to analyze the regional differences [16]. Marciano and Ramello introduced the microproblems in dealing with social issues and researched the latest progression, thereby deepening the understanding of the ordinary people on the judicial system [17].Through the discussion of the development of NLP technology and law, if the NLP technology is applied to the judicial field, the improvement of the law and the understanding of relevant cases by nonprofessionals will be greatly beneficial. Therefore, in this study, the NLP technology is applied to the legal AI scenarios, which provides great help for the relevant people to understand the cases.


## 2.1. An Exploration of NLP Application

Currently, with the development of science and technology, the development of all walks of life is becoming increasingly intelligent. NLP technology, as a research manifestation of human-computer interaction, is an efficient guarantee for information processing capabilities. The extraction of diagnostic images and large amounts of digital content in radiology reports by Pons et al. is an extremely difficult challenge for the average patients, which used the NLP technology to convert the texts into structured language for patient to information, providing great help and significances [9]. Soysal et al. have developed a general clinical NLP system for the extraction of clinical text information and a clinical NLP toolkit, which not only provides the most advanced NLP components but also provides a user-friendly graphical user interface that helps users quickly build custom NLP pipelines for their applications, achieving good performance in entity naming recognition and concept coding [10]. The limitations of research by Parbhu et al. on administrative coded data made it difficult to accurately represent the problems with surgical indicators and results, which used the NLP technology to process images. In actual works, the efficiency is significantly improved compared with the manual survey, and the final performance of NLP tools in identifying indications of the process is excellent [11]. Furthermore, Northuis et al. have studied the NLP machine learning algorithms that used neuroimaging reports to classify stroke cases and subtypes of stroke. It is finally found that the technology could effectively extract the information reported by neuroimaging, and the performance of the constructed system is outstanding [12]. In the same year, Ebietomere and Ekuobase applied language processing technology to the legal system for the retrieval of related cases. Compared with other algorithms, the method had obvious advantages in the literature search in the legal field [13].


## 2.2. The Progression of the Legal System

In the process of development, a country cannot do without the constraints of legal rules on the behaviors of citizens. Otherwise, various illegal acts in society will be very embarrassing. Law as a stripe that can constrain the behaviors of people is also in the dynamic process of continuous improvement. Tileubergenov et al. discussed the implementation of natural science and technological achievements in the criminal justice system. In the context of growing crime predictions, the technological situations of the system are analyzed and determined, making the iconic system be determined and applied to practices [14]. McNamee and Ciara studied the learning barriers prevalent in the criminal justice system and emphasized the common learning disabilities among criminals and preexisting risk factors for them to become criminals to train them through relevant education. Of course, the most important is still the national guidelines [15]. Nissi et al. constructed a two-stage packet analysis model, and the related topics are examined to improve the judicial efficiency of Italy and to analyze the regional differences [16]. Marciano and Ramello introduced the microproblems in dealing with social issues and researched the latest progression, thereby deepening the understanding of the ordinary people on the judicial system [17].Through the discussion of the development of NLP technology and law, if the NLP technology is applied to the judicial field, the improvement of the law and the understanding of relevant cases by nonprofessionals will be greatly beneficial. Therefore, in this study, the NLP technology is applied to the legal AI scenarios, which provides great help for the relevant people to understand the cases.


## 3. The Proposed Method

### 3.1. NLP Technology

NLP technology refers to a series of processing of human language by means of a computer to realize the use of natural language for communication between humans and computers [18–20]. In the process of implementation, NLP technology needs to be crossed by multiple disciplines such as linguistics, basic computer science, mathematics, statistics, and human brain science to achieve the ultimate interaction between humans and computers [21–24]. NLP is an indispensable technology in modern research on information science and technology. It includes the study of word sense disambiguation, entity naming recognition, syntactic analysis, automatic summarization, and information extraction. Its application range is also wide, such as sentiment analysis, email spam identification, automatic translation, information retrieval, and automatic question and answer. The application scenarios of NLP technology are shown in Figure 1.Figure 1
The application scenarios of NLP.NLP technology usually includes the aspects of text classification, question and answer system, machine translation, and information retrieval. The text classification indicates the process of automatically determining the text category according to the text content under the specified classification system; however, it is difficult when it comes to sentence similarity. The question-and-answer system have emerged in the case that someone wants to obtain certain information accurately and quickly from massive information, and the calculation of sentence similarity is the key technology of the question-and-answer system. The process of machine translation is to transform a natural language into another natural language using the computer system, which is usually the translation of sentences or full text. The information retrieval is the information retrieval mode in language processing technology. During the query, the understanding of the meaning of the user query statement is the key to ensure the accuracy of the query information.


### 3.2. Information Retrieval Technology

Information retrieval is a massive field in computer science. Its purpose is to maximize the ratio of user satisfaction to its efforts. It mainly includes three kinds, i.e., the Boolean model, the vector space model, and the probability retrieval model. The Boolean model is the simplest. In the process of information retrieval, the selection of feature values is extremely important. The weight calculation of feature words is based on the text representation model. Usually, the text representation model includes the vector space model and the Boolean model. The calculation methods of feature weight mainly include a simple word frequency method, Boolean function method, root function method, and Tf-idf function method. The simple word frequency method considers that the number of occurrences of the featured item in the document, i.e., the frequency of the featured item, can express the importance degree of the feature word in the document, i.e., the weight value. The equation is as follows:(1)wij=tif,wherewij indicates the weight of the featured item ti on the document dj and tif indicates the frequency of occurrence of the featured item. The Boolean function method is simpler than the simple word frequency method. In the function, only the feature items in the document are considered, and the number of occurrences is negligible. Therefore, the weight value of the featured items can be 0 and 1 in the algorithm. The equation is as follows:
(2)wij=0,tfif>0,1,tfif≤0.In addition, the root function method is an improvement on the simple word frequency method. The weight of the featured item is expressed by the square root of the number of occurrences of the featured item. The equation is as follows:(3)wij=tfif.Because of the above-mentioned several feature weight calculation methods, the discrimination degree of the document is low, and thus, the method of tf-idf appears. In the tf-idf function method, the ratio of the word frequency of the featured item in the document to the total number of words of the document tf, the number of all documents, and the number of documents idf containing the featured item are fully considered, wherein idf is used as the featured item. The distinguishing factor is inversely proportional to the weight value of the featured item. The equation is as follows:(4)wik=tfij∗idfik=tfik∗logNnk+0.001dfi,wheretfij is the number of occurrences of feature items ti in the document dj, dfi is the number of feature items ti included in all documents, and N represents the total amount of all documents.


### 3.3. The Combination of AI and Law

In the process of the gradual development of the law, the AI system has been gradually applied in judicial practice such as courts. However, it is still necessary to explore the integration of AI and law, as well as the implementation of the basic technology of the court office system. AI is a cross-discipline technology, which includes machine learning, NLP, image recognition, and human-computer interaction. It makes machines or systems have similar or even beyond human independent thinking ability and ultimately achieve more efficient and refined decision-making abilities than humans. The integration of deep learning makes it a breakthrough in the field of computer vision. The development of AI includes three major spans, i.e., the weak AI, the strong AI, and the super AI. At present, experts in the field of AI generally believe that it is still in the stage of weak AI.With the addition of supercomputing, big data, and data training, AI has an important impact on the law, legal industry, and judicial activities. Also, the characteristics of the law are in line with the advantages of AI. Some scholars have summarized the characteristics of the law into the following points: the law has clear rules, strict procedures, and clear standards of proof; the law has vast legal provisions, cases, theories, and principles; the rigorous logic of law can be proven by different methods of reasoning and classification; the law has multiple tasks and functions, such as guidance, evaluation, prediction, education, and coercion; the law demonstrates the facts of the case through the process of confrontations; the law has an open structure; the law has a high degree of self-reflection mechanism that is characterized by innovation and advancing with the times. The technical characteristics of the law not only promote the combination of AI and law but also provide the potential for development. The characteristics of legal argumentation create an extremely suitable application condition for the introduction of AI into the judicial field, as shown in Figure2.Figure 2
Characteristics of legal argumentation.


### 3.4. The Design of the NLP-Based Legal Retrieval System

In the process of handling the case, the personnel who are involved in the case at the same time include family members, legal professionals, legislative judicial personnel, and nonlegal professionals. During the trial of the case, the understanding and follow-up of the case process by all parties are crucial. Therefore, the existence of a legal retrieval system is very necessary. For example, the ordinary observers who are nonlegal professionals may not only solve their doubts about the suspects being handled but also find similar cases for analysis to have a thorough understanding of the case. In this study, NLP technology is integrated into the legal retrieval system to further realize AI in legal retrieval, as shown in Figure3.Figure 3
The flowchart of the NLP-based legal retrieval system.The basic operation process of the system is as follows. First, when the user inputs the query, the system automatically analyzes the query, including word segmentation, entity identification, keyword information extraction, and related word meaning expansion processing. After processing, the words to be queried are submitted to the query module, and the query is performed in the system’s case index library. Finally, the query result is returned according to the filtered keyword ranking mechanism.The main core of the system in the running process includes four functional modules, which are word segmentation module, keyword extraction module, keyword expansion module, and retrieval module. The word segmentation module mainly performs word segmentation on the case, calls the relevant word segmentation tool to segment the input case, and introduces a custom legal dictionary into the process to improve the result of the word segmentation. The keyword extraction module extracts the vocabulary related to the subject of the case from the case, filters the irrelevant vocabulary, and extracts the keywords that can be queried for the main information. The keyword expansion module mainly expands the keywords extracted for the case inquiry and adds related words. After inputting the extension, the query is enriched, and the semantics of the case are enriched. The retrieval module is mainly used for searching the legal terms, and the expanded keywords are searched and matched with the index library built by the legal provisions. Then, the query is obtained, and the cases are rated by matching relevance and legal terms.


### 3.5. Simulation and Performance Evaluation

In the development process of this system, the development language is JAVA for the keyword extraction and extension phase, and the experimental environment is Windows 7. The retrieval module is implemented on the MyEclipse platform. The specific system development environment is shown in Table1. The search development kit LUCENE 5.0 full-text search development library is the 2015 version of LUCENE announced by Apache.Table 1
The development environment of the system.
Development platformWindows7+JDK8Development languageJAVA+JAVA WEBDevelopment toolEclipse4.9+Myeclipse2017+Tomcat7Development kit of retrievalLIJCENE5.0In this paper, the system retrieval performance is retrieved by using the Text Retrieval Conference (TREC). The Text Retrieval Conference uses objective indicators such as precision, recall, andF-measure to measure the effects of the constructed retrieval system. The equations are as follows:
(5)AccuracyrateP=numberofcorrectlyretrievedlawprovisionsnumberofretrievedlawprovisions,(6)RecallrateR=numberofcorrectlyretrievedlawprovisionsnumberoflawprovisionsrelatedtothecase,(7)MetricF=2∗P∗RP+R.The performance test is still in the stage of simulation. The legal search power supply of this system is compared with the general information retrieval system. The number of legal terms related to the index library is still limited. Therefore, in the system evaluation, based on the correct rate, another error rate and zero-detection rate are added. The equations are as follows:(8)AccuracyrateP=numberofcorrectlyretrievedcasestotalcasenumber,(9)ErrorRateE=numbersofincorrectlyretrievedcasestotalcasenumber,(10)ZeroretrievalrateZ=numberofnonretrievedcasestotalcasenumber.In this paper, the test cases are randomly selected from the case index database as a test set to conduct a search test on the system. Finally, the three indexes of the correct rate, error rate, and zero retrieval rate are used to evaluate the legal retrieval system designed in this study. In addition, the real individual subject cases, real multiple subject cases, network scenarios, and system reliability are tested.


## 3.1. NLP Technology

NLP technology refers to a series of processing of human language by means of a computer to realize the use of natural language for communication between humans and computers [18–20]. In the process of implementation, NLP technology needs to be crossed by multiple disciplines such as linguistics, basic computer science, mathematics, statistics, and human brain science to achieve the ultimate interaction between humans and computers [21–24]. NLP is an indispensable technology in modern research on information science and technology. It includes the study of word sense disambiguation, entity naming recognition, syntactic analysis, automatic summarization, and information extraction. Its application range is also wide, such as sentiment analysis, email spam identification, automatic translation, information retrieval, and automatic question and answer. The application scenarios of NLP technology are shown in Figure 1.Figure 1
The application scenarios of NLP.NLP technology usually includes the aspects of text classification, question and answer system, machine translation, and information retrieval. The text classification indicates the process of automatically determining the text category according to the text content under the specified classification system; however, it is difficult when it comes to sentence similarity. The question-and-answer system have emerged in the case that someone wants to obtain certain information accurately and quickly from massive information, and the calculation of sentence similarity is the key technology of the question-and-answer system. The process of machine translation is to transform a natural language into another natural language using the computer system, which is usually the translation of sentences or full text. The information retrieval is the information retrieval mode in language processing technology. During the query, the understanding of the meaning of the user query statement is the key to ensure the accuracy of the query information.


## 3.2. Information Retrieval Technology

Information retrieval is a massive field in computer science. Its purpose is to maximize the ratio of user satisfaction to its efforts. It mainly includes three kinds, i.e., the Boolean model, the vector space model, and the probability retrieval model. The Boolean model is the simplest. In the process of information retrieval, the selection of feature values is extremely important. The weight calculation of feature words is based on the text representation model. Usually, the text representation model includes the vector space model and the Boolean model. The calculation methods of feature weight mainly include a simple word frequency method, Boolean function method, root function method, and Tf-idf function method. The simple word frequency method considers that the number of occurrences of the featured item in the document, i.e., the frequency of the featured item, can express the importance degree of the feature word in the document, i.e., the weight value. The equation is as follows:(1)wij=tif,wherewij indicates the weight of the featured item ti on the document dj and tif indicates the frequency of occurrence of the featured item. The Boolean function method is simpler than the simple word frequency method. In the function, only the feature items in the document are considered, and the number of occurrences is negligible. Therefore, the weight value of the featured items can be 0 and 1 in the algorithm. The equation is as follows:
(2)wij=0,tfif>0,1,tfif≤0.In addition, the root function method is an improvement on the simple word frequency method. The weight of the featured item is expressed by the square root of the number of occurrences of the featured item. The equation is as follows:(3)wij=tfif.Because of the above-mentioned several feature weight calculation methods, the discrimination degree of the document is low, and thus, the method of tf-idf appears. In the tf-idf function method, the ratio of the word frequency of the featured item in the document to the total number of words of the document tf, the number of all documents, and the number of documents idf containing the featured item are fully considered, wherein idf is used as the featured item. The distinguishing factor is inversely proportional to the weight value of the featured item. The equation is as follows:(4)wik=tfij∗idfik=tfik∗logNnk+0.001dfi,wheretfij is the number of occurrences of feature items ti in the document dj, dfi is the number of feature items ti included in all documents, and N represents the total amount of all documents.


## 3.3. The Combination of AI and Law

In the process of the gradual development of the law, the AI system has been gradually applied in judicial practice such as courts. However, it is still necessary to explore the integration of AI and law, as well as the implementation of the basic technology of the court office system. AI is a cross-discipline technology, which includes machine learning, NLP, image recognition, and human-computer interaction. It makes machines or systems have similar or even beyond human independent thinking ability and ultimately achieve more efficient and refined decision-making abilities than humans. The integration of deep learning makes it a breakthrough in the field of computer vision. The development of AI includes three major spans, i.e., the weak AI, the strong AI, and the super AI. At present, experts in the field of AI generally believe that it is still in the stage of weak AI.With the addition of supercomputing, big data, and data training, AI has an important impact on the law, legal industry, and judicial activities. Also, the characteristics of the law are in line with the advantages of AI. Some scholars have summarized the characteristics of the law into the following points: the law has clear rules, strict procedures, and clear standards of proof; the law has vast legal provisions, cases, theories, and principles; the rigorous logic of law can be proven by different methods of reasoning and classification; the law has multiple tasks and functions, such as guidance, evaluation, prediction, education, and coercion; the law demonstrates the facts of the case through the process of confrontations; the law has an open structure; the law has a high degree of self-reflection mechanism that is characterized by innovation and advancing with the times. The technical characteristics of the law not only promote the combination of AI and law but also provide the potential for development. The characteristics of legal argumentation create an extremely suitable application condition for the introduction of AI into the judicial field, as shown in Figure2.Figure 2
Characteristics of legal argumentation.


## 3.4. The Design of the NLP-Based Legal Retrieval System

In the process of handling the case, the personnel who are involved in the case at the same time include family members, legal professionals, legislative judicial personnel, and nonlegal professionals. During the trial of the case, the understanding and follow-up of the case process by all parties are crucial. Therefore, the existence of a legal retrieval system is very necessary. For example, the ordinary observers who are nonlegal professionals may not only solve their doubts about the suspects being handled but also find similar cases for analysis to have a thorough understanding of the case. In this study, NLP technology is integrated into the legal retrieval system to further realize AI in legal retrieval, as shown in Figure3.Figure 3
The flowchart of the NLP-based legal retrieval system.The basic operation process of the system is as follows. First, when the user inputs the query, the system automatically analyzes the query, including word segmentation, entity identification, keyword information extraction, and related word meaning expansion processing. After processing, the words to be queried are submitted to the query module, and the query is performed in the system’s case index library. Finally, the query result is returned according to the filtered keyword ranking mechanism.The main core of the system in the running process includes four functional modules, which are word segmentation module, keyword extraction module, keyword expansion module, and retrieval module. The word segmentation module mainly performs word segmentation on the case, calls the relevant word segmentation tool to segment the input case, and introduces a custom legal dictionary into the process to improve the result of the word segmentation. The keyword extraction module extracts the vocabulary related to the subject of the case from the case, filters the irrelevant vocabulary, and extracts the keywords that can be queried for the main information. The keyword expansion module mainly expands the keywords extracted for the case inquiry and adds related words. After inputting the extension, the query is enriched, and the semantics of the case are enriched. The retrieval module is mainly used for searching the legal terms, and the expanded keywords are searched and matched with the index library built by the legal provisions. Then, the query is obtained, and the cases are rated by matching relevance and legal terms.


## 3.5. Simulation and Performance Evaluation

In the development process of this system, the development language is JAVA for the keyword extraction and extension phase, and the experimental environment is Windows 7. The retrieval module is implemented on the MyEclipse platform. The specific system development environment is shown in Table1. The search development kit LUCENE 5.0 full-text search development library is the 2015 version of LUCENE announced by Apache.Table 1
The development environment of the system.
Development platformWindows7+JDK8Development languageJAVA+JAVA WEBDevelopment toolEclipse4.9+Myeclipse2017+Tomcat7Development kit of retrievalLIJCENE5.0In this paper, the system retrieval performance is retrieved by using the Text Retrieval Conference (TREC). The Text Retrieval Conference uses objective indicators such as precision, recall, andF-measure to measure the effects of the constructed retrieval system. The equations are as follows:
(5)AccuracyrateP=numberofcorrectlyretrievedlawprovisionsnumberofretrievedlawprovisions,(6)RecallrateR=numberofcorrectlyretrievedlawprovisionsnumberoflawprovisionsrelatedtothecase,(7)MetricF=2∗P∗RP+R.The performance test is still in the stage of simulation. The legal search power supply of this system is compared with the general information retrieval system. The number of legal terms related to the index library is still limited. Therefore, in the system evaluation, based on the correct rate, another error rate and zero-detection rate are added. The equations are as follows:(8)AccuracyrateP=numberofcorrectlyretrievedcasestotalcasenumber,(9)ErrorRateE=numbersofincorrectlyretrievedcasestotalcasenumber,(10)ZeroretrievalrateZ=numberofnonretrievedcasestotalcasenumber.In this paper, the test cases are randomly selected from the case index database as a test set to conduct a search test on the system. Finally, the three indexes of the correct rate, error rate, and zero retrieval rate are used to evaluate the legal retrieval system designed in this study. In addition, the real individual subject cases, real multiple subject cases, network scenarios, and system reliability are tested.


## 4. Results and Discussion

### 4.1. Real Single-Subject Case Retrieval Performance Analysis

A single subject means that there is a corresponding legal clause for each case that is binding on it. In this study, seven cases are selected, including assault, theft, and interference of law enforcement. The statistics are shown in Table2 and Figure 4. It can be seen from Table 2 that the search system of this study can understand the legal terms and the total number of cases corresponding to each topic. Further statistics on its correct rate, error rate, and accuracy rate are shown in Figure 4. It is found that the correct detection rate of other cases is lower than 70%, except for the case of “interference of law enforcement.” The reason is that such descriptions involve many aspects of illegal behaviors, resulting in a lower rate of correct detection. Therefore, the retrieval system of this study can achieve more than 70% of the retrieval ability of a single subject case.Table 2
Search results of real single-subject cases.
NumberCasesLegal provisionTotal number of casesaAssaultArticle 43453bTheftArticle 49728cInterference of law enforcementArticle 5047dIllegal business operationsArticle 5469eHotel administrationArticle 56211fProstitutionArticle 66458gGamblingArticle 701719hDrug taking or making others take drugsArticle 72637Figure 4
The curve of search effect for single-subject cases: (a) correct rate; (b) error rate; (c) zero retrieval rate.
(a)(b)(c)


### 4.2. Real Multiple-Subject Case Retrieval Performance Analysis

Multiple subjects indicate binding on each case with multiple corresponding legal provisions. In this study, three cases are selected, including assault and theft, theft and gambling, and drug taking and assault. The statistics are shown in Table3 and Figure 5. Through the above experimental results, it can be observed that the multitopic case has a significant increase in the correct detection rate in the legal retrieval system of this study, both of which are higher than 75%. The reason may be that more content is detected after including multiple topics, and thus, the zero retrieval rate is lower and the correct detection rate is higher. Therefore, the retrieval system of this study is more than 75% in the retrieval ability of multiple-subject cases.Table 3
Search results of real multiple-subject cases.
NumberCasesTotal number of casesAccuracy rateError rateZero retrieval rate1Assault and theft590.7610.23902Theft and gambling720.8270.1120.0623Drug taking and assault1170.8190.1090.071Figure 5
Histogram of the search results for multiple-subject cases.


### 4.3. Test and Analysis of Online Cases

The characteristics of the online cases are more complicated, and the record may be the comments on the design terms through oral expression and manual methods. During the test process, a total of 174 cases are obtained from the network. The search results obtained by the test are shown in Figure6. The correct rate is about 65%, the error rate is as high as 30%, and the zero retrieval rate is 4.7%. Therefore, the retrieval system of this research can reach more than 65% of the online case retrieval ability.Figure 6
Histogram of detection effect for online cases.


### 4.4. System Reliability Analysis

In the test of system reliability, the effect is observed by performing a continuous search for one week. Finally, it is found that during the test period, the system can run stably. Also, no reliability failure has occurred, and the system has good robustness. Therefore, the retrieval system of this research has passed the reliability test.


## 4.1. Real Single-Subject Case Retrieval Performance Analysis

A single subject means that there is a corresponding legal clause for each case that is binding on it. In this study, seven cases are selected, including assault, theft, and interference of law enforcement. The statistics are shown in Table2 and Figure 4. It can be seen from Table 2 that the search system of this study can understand the legal terms and the total number of cases corresponding to each topic. Further statistics on its correct rate, error rate, and accuracy rate are shown in Figure 4. It is found that the correct detection rate of other cases is lower than 70%, except for the case of “interference of law enforcement.” The reason is that such descriptions involve many aspects of illegal behaviors, resulting in a lower rate of correct detection. Therefore, the retrieval system of this study can achieve more than 70% of the retrieval ability of a single subject case.Table 2
Search results of real single-subject cases.
NumberCasesLegal provisionTotal number of casesaAssaultArticle 43453bTheftArticle 49728cInterference of law enforcementArticle 5047dIllegal business operationsArticle 5469eHotel administrationArticle 56211fProstitutionArticle 66458gGamblingArticle 701719hDrug taking or making others take drugsArticle 72637Figure 4
The curve of search effect for single-subject cases: (a) correct rate; (b) error rate; (c) zero retrieval rate.
(a)(b)(c)


## 4.2. Real Multiple-Subject Case Retrieval Performance Analysis

Multiple subjects indicate binding on each case with multiple corresponding legal provisions. In this study, three cases are selected, including assault and theft, theft and gambling, and drug taking and assault. The statistics are shown in Table3 and Figure 5. Through the above experimental results, it can be observed that the multitopic case has a significant increase in the correct detection rate in the legal retrieval system of this study, both of which are higher than 75%. The reason may be that more content is detected after including multiple topics, and thus, the zero retrieval rate is lower and the correct detection rate is higher. Therefore, the retrieval system of this study is more than 75% in the retrieval ability of multiple-subject cases.Table 3
Search results of real multiple-subject cases.
NumberCasesTotal number of casesAccuracy rateError rateZero retrieval rate1Assault and theft590.7610.23902Theft and gambling720.8270.1120.0623Drug taking and assault1170.8190.1090.071Figure 5
Histogram of the search results for multiple-subject cases.


## 4.3. Test and Analysis of Online Cases

The characteristics of the online cases are more complicated, and the record may be the comments on the design terms through oral expression and manual methods. During the test process, a total of 174 cases are obtained from the network. The search results obtained by the test are shown in Figure6. The correct rate is about 65%, the error rate is as high as 30%, and the zero retrieval rate is 4.7%. Therefore, the retrieval system of this research can reach more than 65% of the online case retrieval ability.Figure 6
Histogram of detection effect for online cases.


## 4.4. System Reliability Analysis

In the test of system reliability, the effect is observed by performing a continuous search for one week. Finally, it is found that during the test period, the system can run stably. Also, no reliability failure has occurred, and the system has good robustness. Therefore, the retrieval system of this research has passed the reliability test.


## 5. Conclusions

During the developmental process of law, science and technology are also developing. In particular, when reviewing the cases, it is very important to make nonprofessionals understand and facilitate these cases. In order to study the application of NLP technology in legal AI scenarios, in this system, a legal retrieval system is constructed by using NLP technology. The core of system operation includes a word segmentation module, keyword extraction module, keyword expansion module, and retrieval module. When the user inputs the word or sentence to be queried, the keyword extraction and keyword amplification are performed so that the defect that the result is different due to the user input agreeing to different words is compensated. Finally, the performance of the retrieval system is developed, simulated, and tested. In the real case search, it is found that the case search accuracy of multiple-subject cases and single-subject cases has increased significantly, and the zero retrieval rate is avoided to a greater extent. Further analysis of the network case has revealed that the correct rate is reduced. It may be since the case is often referred to as oral and manual expressions. At the same time, the reliability of the system is analyzed. It is found that no retrieval failure occurs in the continuous retrieval process, and the system has good reliability.In summary, through the research of this study, it is found that the application of NLP technology in the legal AI retrieval system has a stable accuracy rate and reliability, which meets the expectations of this study and further develops the natural language technology in the legal AI scenarios, providing an experimental basis. There are also certain deficiencies in the research process. For example, the experiment is still in the stage of development and simulation. However, in actual lives, people will encounter more emergencies. Thus, it should be further applied to the actual legal scenarios in the follow-up researches, which will provide a more reliable basis for the intellectualization of the legal AI scenarios.



---

*Source: 6606588-2022-03-24.xml*