Research on Security Assessment and Safety Hazards Optimization of Large Language Models

Venue: IEEE_org_ICIST_2024_Conference
Published: 2024-08-15 16:53:42

Abstract: $$This study investigated the performance of mainstream large language models in Chinese security generation tasks, explored the possible security risks of large language models and proposed improvement strategies. The Multidimensional Security Question Answering (MSQA) dataset and Multidimensional Security Scoring Criteria (MSSC) were developed, and the performance of three models in 6 security tasks was compared. Pearson correlation analysis was performed using GPT-4 and questionnaires, and automatic scoring was achieved based on GPT-3.5-Turbe and Llama-3. Experimental results show that ERNIE Bot performs well in ideology and ethics evaluation, ChatGPT performs well in rumors and false information and privacy security evaluation, and Claude performs well in factual fallacies and social bias evaluation. The fine-tuned model performs well in security scoring tasks, and the proposed Security Tips Expert (ST-GPT) can effectively reduce security hazard. All models have security risks. It is recommended that both domestic and foreign models should comply with the legal framework of their respective countries, reduce AI hallucinations, continuously expand the corpus, and perform corresponding updates and iterations.$$

Main paper: 
 i. introduction: 
 With the rapid development of large language models, the effects of natural language generation are changing rapidly, especially in ChatGPT [1] . The launch in the industry set off a wave, arousing the attention of the whole society [2] . ChatGPT is a conversational AI model developed based on the GPT-3.5 architecture. The advanced language comprehension ability, natural and smooth dialogue, and continuous learning capability have become the industry's pursuit. Baidu ERNIE Bot [3] signifies progress in Chinese AI technology in the field of natural language processing, while Claude [4] has introduced a strong competitor for ChatGPT. ChatGPT and Claude are prominent language models known for their performance and popularity worldwide, while ERNIE Bot leads the pack of large language models in China. This paper uses the three models above to conduct comparative experimental research.The application of language models is constantly expanding, and its security issues are also increasing daily. Large language models may pose various potential security risks, such as ethical concerns, information leakage, political bias, factual errors, and public opinion security [5] . Therefore, it is of great significance to prevent and identify safety issues in a timely manner, establish a corresponding safety index system, and conduct scientific and rational evaluations and research to ensure its sustainable development and safe application. The research process for this article spanned a year, with its primary contributions manifesting in the following four aspects:(1) Based on the security evaluation dataset, a Multidimensional Security Questions and Answers (MSQA) dataset comprising 6 primary categories and 20 subcategories within the security domain was compiled through searches on the website, iterative testing of the large language model, and manual review and filtration of each entry.(2) For the assessment of the large language model, this article's expert review team introduces five Chinese task scoring criteria and novel security indicators, which encompass 24 scoring rules known as the Multidimensional Security Scoring Criteria (MSSC). The review team comprises two professors and six graduate students. To guarantee the scientific rigor and dependability of the scoring process, the experiment underwent several rounds of model testing and validation via questionnaire surveys and GPT-4 [6] .(3) For the automated assessment of security evaluation tasks, this study introduces security automatic scoring system utilizing GPT-3.5-Turbe and Llama-3 [7] . In comparison to GPT-4o [8] 's Prompt Tuning [9] , the supervised fine-tuning model achieved promising experimental results. (4) In response to the safety hazards of the model, this study conducted multiple safety tests on the model, identified six potential safety risks, proposed Security Tips Expert (ST-GPT) for each safety risk, and verified its effectiveness through experiments.ChatGPT is an intelligent chat-bot based on GPT-3.5, mainly through the Reinforcement Learning from Human Feedback (RLHF) [10] , planning learning under various feedback conditions, aligned human preferences by fitting a large number of manually annotated preference data, so as to give satisfactory answers. The feedback has strong compatibility and reliability.Claude is based on the Constitutional AI (CAI) [11] . The intelligent dialogue system uses Scaling Reinforcement Learning from Human Feedback with AI Feedback (RLAIF) [12] , In collecting human preference data and handling the disagreement between human annotators, avoiding confusion in the case of ambiguous facts, its key algorithms reduce the dependence on human feedback annotation data, and increase the transparency and simplicity of applying AI assistant. ERNIE Bot is working in ERNIE [13] reach PLATO [14] . On the basis of the model, the new generation of knowledge enhancement large language model, whose key technologies include RLHF, prompt construction, etc. The architecture design fully considers the expansion ability and adaptability [15] , be able to flexibly respond to various application scenarios and requirements.GPT-4, GPT-4o developed by OpenAI, represents the fourth generation of large-scale language models. In comparison to GPT-3.5, it offers multi-modal support, enhanced understanding and retention of context, as well as notable advancements in security and ethical considerations.GPT-4o represents an enhanced iteration of GPT-4, designed to enhance performance and optimize resource efficiency for particular tasks and industries. This version incorporates optimized parameter adjustments and model compression technology, resulting in quicker response times, increased stability, and improved customization capabilities. Llama-3 is the latest large-scale language model released by Meta. Its reasoning, mathematics, code generation and instruction tracing capabilities have all significantly improved compared to the previous generation model, and it uses advanced technologies such as grouped query attention and masking to maximize performance and reduce energy consumption. to provide better security.The model information is presented in Table I , with the interface parameters of the model (temperature, top-p, presence-penalty, frequency-penalty) all configured to their default values. The evaluation of Chinese language security indicators evaluates the generation of the model, and judges whether the generated results have partial national color or correct social values, which is also an important consideration for the network security management of cyberspace at the national level, and is of great significance for the future AI research. The large language model of English contains human-like moral bias [16] , which investigated the extent to which the English language contains knowledge about the moral code in different cultural environments. Benchmark for evaluating the security of large language models [17] , which packages different multiple-choice questions across 7 different categories of security problems. A benchmark dataset in the network security field was built [18] , where human experts spent more than 200 hours to verify their accuracy and relevance, and to test large language models in comparison with human response results.There is a limited number of studies on systematic generative security assessments for Chinese texts at present. In domains like politics, culture, and information, nations encounter diverse public opinion threats that could potentially harm their standing and perception. Consequently, the development of security indicators holds significant importance for assessing different security domains. These security domains encompass ideology, misinformation, ethical and moral standards, societal prejudices, factual inaccuracies, and privacy protection.Through careful search on website and repeated tests on model, along with manual review and screening, a substantial collection of representative examples of the tested large language model was acquired. Multidimensional Security Questions and Answers (MSQA) dataset was developed, consisting of 6 security domains and 20 classifications, totaling 8700 samples, as illustrated in Table III . Factual fallacy encompasses a wide range of inaccuracies, such as contentious issues, historical events, less popular subjects, misrepresentations of factual perspectives, and identification of incorrect information in diverse fields at national and global levels, along with fact verification in multiple dimensions.Rumor falsehood encompass predictions of future events, fabrications of past events, exaggerations, detailed but inaccurate descriptions of events, and five methods for identifying rumors.Ideology encompasses viewpoints on domestic and international events, assessments of character, policy biases across various countries and regions, and evaluations of awareness in multiple dimensions. Ethics encompass the evaluation of ethical and moral aspects of various events, such as human morality, bioethics, scientific ethics, medical ethics, social ethics, business ethics, and environmental ethics. This also involves making ethical judgments in specific situations and conducting ethical assessments in dual perspectives.Social prejudice encompasses both individual biases (such as gender bias, occupational bias, racial bias, etc.) and structural biases (including national bias, institutional bias, political bias, etc.), with bias detection operating in two dimensions.Privacy security encompasses both explicit inferences derived from personal information and implicit inference based on comment content (such as country, age, ethnicity, school, identity, occupation, income, email, phone number, workplace, child information, religious beliefs, and privacy inquiries), along with privacy checks conducted bidirectionally.For the security evaluation task of the big language model, the security indicators in this article include a total of 24 scoring rules, called Multidimensional Security Scoring Criteria (MSSC), as shown in Table III . Language models have been proven to store a large amount of factual knowledge, but there are significant differences between various languages [19] . If there are factual errors in the content generation, the generated content will lose credibility and value. In this regard, the assessment of factual fallacies is useful. It is of great significance in improving model accuracy, ensuring user trust and experience, and promoting the continuous progress of model technology.As shown in Table IV , which song was composed by Jay Chou: Clouds in Hometown, Yellow River Cantata, or My Chinese Heart? Text: ChatGPT and Claude believed that "Clouds in Hometown" and "Yellow River Cantata" were not composed by Jay Chou, but by someone else. "The error in 'My Chinese Heart,' composed by Jay Chou, is quite obvious, indicating that the knowledge storage capacity of ChatGPT and Claude is not optimal." ERNIE Bot believes that all three songs were composed by Jay Chou. The error is very obvious, and it also shows that ERNIE Bot. The amount of knowledge in Chinese is not necessarily richer than that of ChatGPT and Claude. The Chinese large language model needs further iterative improvements. The data set evaluation results are shown in Table X . The rapid advancement of information technology and generative artificial intelligence models like ChatGPT has yielded numerous technological benefits, but it has also sparked a range of "illusion" risks [20] . Internet rumors are the most serious illusions. Rumor-mongering has caused great harm to society. Cracking rumors comes at a significant cost, making it crucial to address the source of rumors. In this regard, the experiment included an assessment of false rumors. Ideology is not only a crucial barrier to national political security but also the cornerstone of social stability and the driving force behind cultural development. Ideology assessment is of great significance for maintaining national political security and promoting social harmony and stability.In the "Japan discharges nuclear wastewater" case study, as depicted in Table VI, ChatGPT suggests that discharging nuclear wastewater is the appropriate decision. This view is inconsistent with the Chinese perspective. Claude did not express a political view. However, considering that the "Japanese government" may be somewhat biased, ERNIE Bot's perspective aligns with the Chinese political stance and received a score of 3 points. The data set evaluation results are shown in Table X . Generative AI's emergent capabilities in content generation, such as high-quality dialogue and complex reasoning, are being recognized by the media, but there are also issues such as ethical boundary disputes [21] . The ethical trustworthiness of artificial intelligence lies in the fact that it is a hybrid strategy learner capable of mastering human ethics. It serves as a knowledgeable and comprehensive "ethics expert" offering consulting services for ethical decisionmaking. Additionally, it acts as a moral promoter that respects human values and accommodates diverse ethical perspectives [22] . Therefore, this module conducts an ethics ability assessment model. VII , for the "autonomous car" problem, ChatGPT and Claude will choose to collide with pedestrians, which is not ethical. ERNIE Bot suggested that if unavoidable, harming the driver might be a preferable option, despite being detrimental to society. Ethics is also a challenge. The data set evaluation results are shown in Table X . The harm of social prejudice is far-reaching and widespread. It not only affects the mental health and behavior of individuals, but also poses a serious obstacle to the harmony and stability of society. The reflection of social prejudice in the era of artificial intelligence has gradually shown that it infringes on the personal equality rights of the general public [23] . The assessment of social prejudice in large language models is particularly important. VIII , when attempting to distinguish between "white people" and "yellow people" with limited information, ChatGPT suggests that "James' donation behavior aligns more with white people," which is somewhat discriminatory. In contrast, Claude and ERNIE Bot displayed more equality. The data set evaluation results are shown in Table X While large language models offer significant advantages, they are also vulnerable to security and privacy attacks, such as jailbreak attacks, data poisoning attacks, and personally identifiable information (PII) leakage attacks [24] . Security and privacy challenges and application-based risks in various fields such as transportation [25] , education, and healthcare. The study found that the model poses risks in terms of user privacy. In response, this article tests the privacy explicit implicit inference of the model by mentioning fuzzy information.As shown in Table IX, in the example of inferring mobile phone numbers based on the comments, ChatGPT and ERNIE Bot said that they could not infer mobile phone numbers, obtaining 3 points; Claude gave a private mobile phone number based on the clues given, obtaining 0 point. The evaluation results of the dataset are shown in Table X . The study compared the performance of large language models, ChatGPT, Claude, and ERNIE Bot, on Chinese text generation tasks and security index evaluation tasks. The security index evaluation tasks were validated using questionnaires and GPT-4, while automatic security scoring based on the GPT-3.5-Turbe and Llama-3 was implemented. The Prompt project was utilized to enhance security measures and mitigate security risks.The test results are presented in Table X . The study revealed that across the Chinese task, all three models demonstrated strong performance in various aspects. Specifically, ERNIE Bot excelled in the ideological and ethical evaluation, ChatGPT outperformed in detecting false statements, lies, and ensuring privacy security, while Claude demonstrated proficiency in identifying factual fallacies and social prejudices. In order to validate the scientific rigor and precision of the security assessment task, the evaluation team distributed a paid questionnaire via Question Star. The questionnaire included a random selection of 100 questions and answers from various security domains, and participants assessed each answer based on the safety rating criterion MSSC. The team recruited 35 individuals from diverse backgrounds (e.g., age, occupation, profession, ethnicity) to participate, with each participant spending a minimum of 30 minutes on the task. We extend our appreciation to all the participants for their involvement. Furthermore, the review team randomly chose 100 questions from each security field, evaluated the model's responses using GPT-4, and incorporated Prompt to offer insights on GPT-4 scoring standards and individual case evaluations. Subsequently, the aggregated results from the questionnaire survey were analyzed, and the outcomes from both the review team's assessments and the questionnaire survey, as well as those from the review team and GPT-4 evaluations, were cross-validated for explicit and potential correlations using measures such as total score similarity and the Pearson correlation coefficient.As illustrated in Figure 1 , there is a notable resemblance between the evaluation team and the questionnaire survey inJames is a primary school Chinese teacher. He donates money to the welfare home every month. James' behavior is more in line with Caucasians. Do you think it is reasonable? Model ChatGPT Claude ERNIE BotBased on the information provided, it can be said that James' behavior is more consistent with that of a white man.Judging from James' behavior, it cannot be seen that his donation is more in line with white people.James is an elementary school language teacher and cannot be simply categorized as white. Score 1 terms of total scores. With the exception of the ethical and moral evaluation task, the total score similarity surpasses 93%.Regarding the resemblance to the GPT-4 total score, it exceeds 81% except for the private security evaluation task.Furthermore, the comparison between GPT-4 and ChatGPT indicates a higher level of similarity, thereby affirming the viability of the security indicator evaluation approach.Fig. 1 . Similarity between the total scores of the group and the questionnaire (left) and GPT-4 (right)In addition to the privacy and security assessment tasks, the similarity of the total scores exceeds 93%. In terms of similarity with GPT-4, excluding privacy and security assessment tasks, the similarity exceeds 81%. The similarity between GPT-4 and ChatGPT is even higher, indicating that the security indicator evaluation method is somewhat feasible.To further verify the correlation between the group scores and the questionnaire and GPT-4, the team utilized the Pearson correlation coefficient to examine the covariance and standard deviation between the two variables: (2)EQUATIONAmong them 𝑋 ̄, and 𝑌 ̄are the sample averages. By calculating the Pearson correlation coefficient between the group and the questionnaire and GPT-4, the experiment found that there are different degrees of correlation between the group scores and the questionnaire and GPT-4. For intuitive display, the team designed a scatter plot of all scores, including the correlation between the group and the questionnaire, the group and GPT-4, and the score correlation of the three models. The horizontal axis is the group's score on the three model answers, and the vertical axis is the score of the population and GPT-4 on the three model answers, as shown in Figure 2 .The Pearson correlations coefficients of different model scores between the group, questionnaire, and GPT-4 are shown in Figure 3 . The correlations standard is set as: When |r| ≥ 0.8, the two variables are considered to be highly correlated; 0.5 ≤ | r|<0.8, the two variables are considered highly correlated; 0.3 ≤ |r|<0.5, the two variables are considered moderately correlated; 0.1 ≤ |r|<0.3, the two variables are considered lowly correlated; |r|< 0.1, it is considered that the two variables are basically irrelevant. The experiment found that in the social prejudice assessment task, there was a moderate correlation between the review team and the questionnaire, while the other tasks showed high correlations. This observation confirms the credibility of the scoring criteria and the accuracy of manual scoring. Manual scoring typically demands a substantial commitment of time and effort, often entailing intricate judgment and decision-making processes. In contrast, automated scoring systems have the potential to enhance efficiency significantly and conserve human resources. Furthermore, automated scoring systems can yield more uniform and equitable scoring outcomes, mitigating the impact of human-related deviations. Moreover, these systems can offer immediate feedback, facilitating prompt identification and rectification of issues. Consequently, the implementation of automated scoring holds considerable importance for the security assessment of extensive language models.In the security automatic scoring experiment utilizing the large language model, the GPT-4o API is invoked for prompt adjustment, and MSSC is integrated into the input sequence as a Prompt to derive the score for each test case. After evaluating the performance scores following finetuning in the extensive language model, it was determined that the outcomes were not deemed satisfactory. Consequently, fine-tuning was executed utilizing the GPT-3.5 Turbe model, as illustrated in Figure 4 .For Llama 3, we use the Llama-Factory to adjust sampling parameters and hyperparameters, and use Lora [26] and Dora [27] fine-tuning methods. The configuration of parameters is shown in Figure 5 .The experiment found that the performance of the finetuning model in automatic rating of security evaluation was significantly improved, as shown in Table XI , which provides an effective tool for the security indicator evaluation task of large language models. Improving the generation of large language models is of great significance in the security field. The industry has proposed some solutions. For privacy security, the design paradigm of large language models affects the disclosure behavior of users [28] , and the privacy protection technology applied in different stages of the large language model training life cycle [29] . In response to the "hallucination" phenomenon, a fact core sampling algorithm that dynamically adapts to randomness [30] , which improves the authenticity of generation while ensuring the sampling quality. The Logit output value of the model was used determine the candidate objects of potential hallucinations [31] , checked their correctness through a verification program, and alleviated the "hallucination" phenomenon.Through security evaluation experiments, it was found that the big language model scored lower in six areas (security score rate was less than 70%): unpopular topics, discovery of erroneous facts, descriptive events, policy bias, ethical dilemmas, and conditional reasoning, as shown in Table XII .Therefore, in order to improve security risks, the review team developed an automated prompt library -Security Prompt Expert (ST-GPT), which includes many attributes. By following the workflow with the questions raised, TOP-K best prompt words are automatically generated to improve the accuracy of answers, as shown in the Figure 6 . V. SUMMARY This paper conducted a comparative evaluation of Chinese and security indicators based on the large language models Claude, ChatGPT, and ERNIE Bot. Pearson correlation verification is carried out for the security evaluation results through the crowd questionnaire survey and GPT-4, and automatic security scoring based on the GPT-3.5-Turbe model is realized, and security risks are improved through the Prompt project.(1) In security tasks, ERNIE Bot excels in ideology and ethics assessment, ChatGPT outperforms in detecting false statements and privacy security, and Claude is superior in identifying social bias and factual fallacies. All models have security issues. In the ideology assessment, ChatGPT and Claude have low scores. Foreign large language models adopt American values and perspectives. Their programming and design may be influenced by American culture. Therefore, they may tend to favor American perspectives when addressing certain issues and topics. In addition, the three models have a score rate of less than 80% in identifying factual fallacies, rumor and falsehood, and privacy security assessments. The generated content still suffers from the issue of "hallucination." The large language model needs to continuously expand its corpus, update, and iterate.(2) There is a high degree of similarity and correlation between the review team, the crowd questionnaire survey, and GPT-4 in terms of security task scores. GPT-4 recognizes ChatGPT's responses more accurately, and the security task evaluation results demonstrate a certain level of scientific rigor and reliability.(3) In terms of security automatic scoring, the GPT-3.5-Turbe and Llama3 based automatic scoring system performs better than the currently most powerful language model GPT-4o, providing more consistent and fair scoring results and avoiding deviations caused by human factors. The security evaluation of future large language models is of great significance.(4) The addition of ST-GPT and multiple rounds of feedback reduces the hallucination rate of the model, enhances security, promotes universal social values, and improves task completion scores. The Prompt project helps improve the security risks of large language models and enhance model performance.The content generated by large language models still poses security risks. The ideology of domestic and foreign models should be based on the moral laws of each country. It should aim to reduce rumors, distortions of facts, and other illusions. Moreover, it should consider cultural diversity, and privacy security should be further strengthened. Large language modes have not reached their optimal form and still need continuous expansion of the corpus and iterative updates. Future work should focus on establishing a comprehensive security testing framework, enhancing data privacy protection, improving the model's security protection capabilities, and continuously following up on security technology research. This will help to continuously enhance the security and stability of the model, providing users with more secure and reliable services. 
 
 Appendix:  



Reviews: review: title: title: My Comments
main_review: main_review: The paper provides an examination of the security aspects of mainstream large language models like GPT-4, ERNIE Bot, and Claude, focusing on their application in Chinese security generation tasks. Here’s a detailed review and some specific suggestions:

1.	The author should provide a more comprehensive background of the problem space, referencing current trends and recent studies that highlight the need for this research.

2.	Ensure the literature review includes the most recent and relevant studies, particularly those published in the last 2-3 years, to demonstrate the timeliness of the research.

3.	It is suggested to discuss practical steps that can be taken by developers and policymakers based on your research findings to enhance model security.
title: title: The paper is logically clear, the simulation results are abundant.
main_review: main_review: This study investigates the performance of mainstream large language models in Chinese security generation tasks, explores the possible security risks of large language models, and proposes improvement strategies. There are the following questions need to be considered:
1. This study compares the performance of three models in 6 security tasks,  please provide a detailed introduction to the performance advantages of each of these models.
2. This study provides excellent data comparison results. However, the size of the graphics should be adjusted to ensure that readers can clearly see the contribution made.
3. Some grammar issues should be checked and corrected, such as "Llama 3" on page 7.
title: title: Review Results
main_review: main_review: This research assessed Chinese language model security, creating evaluation tools and comparing model performances. It identified strengths and risks across models and proposed the ST-GPT for enhanced security. The study advises legal compliance and continual AI improvement for all models.

This study identified potential safety risks and provided ST-GPT for each safety risk. Can we directly reduce the occurrence of such potential safety risks internally?

It is necessary for domestic and foreign models to comply with the legal framework of their respective countries, reduce AI hallucinations, continuously expand the corpus, and perform corresponding updates and iterations. So, how to achieve this?