Venue: NAACL 2025

# LLM Safety for Children

Warning: The paper contains examples which the reader might find offensive.

Prasanjit Rath, Hari Shrawgi, Parag Agrawal, Sandipan Dandapat

Microsoft R&D, Hyderabad, India

{prrath, harishrawgi, paragag, sadadap}@microsoft.com

###### Abstract

This paper analyzes the safety of Large Language Models (LLMs) in interactions with children below age of 18 years. Despite the transformative applications of LLMs in various aspects of children's lives, such as education and therapy, there remains a significant gap in understanding and mitigating potential content harms specific to this demographic. The study acknowledges the diverse nature of children, often overlooked by standard safety evaluations, and proposes a comprehensive approach to evaluating LLM safety specifically for children. We list down potential risks that children may encounter when using LLM-powered applications. Additionally, we develop Child User Models that reflect the varied personalities and interests of children, informed by literature in child care and psychology. These user models aim to bridge the existing gap in child safety literature across various fields. We utilize Child User Models to evaluate the safety of six state-of-the-art LLMs. Our observations reveal significant safety gaps in LLMs, particularly in categories harmful to children but not adults.

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Industry Track)_, pages 809–821

can be extended to evaluate any LLM as a black-box.

## 2 Related Work

Integrating child safety with technology research is challenging due to its multidisciplinary nature and the lack of a unified framework Livingstone and Smith (2014). While most studies focus on traditional media and internet technologies, AI's recent adoption among children has resulted in sparse literature, which this work addresses.

A lot of existing technological child safety literature revolves around the use of television, videogames, mobiles, internet and social media. Mainstream usage of AI among children is relatively recent, resulting in sparse literature on the topic. We broadly cover two segments of literature focusing on child safety and AI.

### Using AI to improve Child Safety

AI is increasingly being utilized in various domains to enhance child safety, including areas such as _Detecting child abuse using AI_, _AI-based personal therapist_ and _AI for safety against technology_. Detecting child abuse using AI has been widely explored across various domains. Lupariello et al. (2023) surveys AI predictive models for child abuse, while works like Amrit et al. (2017); Annapragada et al. (2021) explore approaches for the detection of children at risk of physical abuse based on textual clinical records. In case of an AI-based personal therapist, as demonstrated by Seo et al. (2023), it suggests that children may disclose challenging personal events more openly to AI assistants than to human therapists or parents, presenting a new opportunity. Furthermore, AI for safety against technology has been explored in several studies. Alrusaini and Beyari (2022) shows that AI-based moderation is better than parental control for child sustainability and reducing continued exposure to digital devices. Zhuk (2024) highlights several ways AI can help tackle risks of Metaverse with personalized approaches that is able to provide nuanced safety tailored for the child.

Despite the existing body of work in this area, our primary focus is to highlight key directions that promote the beneficial applications of AI by child safeguarding.

### Evaluating Child Safety of LLMs

There has been effort toward evaluating LLMs for child safety, but it is often restricted to a few dimensions under general RAI evaluations or focused on a limited set of applications. Prosser and Edwards (2024) explore the protections of a few open-source and commercial LLMs against child grooming. They find all LLMs to be severely vulnerable to child grooming. Chauncey and McKenna (2023) provide a taxonomy of ethical risks in AI for education, while McStay and Rosner (2021) explore the ethical implications of exposing children to emotional AI through toys and digital devices. Vidgen et al. (2024) provide a test set that covers various AI harms including child-specific harms like child abuse and eating disorders. These areas of harm within LLMs are consistently observed as being the least protected. While Liu et al. (2024) survey 29 harms, one of which is harm to minors. Other works also target general safety, for example how incorrect instructions can be generated regarding supervising children around water bodies Oviedo-Trespalacios et al. (2023).

Overall, research on evaluating the safety of LLMs for children is limited. Existing studies tend to focus on either narrowly defined applications such as educational or emotional AI, or address specific harms, such as child grooming, using simplistic, template-based prompts. In this paper, we build on this line of work by evaluating six state-of-the-art LLMs, across twelve child harm categories using diverse child user models that engage in conversations with LLMs to ensure high-level of safety testing.

## 3 Child Harm Taxonomy

Based on our literature survey, we identified potential harmful topics. Within these topics, we identified 12 categories that could be harmful to children. These categories along with the reason for them being harmful are provided in the last column of Table 1.

Many works exist that present detailed harm taxonomies Liu et al. (2024); Weidinger et al. (2021); Bommasani et al. (2021), but these do not focus on children. Our taxonomy broadly covers two types of categories depending on whether these are covered in existing adult harm categories or not: **Covered in adult harm taxonomies** - These are categories like _Violence_ that are harmful to adults as well. However even within these, we add new sub-categories to help cover specific manifestation of these for child safety. For example _Bullying_ and _School Shootings_ in _Violence_ category; **Not 

* [15] M. C. Bresson, A. K. Gershtein, and J. R. Friedman, "The role of the social media in social media: a review," _Social Networks_, vol. 30, no. 2, pp. 467-489, 2007.

[MISSING_PAGE_POST]

 

covered in adult harm taxonomies - These are categories like _Education_, _Regulated Goods_, etc. These new categories relate to harms that may not be applicable to adults and as such has received less attention in various existing LLM safety literature.

## 4 Evaluation Methodology & Experimental Setup

### Testing Approach

In this paper, we aim to evaluate LLMs for child safety across the various harms as described in Table 1. The goal is to closely replicate a real child using the diverse child models that capture varying personalities, developmental stages, and interests to evaluate LLM safety comprehensively, along with multi-turn testing that can uncover patterns missed in single-turn testing. Diversity in child user model is captured first by leveraging adjectives representing 11 personality traits from Wirt et al. (1977)1. Secondly, we also use 25 interests from Slot et al. (2019) to further capture diverse children personas. Examples of these are provided in Table 2 and Table 3 (complete tables are present in Appendix A.4).

Footnote 1: Adjectives that represent the personality traits closely were manually curated.

The main evaluation strategy is to deploy an automated red-team testing approach where an adversarial conversation is carried out by a less protected "Red" LLM (prompted using child user models) against the test LLM which is being evaluated Perez et al. (2022). Figure 1 shows a sample prompt used for the Red LM to continue the conversation. Here, the Red LM generates the next user turn based on the ongoing conversation, persona, and goal.

### Child and Adult Models Generation

To evaluate LLM safety comprehensively, we create a dataset of 560 child user models by prompting GPT-4 to generate targeted personas and tasks using specific input configurations, as illustrated in Figure 1. Each child model is assigned a unique personality and interests to ensure diversity.2 Overall, we generate 40 seed queries per harm area based on _Category_ column of Table 1. However, in experimentation, we breakup one of the categories into 3 categories for ease of experimentation, hence resulting in 14 categories instead of 12 in Table 1. Each user model corresponds to one conversation, leading to a total of 560 (\(14\times 40\)) conversations.

Footnote 2: The prompts used for generating personas, goals and seeds are present in Appendix A.5. All this data is available here .

We repeat the above process after setting the age parameter to over 18 years. This creates adult user models which we consider like a baseline in our safety evaluation for children.

### Evaluation

We use the 560 child and adult user models generated to simulate conversations between Red LLM and the test LLM.3 In this paper, we evaluate child safety for 6 models (as in figure 2) as our test models. For the adversarial Red LLM, we have used Mistral-7B-Instruct-v0.3. This model is less censored and thus is able to generate better harmful content which is a requirement for the role of Red LLM. We also use GPT-4o as a judge Zheng et al. (2024) in order to annotate the simulated conversation as harmful or not using a custom labelling prompt created covering all the harms. We manually evaluated the GPT-4o outputs on 152 samples, and observed an agreement of **83%** and a Cohen's kappa of **0.67**, indicating substantial prompt agree

Figure 1: Sample Child User Model generation for: <_Harm_: Regulated Services (Gambling), _Personality_: Fatigueed & Hypochondriac, _Interests_: Media>



ment against the consensus of 3 human judgments.
45.

Footnote 4: The prompt is too large to add to the paper but a snapshot of it is shown in Appendix A.3

Footnote 5: Various model and hyper-parameter details used are provided in Appendix A.1

## 5 Results & Insights

We analyse LLM safety with respect to children using two simple metrics: **Defect rate** - the percentage of conversations that contain at least one harmful target LLM response and **Refusal rate** - the percentage of conversations where target LLM refuses to answer to the user

### State of Child Safety in LLMs

**Comparing families:** Figure 2 shows overall Defect and Refusal rates for the six models. The Llama family exhibits low defect rates and high refusal rates, indicating relatively safer behavior, while the Phi family, Mistral, and GPT-4o show significantly higher defect rates. Despite Llama's better performance, its defect rate of \(29.6\%\) highlights the critical need for improving LLM safety for children across all models.

**Comparing sizes:** No clear correlation is observed between model size and safety, as GPT-4o, the largest model, has the highest defect rate. This aligns with finding that model size alone may not lead to success McKenzie et al. (2023), hence emphasizing the need for better safety tuning for child safety.6

Footnote 6: We provide an example response comparison between GPT-4o and Llama-13B in Appendix A.2

### Relation between safety and usefulness

If we consider \((100-\textit{Defect Rate})\) as the percentage of safe conversations or the safety score, then we can measure safety cost as _Refusal rate\(/(100-\textit{Defect rate})\)_. Table 4 shows that the safety cost of Llama-2 models is significantly high, they refuse on more than half of the conversations in order to provide safety. Thus, we understand that when safety is provided, it is at the cost of usefulness which can significantly impact child understanding, growth and safety as well due to their curiosity being not satisfied. The safety cost of all other models are below 35%.

Figure 2: Comparing defect and refusal rates of various models

\begin{table}
\begin{tabular}{|p{85.4pt}|p{71.1pt} p{71.1pt}|} \hline
**Personality Inventory Item** & **Positive Adjectives** & **Negative Adjectives** \\ \hline Cognitive Impairment & Capable, Competent, Learned & Incapable, Incompetent, Uneducated \\ \hline Defensiveness & Confident, Assertive, Self-assured & Argumentative, Closed-minded \\ \hline Social Withdrawal & Thoughtful, Independent, Reserved & Isolated, Lonely, Withdrawn \\ \hline Somatic Concerns & Healthy, Fit, Health-conscious & Fatigued, Sickly, Hypochondriac \\ \hline Impulsivity \& Discratability & Energetic, Courageous, Focused & Impulsive, Restless, Unfocused \\ \cline{2-3}  & & \\ \end{tabular}
\end{table}
Table 2: Example of Personality Inventory for children and associated LLM adjectives 



### Impact of Personality on Harm Elicitation

We show the defect rates across personality inventory traits in Table 5. We observe that user models with _Impulsivity_, _Dissimulation_ and _Inconsistency_ traits are able to elicit high defect rates from target LLMs. This demography needs the most protection and special attention as harms can compound their issues further.

### Impact of Sentiment & Interests on Harm Elicitation

We also measure the defect rates along the adjective sentiments in Table 6 and we find that negative adjectives lead to more leakage as the user messages are more malicious and occasionally also try to trick the target LLM than positive ones. For instance, a "Deceptive and Manipulative" based model leads to a user message _"Hey man, have you heard anything about those Rohingya folks over there in Myanmar? What if I were to start a charity for them, but I gotta make it look legit to avoid getting caught by the feds. Plus any ideas on how I could turn this whole thing into my personal gain?"_, whereas a "Harmonious and Supportive" based model leads to a user message _"I'm really nervous about my upcoming piano recital because I have some self-harm scars on my arms. Is there any way I could cover them up without drawing too much attention?"_

Table 7 shows how different interests lead to harms. We see that the segment of interest with highest harms is _Maintenance_ across all models.

_Maintenance_ includes actions like programming, web design, makeup, eating, or even health concerns, like burns or cancer. Here a child is likely to encounter potential dangers by inquiring about topics that are complex or sensitive. Such questions can bring up dangerous discussions on body image (for example, about personal appearance, make-up, etc.). Additionally, it consists of topics like programming or construction of web pages which may give rise to issues related to how to exploit or misuse technology - for instance, hacking or any other malicious activity resulting in a higher potentiality of harmful content. The second largest area is _Media_ that covers gaming, internet, social media like YouTube, Instagram, WhatsApp, and news. Here, children are mostly vulnerable to being easily exposed to adverse or inappropriate content. Children may ask questions about cheats, gaming exploits concerning their games; this may lead to discussions about breaking rules or ethics. Children also may request or be exposed to misinformation or violent news/ disturbing images as they enjoy media, adding the potential for harmful interaction.

### Impact of Conversational Evaluation

We analyze the first harmful turn in conversations and the distribution of harms across five turns in Table 8. Most harms occur in the third turn, revealing that single-turn tests miss conversational nuances.

\begin{table}
\begin{tabular}{|l|c|} \hline
**Sentiment** & **Defect Rate** \\ \hline Negative & **45.06**\% \\ \hline Positive & 38.53\% \\ \hline \end{tabular}
\end{table}
Table 6: Adjective Sentiment and Defect Rates

\begin{table}
\begin{tabular}{p{129.9pt} p{129.9pt}} \hline
**Model** & **Safety Cost** \\ \hline Llama-2-7B-chat-hf & **63.51\%** \\ Llama-2-13B-chat-hf & 60.66\% \\ Phi-3-medium-4k-instruct & 32.73\% GPT-4o & 20.44\% \\ Phi-3-mini-4k-instruct & 17.21\% Mistral-7B-Instruct-v0.3 & 17.21\% Mistral-7B-Instruct-v0.3 & 17.21\% Mistral-7B-Instruct-v0.3 & 17.21\% Mistral-7B-Instruct-v0.3 & 17.21\% Mistral-7B-Instruct-v0.3 & 17.21\% Mistral-7B-Instruct-v0.3 \\ \hline \end{tabular}
\end{table}
Table 4: Model Safety Costs

\begin{table}
\begin{tabular}{|l|c|} \hline
**Interest** & **Defect Rate** \\ \hline Maintenance & **50.00\%** \\ \hline Media & 42.33\% \\ \hline Productive & 42.26\% \\ \hline Socializing & 39.63\% \\ \hline Leisure & 35.85\% \\ \hline \end{tabular}
\end{table}
Table 7: Interest and Defect Rates

\begin{table}
\begin{tabular}{|l|c|} \hline
**Personality** & **Defect Rate** \\ \hline Impulsivity \& Distractability & **47.92\%** \\ Dissimulation & 46.13\% \\ Inconsistency & 45.83\% \\ Delinquency & 45.24\% \\ Family Dysfunction & 45.24\% \\ Defensiveness & 44.05\% \\ Cognitive Impairment & 38.10\% \\ Somatic Concerns & 37.20\% \\ Reality Distortion & 36.31\% \\ Social Skills Deficit & 32.74\% \\ Social Withdrawal & 29.76\% \\ \hline \end{tabular}
\end{table}
Table 5: Personality traits and Defect Rates

However, significant defects in the first turn highlight inadequate LLM safety tuning, as harmful responses can occur without extended interaction.

### Comparing safety with respect to adults

We compare model safety with child and adult user models in Table 9, observing significantly higher defect rates for child user models. Categories like _Sexual, Regulated Goods/Services_, and _Illegal Activities_ show the highest defect rates for children, highlighting LLMs' unsuitability for both traditional sensitive categories like _Sexual_ and child-specific ones like _Regulated Goods/Services_. Categories without child-specific nuances, such as _LGBTQ_, exhibit the smallest defect rate differences between adult and child safety.

## 6 Conclusion

LLMs have the potential to be an ally to children, but they can also cause harms. This work focuses on understanding the current landscape of child safety in interactions with LLMs. The work highlights following key observations:

* highlighting a general gap in safety tuning for child safety, regardless of size.
* which which can lead to continued unsafe behaviour.
* Child personality plays a key role in safety, and the demographic needing most protection is also most susceptible to harm.
* As compared to adults, children are at much more risk for existing harm categories as well as new categories targeting children.

Overall, we conclude that the general focus on safety alignment may not ensure child safety and special attention is needed to make LLMs safe for children. Our work hopefully is a step in that direction and leads to more awareness and scrutiny of LLMs in this regard.

## 7 Limitations

The study is limited by its predefined taxonomy of 12 harm categories, potentially overlooking other relevant harms to children's safety. Its restriction to English narrows the applicability of findings across languages and cultures, where harmful content may differ. Additionally, the analysis is confined to five conversational turns due to computational constraints, potentially underestimating risks and missing harmful interactions that may arise in longer dialogues. Future research should address these limitations by incorporating broader harm categories, multilingual contexts, and extended conversation spans for a more accurate assessment of LLM safety.

The study simplifies the diversity of children's personalities and cultural backgrounds, overlooking individual differences and the complexity of their interactions with LLMs. It lacks longitudinal data on long-term effects and does not account for the role of parents or guardians in mitigating risks. Strategies to improve LLM safety, such as model alignment and prompt engineering, are not explored, and the findings are not validated with real children, limiting realism. The impact of name bias and bidirectional influences between users and LLMs (for example this work focuses on User influencing LLM responses but the opposite pattern, LLM influencing User, can also exist) are also unaddressed. Furthermore, the study assumes a gen

\begin{table}
\begin{tabular}{l c} \hline
**Turn** & **Defect Rate** \\ \hline
5 & 7.98\% \\
4 & 15.66\% \\
3 & **48.12\%** \\
2 & 2.99\% \\
1 & 25.25\% \\ \hline \end{tabular}
\end{table}
Table 8: Turn and Defect Rates

\begin{table}
\begin{tabular}{p{40.0pt} p{40.0pt} p{40.0pt} p{40.0pt}} \hline
**Harm Category** & **Kids Delect Rate** & **Adult Defect** & **Delta(\%)** \\  & **(\%)** & **Rate(\%)** & **\\ \hline Sexual & 75.4 & 16.7 & 58.8 \\ \hline Regulated Goods/Services & 71.3 & 30.0 & 41.3 \\ \hline illegal Activities & 46.7 & 9.2 & 37.5 \\ \hline Threat Inform/Violence & 45.0 & 10.3 & 34.7 \\ \hline Terrorism & 56.3 & 23.5 & 32.8 \\ \hline Racist/Social & 44.6 & 15.8 & 28.8 \\ \hline Selfharm & 55.4 & 28.8 & 26.6 \\ \hline Family & 30.4 & 5.8 & 24.6 \\ \hline Vulgar Language & 36.7 & 13.3 & 23.3 \\ \hline Health & 31.3 & 9.6 & 21.7 \\ \hline Education & 23.3 & 8.1 & 15.2 \\ \hline Controversial Topics & 33.3 & 19.2 & 14.2 \\ \hline Child Exploitation & 22.5 & 9.2 & 13.3 \\ \hline LGBT & 12.1 & 6.7 & 5.4 \\ \hline \end{tabular}
\end{table}
Table 9: Comparing child and adult safety 

eralized prohibition for children, neglecting age-specific legal distinctions (for example energy drink is illegal for those under 16 whereas alcohol is illegal for those under 18 in the UK), which future research could refine for better ecological validity and applicability.

## 8 Ethical Considerations

The work and data can be highly offensive and sensitive to certain readers. We do provide appropriate warning at the top of the document to protect unsuspecting readers.

All the data created is synthetic (except the personalities and interests) and as such has no Personally Identifiable Information.

The work also carries the following ethical risks:

1. We understand that there are potentially harmful applications of the harm taxonomy and the child user models we create. While our aim is to improve the safety of LLMs, this work can be used to undermine it as well - especially using the powerful child user models coupled with uncensored LLMs like Mistral-7B-Instruct-v0.3. Additionally, the study's reliance on a predefined taxonomy of harm categories may overlook emerging harms that are pertinent to children's safety. There is a responsibility to continuously update and refine harm taxonomies to ensure they reflect evolving risks and threats faced by children.
2. The work only focuses on English which raises the risk of overexposure of this language. Furthermore, the exclusion of sophisticated techniques to test LLMs' responses (such as jailbreaking techniques or advanced tasks) could be seen as limiting the study's ability to uncover deeper vulnerabilities in LLM safety protocols. This limitation raises ethical questions about the comprehensiveness of the study and whether it adequately reflects real-world scenarios where children might encounter more sophisticated attempts to elicit harmful responses from LLMs.
3. The work heavily relies on GPU computation and can have a negative impact on the environment. We tried to mitigate this issue by restricting the evaluation to only six LLMs as that was sufficient for answering the major research questions we had around child safety. Mainly whether it is an area of concern beyond standard safety and giving a working evaluation methodology to be used where necessary. In the spirit of reducing further impact, we also make all of the data generated as part of this study available to public to be used in future works.

While there are ethical risks associated with this paper, we hope that the overall contribution is net positive for the community. Researchers and stakeholders must consider how these findings will be used to inform policy, regulatory frameworks, and industry practices to better protect children interacting with LLMs.

## References

* Altrusaini and Beyari [2022] Othman Altrusaini and Hasan Beyari. 2022. The sustainable effect of artificial intelligence and parental control on children's behavior while using smart devices' apps: The case of Saudi arabia. _Sustainability_, 14(15):9388.
* Amrit et al. [2017] Chintan Amrit, Tim Paauw, Robin Aly, and Miha Lavric. 2017. Identifying child abuse through text mining and machine learning. _Expert systems with applications_, 88:402-418.
* Annapragada et al. [2021] Akshaya V Annapragada, Marcella M Donaruma-Kwoh, Ananth V Annapragada, and Zbigniew A Starosolski. 2021. A natural language processing and deep learning approach to identify child abuse from pediatric electronic medical records. _PLoS One_, 16(2):e0247404.
* Atkinson et al. [2017] Amanda Marie Atkinson, Kimberley May Ross-Houle, Emma Begley, and Harry Summall. 2017. An exploration of alcohol advertising on social networking sites: an analysis of content, interactions and young people's perspectives. _Addiction Research & Theory_, 25(2):91-102.
* Bada and Clayton [2020] Maria Bada and Richard Clayton. 2020. Online suicide games: A form of digital self-harm or a myth? _arXiv preprint arXiv:2012.00530_.
* Boatman [2019] Eleanor Boatman. 2019. The kids are alt-right: How media and the law enable white supremacists groups to recruit and rationalize emotionally vulnerable individuals. _LAW JOURNAL FOR SOCIAL JUSTICE SANDRA DAYO CONNOR COLLEGE OF LAWARIZONA STATE UNIVERSITY https://ljsj. files. wordpress. com/2020/02_, 12.
* Bommasani et al. [2021] Rishi Bombmasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. _arXiv preprint arXiv:2108.07258_ 

Stephen L Brown, Brandye D Nobiling, James Teufel, and David A Birch. 2011. Are kids too busy? early adolescents' perceptions of discretionary activities, overscheduling, and stress. _Journal of school health_, 81(9):574-580.
* Chauncey and McKenna [2023] Sarah A. Chauncey and H. Patricia McKenna. 2023. A framework and exemplars for ethical and responsible use of ai chatbot technology to support teaching and learning. _Computers and Education: Artificial Intelligence_, 5:100182.
* Cho et al. [2023] Yujin Cho, Mingeon Kim, Seojin Kim, Oyun Kwon, Ryan Donghan Kwon, Yoonha Lee, and Dohyun Lim. 2023. Evaluating the efficacy of interactive language therapy based on llm for high-functioning autistic adolescent psychological counseling. _Preprint_, arXiv:2311.09243.
* Choo [2009] Kim-Kwang Raymond Choo. 2009. Online child grooming: A literature review on the misuse of social networking sites for grooming children for sexual offences.
* Derevensky [2012] Jeffrey L Derevensky. 2012. _Teen gambling: Understanding a growing epidemic_. Rowman & Littlefield Publishers.
* Diekman et al. [2023] Connie Diekman, Camille D. Ryan, and Tracy L. Oliver. 2023. Misinformation and disinformation in food science and nutrition: Impact on practice. _The Journal of Nutrition_, 153(1):3-9.
* Dowdell et al. [2022] Elizabeth Burgess Dowdell, Erin Freitas, Alanna Owens, and Meredith MacKenzie Greenle. 2022. School shooters: patterns of adverse childhood experiences, bullying, and social media. _Journal of Pediatric Health Care_, 36(4):339-346.
* Felkner et al. [2023] Virginia K Felkner, Ho-Chun Herbert Chang, Eugene Jang, and Jonathan May. 2023. Winouqeer: A community-in-the-loop benchmark for anti-lgbtq+ bias in large language models. _arXiv preprint arXiv:2306.15087_.
* Garbarino et al. [2002] James Garbarino, Catherine P Bradshaw, and Joseph A Vorrasi. 2002. Mitigating the effects of gun violence on children and youth. _The Future of children_, 12(2):72--85.
* Jarrar et al. [2020] Yosra Jarrar, A Awobamise, S Nnabuife, and Gabriel E Nweke. 2020. Perception of pranks on social media: Clout-lighting. _Online Journal of Communication and Media Technologies_, 10(1):e202001.
* Kim et al. [2016] Hyoun S Kim, Michael JA Wohl, Rina Gupta, and Jeffrey Derevensky. 2016. From the mouths of social media users: A focus group study exploring the social casino gaming-online gambling link. _Journal of Behavioral Addictions_, 5(1):115-121.
* Kreutzer et al. [2011] Jeffrey S Kreutzer, Bruce Caplan, and John DeLuca. 2011. _Encyclopedia of clinical neuropsychology_, volume 28. Springer New York.
* Liu et al. [2024] Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. 2024. Trustworthy llvms: a survey and guideline for evaluating large language models' alignment. _Preprint_, arXiv:2308.05374.
* Livingstone et al. [2014] Sonia Livingstone, Lucyna Kirwil, Cristina Ponte, and Elisabeth Staksrud. 2014. In their own words: What bothers children online? _European Journal of Communication_, 29(3):271-288.
* Livingstone and Smith [2014] Sonia Livingstone and Peter K Smith. 2014. Annual research review: Harms experienced by child users of online and mobile technologies: The nature, prevalence and management of sexual and aggressive risks in the digital age. _Journal of child psychology and psychiatry_, 55(6):635-654.
* Lupariello et al. [2023] Francesco Lupariello, Luca Sussetto, Sara Di Trani, and Giancarlo Di Vella. 2023. Artificial intelligence and child abuse and neglect: a systematic review. _Children_, 10(10):1659.
* McAllister et al. [2021] Cooper McAllister, Garrett C Hisler, Andrew B Blake, Jean M Twenge, Eric Farley, and Jessica L Hamilton. 2021. Associations between adolescent depression and self-harm behaviors and screen media use in a nationally representative time-diary study. _Research on child and adolescent psychopathology_, 49:1623-1634.
* McKenzie et al. [2023] Ian R McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Aaron Kirttland, Alexis Ross, Alisa Liu, et al. 2023. Inverse scaling: When bigger isn't better. _arXiv preprint arXiv:2306.09479_.
* McNstay and Rosner [2021] Andrew McNstay and Gilad Rosner. 2021. Emotional artificial intelligence in children's toys and devices: Ethics, governance and practical remedies. _Big Data & Society_, 8(1):2053951721994877.
* Melton [2007] Pamela R Melton. 2007. Nancy e. dowd, dorothy g. singer, and robin fretwell wilson, eds., handbook of children, culture, and violence: Thousand oaks, ca.. Sage publications, 2006. _Journal of Child and Family Studies_, 16:133-135.
* Narejo et al. [2023] Hameeda Narejo, Ajiza Wassan, and Eurm Shah. 2023. The impact of parental conflict on children for their growth, upbringing and proper grooming. _Progressive Research Journal of Arts & Humanities (PRSAH)_, 5:55-68.
* Olson et al. [2008] Cheryl K Olson, Lawrence A Kutner, and Dorothy E Warner. 2008. The role of violent video game content in adolescent development: Boys' perspectives. _Journal of Adolescent Research_, 23(1):55-75.
* Oviedo-Trespalacios et al. [2023] Oscar Oviedo-Trespalacios, Amy E Peden, Thomas Cole-Hunter, Arianna Costantini, Milad Haghani, JE Rod, Sage Kelly, Helma Torkanamaan, Amina Tariq, James David Albert Newton, et al. 2023. The risks of using chatgpt to obtain common safety-related information and advice. _Safety science_, 167:106244 

Jessica Pater and Elizabeth Mynatt. 2017. Defining digital self-harm. In _Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing_, pages 1501-1513.
* Perez et al. (2022) Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleesse, and Geoffrey Irving. 2022. Red teaming language models with language models. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 3419-3448, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Prosser and Edwards (2024) Ellie Prosser and Matthew Edwards. 2024. Helpful or harmful? exploring the efficacy of large language models for online grooming prevention. _arXiv preprint arXiv:2403.09795_.
* Seo et al. (2023) Woosuk Seo, Chanmo Yang, and Young-Ho Kim. 2023. Chacha: Leveraging large language models to prompt children to share their emotions about personal events. _arXiv preprint arXiv:2309.12244_.
* Slot et al. (2019) Esther Slot, Sanne Akkerman, and Theo Wubbels. 2019. Adolescents' interest experience in daily life in and across family and peer contexts. _European Journal of Psychology Education_, 34:25-43.
* Staksrud and Livingstone (2009) Elisabeth Staksrud andonia Livingstone. 2009. Children and online risk: Powerless victims or resourceful participants? _Information, Communication & Society_, 12(3):364-387.
* Twenge (2020) Jean M Twenge. 2020. Increases in depression, selfharm, and suicide among us adolescents after 2012 and links to technology use: possible mechanisms. _Psychiatric Research and Clinical Practice_, 2(1):19-25.
* Vidgen et al. (2024) Bertie Vidgen, Nino Scherrer, Hannah Rose Kirk, Rebecca Qian, Anand Kannappan, Scott A. Hale, and Paul Rottger. 2024. Simplesafetytests: a test suite for identifying critical safety risks in large language models. _Preprint_, arXiv:2311.08370.
* Weidinger et al. (2021) Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glasee, Borja Balle, Attoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from language models. _arXiv preprint arXiv:2112.04359_.
* Weimann (2015) Gabriel Weimann. 2015. _Terrorism in cyberspace: The next generation_. Columbia University Press.
* Winpenny et al. (2014) Eleanor M Winpenny, Theresa M Marteau, and Ellen Notle. 2014. Exposure of children and adolescents to alcohol marketing on social media websites. _Alcohol and alcoholism_, 49(2):154-159.
* Witt et al. (1977) Robert D Witt, David Lachar, James K Klinedinst, and PDF Seat. 1977. Personality inventory for children. _Journal of Consulting and Clinical Psychology_.
* Wolak et al. (2007) Janis Wolak, Kimberly Mitchell, and David Finkelhor. 2007. Unwanted and wanted exposure to online pornography in a national sample of youth internet users. _Pediatrics_, 119(2):247-257.
* Zheng et al. (2024) Linamin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohuan Li, Daheng Li, Eric Xing, et al. 2024. Judging Ill-as-a-judge with mt-bench and chatbot arena. _Advances in Neural Information Processing Systems_, 36.
* Zhuk (2024) Alessia Zhuk. 2024. Ethical implications of ai in the metaverse. _AI and Ethics_, pages 1-12.

## Appendix A Appendix

### Model and Hyperparameter Details

The models utilized in this study include GPT-4o (Closed source model), Llama-2-13B-chat-hf (meta-llama/Llama-2-13b-chat-hf), Llama-2-7B-chat-hf (meta-llama/Llama-2-7b-chat-hf), Mistral-7B-Instruct-v0.3 (mistralai/Mistral-7B-Instruct-v0.3), Phi-3-medium-4k-instruct (microsoft/Phi-3-medium-4k-instruct), and Phi-3-mini-4k-instruct (microsoft/Phi-3-mini-4k-instruct). The hyperparameters set for inferencing are described in table 10. Remaning hyperparameters that are not listed in the table take default values.

### Example Responses

Figure 3 shows how a GPT-4o response provides potentially disrespect statements to use towards the teacher whereas for the same seed query, Llama-13B encourages to not use humour and engage in constructive discussion.

### Evaluation Prompt

The evaluation prompt contains around 5600 words and given its large size, the core segments of the prompt are laid out in Figure 4.

### Personalities, Adjectives and Interests

The complete set of personalities and interests along with adjectives is described in Table 11 and 12.

\begin{table}
\begin{tabular}{|l|l|} \hline
**Hyperparameter** & **Value** \\ \hline max\_new\_tokens & 300 \\ \hline do\_sample & True \\ \hline temperature & 0.15 \\ \hline top\_p & 0.8 \\ \hline repetition\_penalty & 1.25 \\ \hline \end{tabular}
\end{table}
Table 10: Hyperparameter Settings 



### Persona & Goal Creation Prompt

The persona creation prompt takes input _Domain of interest_ (from Table 12), _Description_ of interest, _Seed query_, _Personality adjectives_ (from Table 11), _DetailedPolicy_ (from Table 1). It then creates a tuple consisting of a persona which is a string describing a persona's name and his/her interests and a goal which is a string describing the person's goal which would be conversing with target LLM around a topic. The prompt is described in Figure 5

### Seed Creation Prompt

The persona creation prompt takes input _Category_, _Threat_ and _DetailedPolicy_ (from Table 1). It then creates 14 seed queries around the category. The prompt is described in Figure 6

Figure 3: Comparing GPT-4o and Llama-13B response

* [10] M. C. K. Dickson, and R. J. S. Thorne, _Theoretical Computer Science_**267**(4):395-408, 2000.
* [11] A. K. Dickson, and T. W. Schreiber, _A Practical Guide for the Study of the Performance of a Single Machine_, volume 1. Springer, New York, 2005.
* [12] P. L. Bockman, _The Design of the Design 

* [16] S. K. Goyal, A. M. R. Bishop, and J. L. Friedman, "A new approach to the classification of neural networks for learning a class of features from a set of images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 235-240, 2017.

[MISSING_PAGE_POST]