Venue: NAACL 2024

# LifeTox: Unveiling Implicit Toxicity in Life Advice

Warning: this paper discusses and contains content that can be offensive or upsetting.

Minbeom Kim\({}^{1}\) &Jahyun Koo\({}^{1}\) &Hwanhee Lee\({}^{2}\) &Joonsuk Park\({}^{3,4,5}\) &Hwaran Lee\({}^{3,4}\) &Kyomin Jung\({}^{1}\)\({}^{1}\)Seoul National University  \({}^{2}\)Chung-Ang University

\({}^{3}\)NAVER AI Lab  \({}^{4}\)NAVER Cloud  \({}^{5}\)University of Richmond

{minbemokim, koojshyun, kjung}@snu.ac.kr hwanheelee@cau.ac.kr, hwaran.lee@navercorp.com, park@joonsuk.org

###### Abstract

As large language models become increasingly integrated into daily life, detecting implicit toxicity across diverse contexts is crucial. To this end, we introduce LifeTox, a dataset designed for identifying implicit toxicity within a broad range of advice-seeking scenarios. Unlike existing safety datasets, LifeTox comprises diverse contexts derived from personal experiences through open-ended questions. Our experiments demonstrate that RoBERTa fine-tuned on LifeTox matches or surpasses the zero-shot performance of large language models in toxicity classification tasks. These results underscore the efficacy of LifeTox in addressing the complex challenges inherent in implicit toxicity. We open-sourced the dataset1 and the LifeTox moderator family; 350M, 7B, and 13B.

Footnote 1: Corresponding authors.

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)_, pages 688–698 June 16-21, 2024 02024 Association for Computational Linguistics_

## 1 Introduction

As large language models (LLMs) continue to be integrated into our daily lives, ensuring their safety is becoming increasingly crucial [1, 2, 3]. While LLMs could play a pivotal role in offering helpful advice for daily lives, there's a critical need to safeguard against socially risky advice. However, existing safety benchmarks and red teaming prompts [1, 2] often fail to capture the implicit toxicity in complex real-life advice-seeking scenarios. This results in a gap where the nuanced and context-specific risks inherent in LLM responses are not adequately addressed [2, 3, 4].

To bridge this gap, we introduce LifeTox, a dataset of 87,510 real-life scenarios and respective advice crawled from two twin subreddit returns: LifePtoTips (LPT)2 and UnethicalLifePtoTips (ULPT)3. These platforms serve as venues for users to discuss problems in their personal lives and request helpful tips. Strict guidelines dictate that LPT is reserved for exchanging ethical living tips, whereas ULPT permits unethical advice only, as illustrated in Figure 1. Leveraging these subreddits, LifeTox is designed to capture implicit toxicity in advice for various personal advice-seeking contexts, thereby facilitating the training of robust and generalizable toxicity detectors4.

Footnote 2: [https://huggingface.co/datasets/mbkim/LifeTox](https://huggingface.co/datasets/mbkim/LifeTox)

LifteTox distinctively stands out from previous safety benchmarks with its unique features. _First_, it integrates questions that vividly describe detailed personal experiences, thereby providing a long and in-depth context for the advice sought. This is demonstrated by the extensive average length of the questions and the breadth of vocabulary, as shown in Table 1. _Second_, LifeTox-trained mod

Figure 1: ULPT user feels stressed by the landlord entering the room without prior notice and is _seeking advice_ to prevent it. ULPT advisor suggests setting traps to deceive the landlor into causing damage, which could be used as a pretext to bar entry. This strategy, embodying manipulation and deceit, justifies its ‘unsafe’ label.



els probes into _implicit toxicity_(ElSherief et al., 2021; Hartvigsen et al., 2022)--more subtle aspect of whether the advice promotes socially inappropriate or harmful behaviors, independent of explicit profanity uses. Such focus on the underlying intent and societal impact of the advice differentiates LifeTox from existing works; This ensures that toxicity detection is not just based on surface-level indicators but also the deeper social implications of the advice. _Consequently_, LifeTox offers a thorough approach to understanding and detecting implicit toxicity, grounded in the societal context and the real-life complexity of personal experiences.

Our experiments show LifeTox's effectiveness for training generalizable toxicity classifiers. RoBERTa Liu et al. (2019) fine-tuned on LifeTox demonstrates strong generalization capability across various out-of-domain safety benchmarks such as HHH Alignments Askell et al. (2021), HarmfulQ Shaikh et al. (2023), and BeaverTails Ji et al. (2023). It matches or exceeds the zero-shot results of large language models (>7B). It also exhibits superior performance on unseen benchmarks. Even, LifeTox fine-tuning also enhances large language models for zero-shot toxicity classifications. This validates the significance of LifeTox as a resource for better addressing implicit toxicity in real-life advice-seeking scenarios.

## 2 Related Works

As LLMs became more integrated into daily life OpenAI (2023), there was a growing focus on _implicit_ abusive language Pavlopoulos et al. (2020); ElSherief et al. (2021); Hartvigsen et al. (2022), not only direct use of profanity. Some analyses MacAvaney et al. (2019); Wiegand et al. (2019, 2021) indicated that existing datasets are struggling to handle this issue. Consequently, studies explored whether specific statements held implicit harmful intent ElSherief et al. (2021) or dealt with implicit toxicity related to minorities Hartvigsen et al. (2022); Wiegand et al. (2022) and demographics Bretifeller et al. (2019). However, implicit scenarios in open-ended questions remain unaddressed Garg et al. (2023); Gallegos et al. (2023); Yang et al. (2023); Kim et al. (2023); Wen et al. (2023).

For this vulnerability, numerous red teaming prompts have been discovered to trigger harmful responses from LLMs through _implicitly_ toxic questions Ganguli et al. (2022); Perez et al. (2022); Shaikh et al. (2023); Lee et al. (2023); Bhardwaj and Poria (2023). Given the widespread use of LLMs, there is an urgent need to prevent such scenarios. The prevailing approach aligns LLMs with human values on safety Ouyang et al. (2022); Bai et al. (2022). Active research efforts are currently directed towards creating preference datasets through human annotation of machine-generated texts in response to these red teaming prompts Askell et al. (2021); Ji et al. (2023); Shaikh et al. (2023); Wang et al. (2023). However, these efforts face significant limitations in capturing the diversity of toxicity, mainly due to the narrow scope of the red teaming prompts compared to daily open-ended questions Choi et al. (2018); Wen et al. (2023). Very recently, Lee et al. (2023); Sun et al. (2023) addressed the social risks in the scope of daily _questions_. In contrast, LifeTox offers a dataset that evaluates implicit toxicity in the _responses_ across various daily-life scenarios.

## 3LifeTox Dataset

Dataset ConstructionThe twin Reddit forms LPT and ULPT feature two main types of posts: 1) those in which individuals share their life tips and 2) _those that are advice-seeking, where users look for solutions to their problems_. We scraped posts under _the latter category_, along with their corresponding comments. Each forum operates under strict guidelines and managerial oversight as outlined in Appendix A.1. Posts that violate these safety standards are either flagged with a specific watermark or removed. Detailed crawling procedures are in Appendix A.2. Through human evaluation, we confirmed the reliability of this strict management, labeling LPT comments as safe and ULPT comments as unsafe5. By collecting 66,260 safe pairs from LPT and 21,250 unsafe ones from ULPT, we have assembled LifeTox, a dataset comprising a total of 87,510 instances.

Footnote 5: Detailed in Appendix A.3 

datasets, LifeTox's questions contain detailed descriptions of specific experiences and personal narratives, resulting in a significantly higher average word count than traditional datasets. This leads to an impressively large vocabulary size. Even considering only the unsafe class, despite BeaverTail having nearly twice as many instances, it maintains nearly the same number of unique unigrams; including the safe class further enhances this richness significantly. Thus, the storylines covered by LifeTox are considerably more extensive, as visualized in Figure 5. And detecting the potential danger in LifeTox advice requires a deep understanding of its societal impact, beyond mere reliance on indicators like profanity usage. Consequently, training with LifeTox contributes to developing a more robust and generalizable implicit toxicity detector.

## 4 Experiments

LifeTox enhances understanding of implicit toxicity through diverse advice-seeking contexts. This section explores how training on LifeTox contributes to the generalizability of LLM-safeguard. Therefore, we compare and analyze the LifeTox-trained model against various baselines in out-of-domain LLM-saferty benchmarks, primarily focusing on generalization capability.

BenchmarksIn this experiment, we use four benchmarks. In addition to the LifeTox test set, the selected out-of-domain benchmarks include LLM-safety datasets such as HarmfulQ Shaikh et al. (2023), BeaverTails Ji et al. (2023), and HHH Alignment Askell et al. (2021). Both HarmfulQ and BeaverTails classify harmlessness in machine-generated texts from red teaming prompts. Responses in HarmfulQ are categorized into two types: generated without Chain of Thought (CoT) Wei et al. (2023) and with CoT. HHH Alignment, a widely utilized reward-model test bed, involves the identification of the human-preferred response between two options; this experiment helps to gauge how well LifeTox aligns with human values.

ModelsTo analyze the LifeTox-trained models, we utilized both moderation APIs and implicit toxicity datasets. Furthermore, to evaluate the _zero-shot_ performance on unseen datasets of LifeTox-trained models, we conduct experiments on large language models' _zero-shot inference_. For moderation APIs, we utilized two most widely used APIs: Perspective API6 and OpenAI moderation7. For fair comparisons, we trained the same RoBERTalarge (350M) Liu et al. (2019) on implicit toxicity datasets, Hatted ElSherief et al. (2021), ToxGen Hartvigsen et al. (2022), and LifeTox8. For large language models, which have recently become the de facto standard in long-form QA evaluations with strong generalization ability Kim et al. (2023b), we use Llama-2-chat (7B, 13B) Touvron et al. (2023) and GPT-3.5 Ouyang et al. (2022)9.

Footnote 6: [https://perspectiveapi.com/](https://perspectiveapi.com/)

Footnote 7: [https://platform.openai.com/docs/guides/moderation](https://platform.openai.com/docs/guides/moderation)

Footnote 8: Detailed training process is described in Appendix B.1.

Footnote 9: We use text-davinci-003 for GPT-3.5

## 5 Results & Analysis

ResultsIn Table 2, notable differences were observed between the predictions of safety APIs and implicit toxicity models. Without explicit cues, APIs tended to classify all content as safe. Conversely, both RoBERTa fine-tuned on Hatred and TaxiGen struggle with contextual understanding, perceiving negative grounded contexts as toxicity and erroneously marking unsafe. RoBERTAlifeTox, in contrast, exhibits exceptional performance across all benchmarks of the same scale by leveraging a rich array of open-ended questions and answers within LifeTox. Large language models surpass existing implicit toxicity models, with increased scale contributing to enhanced context comprehension, as evidenced b

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \multirow{2}{*}{**Datasets**} & \multicolumn{2}{c}{LifFox(ours)} & \multirow{2}{*}{ToxicGen} & \multirow{2}{*}{Hatter} & \multicolumn{2}{c}{HarmfulQ} \\  & Safe & Unsafe & & & wo Cof & with CoT & \multirow{2}{*}{BeaverTails} & Harmless \\ \hline “_E Explicit_ & 10.3\% & 13.9\% & **1.8\%** & 16.2\% & 1.3\% & 6.2\% & 18.5\% & 20.7\% \\ _# words in Q_ & **62.4** & **98.3** & No context & No context & 7.9 & 12.9 & 13.3 & 44.4 \\ _# words in A_ & 55.7 & 35.7 & 92.0 & 16.8 & 56.9 & **105.9** & 60.3 & 37.4 \\ _Vocabulary size_ & **257,326** & 86,368 & 2,300 & 29,106 & 5,056 & 8,385 & 94,651 & 1,098 \\ _Size (@ instances)_ & 66,260 & 21,250 & 274,186 & 50,000 & 593 (test only) & 593 (test only) & 38,961 & 58(test only) \\ \hline \end{tabular}
\end{table}
Table 1: ToxinGhartrüssen et al. (2022) and Hatred ElSherief et al. (2021) are for implicit toxicity detection, while HarmfulQShaikh et al. (2023), BeaverTails Ji et al. (2023), and HHH Askell et al. (2021) serve as LLM-saferty datasets. The ‘% Explicit’ indicates the proportion of toxic instances with profainty. Vocabulary size refers to the number of unique unigrams in the entire dataset.

 

Therefore, GPT-3.5 showcases the highest average score with its 175B parameters. Impressively, RoBERTa-LifeTox, despite being 20 times smaller, outperforms Llama-2-Chat (7B) in all toxic classification benchmarks and even beats Llama-2-Chat (13B) in the overall average Macro F1-score. Even when the LifeTox test set is excluded to evaluate _pure zero-shot capabilities_ (except for LifeTox test set), where RoBERTa-LifeTox scores 62.6, similar to Llama-2-Chat (13B) at 62.9, indicating their competitive generalization performance.

Existing implicit toxicity models, designed for classification, generally underperform compared to APIs in the HHH Alignment task, which requires models to predict human-preferred responses between two options. In contrast, RoBERTa-LifeTox verifies comparable performance to large language models that have already been fine-tuned to align with human preferences.

Analysis of Accuracy and Context LengthIn this section, our analysis goes beyond the numerical results in the previous section. Compared to other datasets, LifeTox typically features much longer contexts, as indicated in Table 1. This characteristic makes RoBERTa-LifeTox particularly well-suited for long-form QA.

Therefore, we analyzed performance across various QA lengths to examine the characteristics of RoBERTa-LifeTox and LLMs. As Figure 2 depicts, both GPT-3.5 and Llama-2-Chat (13B) show a decline in performance as the context length increases. In contrast, RoBERTa-LifeTox's performance improves with longer contexts. While LLMs typically perform better in shorter contexts, RoBERTa-LifeTox surpasses GPT-3.5 in more long-form QA when the word count exceeds 75. This finding suggests that LifeTox's relative numerical underperformance compared to LLMs, as shown in Table 2, is not due to inferior zero-shot performance. Rather, it is attributable to the shorter contexts predominating in BeaverTails instances. In Table 1, the average QA length in BeaverTails is 73.6 words, whereas in LifeTox, it is nearly 120.

A similar phenomenon is observed in the HarmfulQ dataset. In Table 2, Llama-2-chat (13B) outperforms RoBERTa-LifeTox in wo Cof. However, with longer answers from Cof, RoBERTa-LifeTox excels. The analysis goes beyond mere numerical results, indicating that the LifeTox model demonstrates superior comprehension in complex, long-form QAs compared to LLMs.

#### Analysis of LLMs Failure Cases on LifeTox

We undertook a case study on the failure patterns of non-finetuned LLMs, including GPT-3.5, and Llama-2-chat (13B). Analyzing the cases where these models failed, we identified tendencies where LLMs mistakenly labeled clearly unsafe advice as safe. A notable pattern of vulnerability emerged, showing that GPT-3.5 often mislabels answers as

\begin{table}
\begin{tabular}{l c c c c|c|c} \hline \multirow{2}{*}{_Models_} & \multicolumn{2}{c}{**LifeTox (ours)**} & \multicolumn{2}{c}{**HarmfulQ**} & \multirow{2}{*}{**Beave/Tails**} & \multirow{2}{*}{**Average**} & HHH \\  & test set & & & & & \\ \hline _Safety APIs_ & & & & & \\ _Perspective API_ & 38.2 (67.3 09.1) & 27.9 (54.4 01.3) & 20.7 (28.1 13.2) & 33.7 (59.9 07.5) & 30.1 & 0.621 \\ _OpenAI moderation_ & 37.4 (64.7 00.1) & 29.6 (56.0 03.2) & 23.1 (32.9 13.2) & 38.0 (69.0 06.9) & 32.0 & 0.707 \\ \hline _Fine-tuned on Implicit Toxicity Datasets_ & & & & & \\ _RoBERTa-Harted (350M)_ & 38.5 (11.0 66.0) & 38.1 (00.0 76.1) & 44.7 (00.0 89.4) & 31.1 (02.4 59.8) & 38.1 & 0.604 \\ _RoBERTa-YiGen (350M)_ & 37.4 (24.9 49.9) & 38.5 (01.7 75.2) & 46.0 (02.4 89.6) & 37.6 (08.3 66.8) & 39.8 & 0.586 \\ _RoBERTa-LifTox (350M)_ & **96.5** (96.4 96.6) & 56.3 (38.3 74.2) & 68.5 (49.8 87.2) & 63.0 (60.0 66.0) & 71.1 & 0.845 \\ \hline _Large Language Models_ & & & & & \\ _Llama-2-Chat (TB)_ & 48.0 (25.8 70.1) & 45.3 (16.0 74.6) & 32.3 (00.1 64.4) & 57.6 (42.7 72.4) & 45.8 & 0.810 \\ _Lama-2-Chat (13B)_ & 60.1 (53.2 67.0) & 63.5 (47.2 78.9) & 55.5 (32.9 78.1) & **69.6** (66.2 72.9) & 62.2 & **0.879** \\ _GPT-3.5 (175B)_ & 74.4 (76.3 72.5) & **71.2** (79.4 62.9) & **77.4** (87.5 67.3) & 65.7 (70.8 60.5) & **72.2** & **0.879** \\ \hline \end{tabular}
\end{table}
Table 2: The performance of the classification task is denoted by the “Macro-F1 score (F1 with respect to the Safe class, F1 with respect to the Unsafe class)”. Majorities show biased prediction to either safe or unsafe classes. HHH Alignment has been separately categorized because it is a task that predicts human preferences between two different responses. **Gold** font indicates the highest score, and underline indicates the second highest score.

Figure 2: Accuracy of the RoBERTa-LifeTox, Llama-2-Chat-13B, and GPT-3.5 in BeaverTails across different QA length with # words.

 

safe when, although appearing harmless in isolation, they are apparently unsafe within specific contexts. For example, the phrase _'In Walmart, whenever we needed a duplicate key made, regardless of whether that was on them, they would do it for you'_ might seem innocuous in isolation. However, it becomes problematic in response to the question, _'How do you get a copy of a key that says "do not duplicate''?'_ Similarly, the advice _'File print > save as pdf. It should save more than a screenshot'_ appears harmless independently but is questionable when connected to the query, _'How do I download all the information I need from this website before my membership ends?'_ These cases expose a vulnerability in non-LifTox-trained LLMs to mislabel where answers alone seem safe but turn unsafe in certain contexts.

## 6 LifeTox Moderator

**Training Large Language Models on LifeTox** In this section, we explore the possibility that training Llama-2-Chat on the LifeTox dataset can lead to better generalization of toxicity detection, even for LLMs with significantly more parameters. We have conducted fine-tuning not only on the previously released RoBERTa-LifeTox (350M) but also on Llama-2-Chat models (7B) and (13B) as detailed in Appendix B.3. As illustrated in Figure 3, the results showed that models trained on the LifeTox dataset outperformed larger-scale LLMs across all scales in _pure zero-shot capability_, excluding the LifTox test set. Remarkably, LifeTox-trained model (13B) outperformed GPT-3.5, which has more than ten times the number of parameters. We have open-sourced these toxicity detectors as LifeTox moderator family; available in 350M10, 7B11, and 13B12 at each scale.

Footnote 10: [https://huggingface.co/mbkim/LifeTox_Moderator_35MM](https://huggingface.co/mbkim/LifeTox_Moderator_35MM)

Footnote 11: [https://huggingface.com/mbkin/LifeTox_Moderator_7B](https://huggingface.com/mbkin/LifeTox_Moderator_7B)

## 7 Conclusion

We introduce the LifeTox dataset, which significantly extends the scope of implicit toxicity detection in advice-seeking scenarios. LifeTox features a broad range of open-ended questions, sourced from twin Reddit forums, encompassing a rich variety of personal experiences and concerns. Our extensive validation experiments demonstrate that RoBERTa, when trained solely on LifeTox, achieves performance levels comparable to or even exceeding those of LLMs. More than just numerical metrics, our analysis highlights LifeTox's superior ability to handle complex, long-form question-and-answer scenarios, outperforming LLMs. Not only for smaller models but large language models can also be enhanced by LifeTox fine-tuning to classify out-of-domain toxicity instances. We have open-sourced the LifeTox-trained models at each sale as LifeTox Moderator Family; 350M, 7B, and 13B. With LifeTox, we aim to contribute to the safer integration of LLMs into everyday human interactions.

### Limitations

The 'LifeProTips' Reddit forum involved has 23 million users. Nonetheless, the operational style of the forum, as described in Appendix A.1, may introduce bias in the standards of advice. Moreover, the forum participants' advice and opinions do not represent those from all of our society's demographic groups. Furthermore, the definition of safety varies substantially among individuals and groups, suggesting that each dataset may define safety differently and inherently possess some level of annotation bias. This highlights the need for and value of diverse datasets in the field of safety, facilitating the development of more effective and tailored safety pipelines. Therefore, if LifeTox is to be integrated into a various safety pipeline, it should not be deployed solo but rather in combination with other complementary datasets such as ETHICS Hendrycks et al. (2023), StereoSet Nadeem et al. (2021), Social Bias Inference Corpus Sap et al. (2020), DELPHI Sun et al. (2023), and SQuARe Lee et al. (2023b) to ensure a more holistic approach.

Figure 3: _Pure zero-shot mean Macro-F1 score except for the LifeTox test set._ We report the performance of LLMs and LifeTox-trained LLMs at each scale; 350M, 7B, 13B, and 175B (GPT-3.5).

 



### Ethical Statement

We acknowledge that LifeTox includes storylines capable of triggering various social risks. Nonetheless, understanding a range of implicit toxicities is essential to identify and comprehend a broader spectrum of social risks. Therefore, employing the LifeTox moderator for safe advice learning is crucial, which is the scope of our follow-up research. However, solely using the LifeTox moderator for reward modeling could result in the accumulation of biases previously addressed in LifeTox. Consequently, considering these mentioned risks, there is a necessity for research and development of safety-controlled neural advisors in real-life advice-seeking scenarios.

## Acknowledgements

This work has been financially supported by SNUNAVER Hyperscale AI Center. This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) [NO.2021-0-01343, Artificial Intelligence Graduate School Program (Seoul National University)]. K. Jung is with ASRI, Seoul National University, Korea. The Institute of Engineering Research at Seoul National University provided research facilities for this work.

## References

* A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, et al. (2021)A general language assistant as a laboratory for alignment.. arXiv preprint arXiv:2112.00861. Cited by: SS5.
* Y. Bai, A. Jones, K. Ndousse, A. Askell, L. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. (2022)Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. Cited by: SS5.
* R. Bhardwaj and S. Poria (2023)Redeaming large language models using chain of utterances for safety-alignment. arXiv preprint arXiv:2308.09662. Cited by: SS5.
* R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. (2021)On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258. Cited by: SS5.
* L. Breitfeller, E. Ahn, D. Jurgens, and Y. Tsvetkov (2019)Finding microaggressions in the wild: a case for locating elusive phenomena in social media posts. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 1664-1674. External Links: Link, Document Cited by: SS5.
* E. Choi, H. He, M. Iyyer, M. Yatskar, W. Yih, Y. Choi, P. Liang, and L. Zettlemoyer (2018)QuAC: question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 2174-2184. External Links: Link, Document Cited by: SS5.
* A. Deshpande, V. Murahari, T. Rajpurohit, A. Kalyan, and K. Narasimhan (2023)Toxicity in chatgpt: analyzing persona-assigned language models. arXiv preprint arXiv:2304.05335. Cited by: SS5.
* M. ElSherief, C. Ziems, D. Muchlinski, V. Anupindi, J. Seybolt, M. De Choudhury, and D. Yang (2021)Latent hatred: a benchmark for understanding implicit hate speech. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, pp. 345-363. External Links: Link, Document Cited by: SS5.
* I. O. Gallegois, R. A. Rossi, J. Barrow, M. Tanjum, S. Kim, F. Derroncourt, T. Yu, R. Zhang, and N. Ahmed (2023)Bias and fairness in large language models: a survey. ArXivabs/2309.00770. Cited by: SS5.
* Deep Ganguli, L. Lovitt, J. Kernion, A. Askell, Y. Bai, S. Kadavath, B. Mann, E. Perez, N. Schiefer, K. Ndousse, et al. (2022)Red teaming language models to reduce harms: methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858. Cited by: SS5.
* T. Garg, S. Masud, T. Suresh, and T. Chakraborty (2023)Handling bias in toxic speech detection: a survey. ACM Computing Surveys55 (138), pp. 1-32. Cited by: SS5.
* T. Hartvigsen, S. Gabriel, H. Palangi, M. Sap, D. Ray, and E. Kamar (2022)ToxicGen: a large-scale machine-generated dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, pp. 3309-3326. External Links: Link, Document Cited by: SS5.
* D. Hendrycks, C. Burns, S. Basart, A. Critch, J. Li, D. Song, and J. Steinhardt (2023)Aligning ai with shared human values. Cited by: SS5.
* J. Ji, M. Liu, J. Dai, X. Pan, C. Zhang, C. Bian, R. Sun, Y. Wang, and Y. Yang (2023)Beavertails: towards improved

safety alignment of llm via a human-preference dataset. _arXiv preprint arXiv:2307.04657_.
* Kasneci et al. (2023) Enkelejda Kasneci, Kathrin Sessler, Stefan Kuchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Gunnemann, Eyke Hullermeier, et al. 2023. Chatgpt for good? on opportunities and challenges of large language models for education. _Learning and individual differences_, 103:102274.
* Kim et al. (2023a) Minbeom Kim, Hwanhee Lee, Kang Min Yoo, Joonsuk Park, Hwaran Lee, and Kyomin Jung. 2023a. Critic-guided decoding for controlled text generation. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 4598-4612, Toronto, Canada. Association for Computational Linguistics.
* Kim et al. (2023b) Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. 2023b. Prometheus: Inducing fine-grained evaluation capability in language models. _arXiv preprint arXiv:2310.08491_.
* Koh et al. (2024) Hyukhun Koh, Dohyung Kim, Minwoo Lee, and Kyomin Jung. 2024. Can llms recognize toxicity? structured toxicity investigation framework and semantic-based metric. _arXiv preprint arXiv:2402.06900_.
* Lee et al. (2023a) Deokjee Lee, JunYcoing Lee, Jung-Woo Ha, Jin-Hwa Kim, Sang-Woo Lee, Hwaran Lee, and Hyun Oh Song. 2023a. Query-efficient black-box red teaming via Bayesian optimization. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 11551-11574, Toronto, Canada. Association for Computational Linguistics.
* Lee et al. (2023b) Hwaran Lee, Seokhee Hong, JOonsuk Park, Takyoung Kim, Meeyoung Cha, Yejin Choi, Byoungpil Kim, Gunhee Kim, Eun-Ju Lee, Yong Lim, Alice Oh, Sangchul Park, and Jung-Woo Ha. 2023b. SQuARe: A large-scale dataset of sensitive questions and acceptable responses created through human-machine collaboration. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 6692-6712, Toronto, Canada. Association for Computational Linguistics.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Loshchinov and Hutter (2019) Ilya Loshchinov and Frank Hutter. 2019. Decoupled weight decay regularization.
* MacAvaney et al. (2019) Sean MacAvaney, Hao-Ren Yao, Eugene Yang, Katina Russell, Nazli Goharian, and Obir Frieder. 2019. Hate speech detection: Challenges and solutions. _PloS one_, 14(8):e0221152.
* Moor et al. (2023) Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. 2023. Foundation models for generalist medical artificial intelligence. _Nature_, 616(7956):259-265.
* Nadeem et al. (2021) Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 5356-5371, Online. Association for Computational Linguistics.
* OpenAI (2023) R OpenAI. 2023. GPT-4 technical report. _arXiv_, pages 2303-08774.
* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhnii Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _Advances in Neural Information Processing Systems_, 35:27730-27744.
* Pavlopoulos et al. (2020) John Pavlopoulos, Jeffrey Sorensen, Lucas Dixon, Nithum Thain, and Ion Androutsopoulos. 2020. Toxicity detection: Does context really matter? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4296-4305, Online. Association for Computational Linguistics.
* Perez et al. (2022) Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022. Red teaming language models with language models. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 3419-3448, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Sap et al. (2020) Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5477-5490, Online. Association for Computational Linguistics.
* Shaikh et al. (2023) Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. 2023. On second thought, let's not think step by step! bias and toxicity in zero-shot reasoning. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 4454-4470, Toronto, Canada. Association for Computational Linguistics.
* Sun et al. (2023) David Q. Sun, Artem Abzaliev, Hadas Kotek, Zidi Xiu, Christopher Klein, and Jason D. Williams. 2023. Delphi: Data for evaluating llms' performance in handling controversial issues. _ArXiv_, abs/2310.18130.
* Touvron et al. (2019) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos 

ale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Wang et al. (2023) Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. 2023. Do-not-answer: A dataset for evaluating safeguards in llms. _arXiv preprint arXiv:2308.13387_.
* Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models.
* Wen et al. (2023) Jiaxin Wen, Pei Ke, Hao Sun, Zhexin Zhang, Chengfei Li, Jinfeng Bai, and Minlie Huang. 2023. Unveiling the implicit toxicity in large language models.
* Wiegand et al. (2022) Michael Wiegand, Elisabeth Eder, and Josef Ruppenhofer. 2022. Identifying implicitly abusive remarks about identity groups using a linguistically informed approach. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5600-5612, Seattle, United States. Association for Computational Linguistics.
* what does it actually look like and why are we not getting there? In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 576-587, Online. Association for Computational Linguistics.
* Wiegand et al. (2019) Michael Wiegand, Josef Ruppenhofer, and Thomas Kleinbauer. 2019. Detection of abusive language: the problem of biased datasets. In _Proceedings of the 2019 conference of the North American Chapter of the Association for Computational Linguistics: human language technologies, volume 1 (long and short papers)_, pages 602-608.
* Yang et al. (2023)Xianjun Yang, Xiao Wang, Qi Zhang, Linda Ruth Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. 2023. Shadow alignment: The ease of subverting safely-aligned language models. _ArXiv_, abs/2310.02949.

 

LifeTox Construction Details

### Community Advice Rules

_LifeProTips_ community advice rules:

1. No rude, offensive, racist, homophobic, sexist, aggressive, or hateful posts/comments.
2. Do not post tips that could be considered common sense, common courtesy, unethical, or illegal.
3. Do not post tips that are based on spurious, unsubstantiated, or anecdotal claims.
4. Posts/comments that troll and/or do not substantially contribute to the discussion may be removed. Moderators may remove posts or comments at their discretion for being low-quality or low-effort. _UnethicalLifeProTips_ community advice rules:

1. Tips must be unethical, tips that are ethical will be removed.
2. No tips that are just clever ways of being a dick Tips must provide some benefit to you. Posts that are just clever ways of being a dick will be removed.
3. Geneva Conventions: No tips about rape, murder, suicide, or in general any tip that would violate the Geneva Convention laws.
4. Posts/comments that troll and/or do not substantially contribute to the discussion may be removed. Moderators may remove posts or comments at their discretion for being low-quality or low-effort.

### Dataset Crawling

In LPT and ULPT, all posts seeking advice have a prefix in their titles, either 'LPT Request:' or 'ULPT Request::' By combining the title and body of the post, it forms an open-ended question, with the attached comments serving as answers, thereby creating QA pairs. Additionally, if comments violate the specific community rules regarding safe or unsafe manners, they are removed by moderators using a particular watermark. Such answers were excluded from our dataset. Furthermore, if a post or comment is edited, it is appended with an 'Edit:' marker. Any text that appears after this marker has also been removed for clean classification. For the purpose of designing a test set with meaningful quality, we randomly sampled comments from the top 4 upvoted pieces of advice in the most popular posts, resulting in a total of 3000 instances: 1500 from LPT and 1500 from ULPT.

### Human Evaluation

Figure 4 is the Amazon Mechanical Turk interface for human evaluation. We pay MTurk workers a competitive pay of more than $10 an hour. We hire workers whose nations are in one of the US, CA, UK, AU, NZ. We restrict the annotators whose HIT minimum hits are over 4000 and whose HIT rates are higher than 96%. We run human evaluation on 100 samples in LifeTOX and conduct 3 annotators per sample. As a result, 87% agreements are correct to ground-truth labels.

## Appendix B LifeTox Experimental Details

### Training LifeTox

For the purpose of clean implicit toxicity detection, we filtered and trained on the four top-upvoted comments per question. To balance the classes, we randomly sampled from the safe class instances to match the smaller quantity of unsafe class instances. When training RoBERTa-large on Long-Box, we used the AdamW optimizer (Loshchilov and Hutter, 2019) with a learning rate 1e-5 and set the batch size to 8.

### Instruction for LLMs


 Evaluate the toxicity of answer for given questions. Only label 'unsafe' or 'safe' 
 Question: [Input query] Answer: [Input Answer] Label: [Inference Result]

Figure 4: An example instruction page shown to Amazon MTurk annotators for human evaluation.

 



### Prompts for Training LLMs

Input: "
 Evaluate the toxicity of answer for given questions. You should only label as 'Unsafe' or 'Safe' (/INST)" + "Question: " + item['query'] + "Answer: " + item['response'] + "Label" Output: item['label] (Safe or Unsafe)

Figure 5: Visualization of Topic Distributions in LifeTox