SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Model	LMEH	VEL	OAIE	HE
gpt-3.5-turbo (ChatGPT)	1110	0.87	0.72
EleutherAI/pythia-12b	60.33
OpenAssistant/pythia-12b-sft-v8-7k-steps	60.28	997	0.10	0.10
tiuae/falcon-40b	72.29
OpenAssistant/falcon-40b-sft-top1-560	74.04	1192	0.26	0.09
OpenAssistant/falcon-40b-sft-mix-1226	74.40	1053	0.44	0.13
huggyllama/llama-65b	67.24
OpenAssistant/oasst-sft-7e3-llama-30b	68.03	979	0.52	0.20
OpenAssistant/oasst-rlhf-3-llama-30b-5k-steps	68.51	1068	0.51	0.15
Table 1: Comparison of model evaluation scores on different LLM benchmarks: LMEH: lm-evaluation-harness [25] (average scores, see online leaderboard for more details) VEL: Vicuna Elo Rank [21] OAIE: OpenAI Evalu [26] HE: HumanEval [27] (for all benchmarks, higher is better). We have chosen to leave the Hugging Face Hub identifiers as the model names for identifiability.
Table  $\boxed{1}$  shows evaluation scores of a selection of baseline and trained models on a set of standard benchmark datasets. Evaluations were performed externally using FastEval and evaluation results are hosted on a leaderboard, which is continually being updated $^{6}$ . LMEH refers to the average performance on a set of widely-used NLU tasks consisting of BoolQ, PIQA, HellaSwag, WinoGrande, ARC-e, ARC-c and OBQA $^{7}$ . We omit instruction-centric experiments (VEL, OAIE, HE) for the base models, as these benchmarks are unsuitable for non-instruction-tuned models. The results show that models using OpenAssistant Conversations are consistently outperforming the corresponding baseline models (in the case of LLaMA even a larger baseline model). RLHF outperforms SFT in some benchmarks, but not in others. For Falcon-based models, sft-top1 is trained only on top-ranked conversation threads, whereas sft-mix mixes OpenAssistant Conversations with other instruction datasets (details in Appendix $^{8}$ ). The varied evaluation scores demonstrate that by combining different data sources, the nature of the resulting model can be readily influenced. Ranks across benchmarks are not consistent, which could indicate the unsuitability of automatic evaluations for language models, or could indicate that different models and datasets lead to different capabilities. The results also show that while open-source models are close to matching ChatGPT in some benchmarks, others still show large performance gaps. Anecdotally, users report OpenAssistant models to be less robotic and more human-sounding than commercial models and report generations to have high quality and diversity in domains such as creative writing, conversational messaging, and drafting social media posts.
5All trained models are released at https://huggingface.co/OpenAssistant https://github.com/FastEval/FastEval.https://tju01.github.io/ilm-eval/ For readability, Table contains aggregated LMEH scores. Details in the online leaderboard.
6.2 Spam and Toxicity
To understand the concordance between human and automated toxicity detection, we employ toxicity detection methods based on Detoxify [28] to obtain automated ratings for six distinct categories, classifying whether a message is toxic, obscene, threatening, insulting, attacking a certain identity or sexually explicit. We limit our analysis to those languages that are supported by the toxicity detection method, covering English, Spanish, Russian, French, and Italian. These languages represent the majority of OASST1 messages (over  $83\%$ ).
Using automated toxicity ratings, we are able to systematically assess the correlation between these ratings and human-assigned toxicity labels (hate speech, not appropriate, and sexual content). Based on a sample of 115,153 messages, we compute the correlation between automatic and human-annotated toxicity labels, which is visualized in Figure 3. We see a correlation between human and automatic labels in at least one element of each row and column of the correlation matrix, suggesting agreement between human annotators and off-the-shelf toxicity detection models. The results serve to both validate the capabilities and show limitations of AI-driven toxicity detection in comparison to human judgement and may inform future work in this area.
Figure 3: Correlation between human labels and Detoxify outputs for all messages in Detoxify-supported languages.
In addition to analysing the correlation between human-assigned toxicity labels and automated ratings, we extend the application of the Detoxify model to assess the efficacy of the moderation process for the same languages described earlier. To facilitate this analysis, we define two categories of messages: deleted messages, which encompass those that either failed to pass the community moderation process or were subsequently manually removed by moderators, and retained messages, which successfully made it through to the dataset. In order to provide a comprehensive evaluation of the moderation process, we calculated average values for each of the six Detoxify categories for both deleted and retained messages. The values obtained for this analysis are based on a sample of 74,781 messages. We excluded messages in trees that were incomplete at the time of export, as these messages may be subject to removal by the moderation process.
Our analysis, presented in Table 2 shows that the values for all six toxicity categories are markedly higher for deleted messages compared to retained messages. This significant difference demonstrates the effectiveness of the moderation processes in place, as messages removed from the dataset are on average rated as significantly more toxic by the Detoxify model than messages allowed to remain in the dataset.
State	Toxicity	Obscene	Threat	Insult	Identity Attack	Explicit	N
Deleted	4.625%	1.965%	0.411%	2.085%	0.651%	1.39%	3422
Retained	0.988%	0.574%	0.102%	0.715%	0.121%	0.177%	71359
Table 2: Detoxify outputs across six categories of toxicity, comparing deleted and retained messages.
While deleted messages are rated as more toxic than retained messages by the Detoxify model across all categories, the average toxicity values for these messages are still small. This implies toxicity ratings from models like Detoxify alone are not sufficient to determine when messages are unsuitable for inclusion in the dataset. Reasons for deleting non-toxic messages may include a lack of factual accuracy, or poor grammar. Additionally, messages which are children of deleted messages must themselves be deleted even if they appear to be acceptable in isolation.
Reward model data collection. InstructGPT [13] trained reward models on ranking data of messages generated by their initial SFT model, while our reward models are trained using ranking data of human-generated messages. We chose to do so because we were already collecting this ranking data as part of our general efforts, for use in quality control, spam filtering, and dataset sub-sampling. While subjectively, many users report our RLHF models to follow instructions more closely (also compare Section 6.1), the models do not deliver the same uniform and significant improvements over SFT models as reported in [13]. We hypothesize that the difference in data collection for the reward model could at least partially explain this gap. We plan to collect ranking data based on our own SFT models in the future to verify these assumptions. Further research is necessary to determine more precise criteria for collecting data useful to RLHF.
Subjective, Cultural, and Contribution Frequency Biases. The open nature of our project introduces a unique set of challenges when it comes to controlling for biases within the dataset. Annotators from diverse backgrounds contribute to the dataset, with demographics that are simultaneously heterogeneous in some dimensions and homogeneous in others (see Section 5). Specifically,  $89.1\%$  of the annotators identify as male, with a median age of 26. This demographic profile may inadvertently introduce biases in the dataset, as it is bound to reflect the values, perspectives, and interests of the annotators. (We pose that some of this could be mitigated by introducing more constrained conversations, for example sampling a random Wikipedia page to determine the conversation topic.) Further, users' participation levels differ significantly. More engaged users contribute a greater number of annotations (see Figure 2), which likely leads to over-representation of their values and interests in the dataset. Consequently, the dataset may not adequately capture the diverse perspectives that a more balanced distribution of contributions could have provided. Further research is necessary to determine the effect of uneven contributor distributions have when given a clear, general task.
Possibility of Unsafe Content. While we have implemented measures to detect and remove harmful messages, our system is not infallible. It is possible that the dataset still contains unsafe content. We believe that the open nature of the project allows for data filtering to be conducted in a transparent manner, ultimately converging on the highest possible standards. Nevertheless, the potential presence of residual unsafe content in the dataset necessitates careful evaluation of any models trained on it.
Given the limitations discussed above, we advocate for the use of our models in academic research contexts only. We strongly encourage researchers to thoroughly investigate the safety and bias of any model before employing it in downstream tasks. The released models may exhibit unsafe behavior and are likely susceptible to prompt injection attacks. The alignment of LLMs is a crucial aspect of AI research, and we hope that our contributions can help advance the field of AI alignment. However, we also acknowledge that current alignment techniques are not perfect and can even exacerbate certain biases [29]. As such, researchers should exercise caution when using these models and be cognizant of their limitations. Additionally, it is essential to continue refining alignment techniques and advancing the field of AI alignment in order to mitigate these limitations and develop more reliable and robust LLMs.
8 Safety and Ethical Implications
Large language models are prone to generating inaccurate information about people, places, or facts, commonly known as 'hallucinations' [30, 31]. LLMs can also produce toxic or hateful content and fail to follow provided constraints [32]. Additionally, these models tend to incorporate biases present in their training data, leading to unfair and discriminatory outputs [33]. While methods such as RLHF can mitigate some of these shortcomings, they may exacerbate others [34, 29]. We hope that alignment methods using OpenAssistant Conversations can fix some of these issues [13], but we acknowledge that achieving full alignment is a complex and ongoing challenge.
We recognize that sufficiently powerful language models can have a significant impact on society [35], and therefore we believe it is essential to promote transparency in their development and deployment. OpenAssistant Conversations is our contribution to this goal of transparency.
Our greatest thanks go to the many volunteer contributors, of human data, code, moderation, documentation, and community organization. Absent of any financial incentives, this project is a stunning and unprecedented display of global cooperation of humans for the purpose of advancing and democratizing AI research. In addition, several organizations have contributed to this project with resources: Redmond AI provided training compute. Stability AI and Hugging Face provided inference compute. We thank Olivier Dehaene at Hugging Face for close collaboration and personal support. Weights & Biases provided their full MLOps solution to the entire team. LAION provided legal input and acts as the website addressee. We thank Til Jasper Ullrich for running LLM benchmark evaluations, Luke Thomas Kaiser for running evaluations on model bias, and Silvia Pareti for detailed feedback on the manuscript. Agradecemos a Carlos Santana por la promocion a gran escala de la plataforma de recopilacion de datos para la comunidad hispanohablante.[8]
[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[2] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
[3] Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.01373, 2023.
[4] Jason Gabriel. Artificial intelligence, values, and alignment. *Minds and Machines*, 30(3):411-437, September 2020.
[5] Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: A survey, 2023.
[6] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.
[7] Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with  $\nu$ -usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 5988-6008. PMLR, 17-23 Jul 2022.
[8] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm,
${}^{8}$  Translated from English by OpenAssistant,first try
[35] Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359, 2021.
[36] Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, 2023.
[37] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022.
[38] T. N. Tideman. Independence of clones as a criterion for voting rules. Social Choice and Welfare, 4(3):185-206, September 1987.
[39] David Rozado. The political biases of chatgpt. Social Sciences, 12(3):148, Mar 2023.