Early External Safety Testing of OpenAI's o3-mini: Insights from the
  Pre-Deployment Evaluation
Authors: name: Aitor Arrieta, name: Miriam Ugarte, name: Pablo Valle, name: José Antonio Parejo, name: Sergio Segura

Large Language Models (LLMs) have become an integral part of our daily lives.
However, they impose certain risks, including those that can harm individuals'
privacy, perpetuate biases and spread misinformation. These risks highlight the
need for robust safety mechanisms, ethical guidelines, and thorough testing to
ensure their responsible deployment. Safety of LLMs is a key property that
needs to be thoroughly tested prior the model to be deployed and accessible to
the general users. This paper reports the external safety testing experience
conducted by researchers from Mondragon University and University of Seville on
OpenAI's new o3-mini LLM as part of OpenAI's early access for safety testing
program. In particular, we apply our tool, ASTRAL, to automatically and
systematically generate up to date unsafe test inputs (i.e., prompts) that
helps us test and assess different safety categories of LLMs. We automatically
generate and execute a total of 10,080 unsafe test input on a early o3-mini
beta version. After manually verifying the test cases classified as unsafe by
ASTRAL, we identify a total of 87 actual instances of unsafe LLM behavior. We
highlight key insights and findings uncovered during the pre-deployment
external testing phase of OpenAI's latest LLM.