o3-mini vs DeepSeek-R1: Which One is Safer?
Authors: name: Aitor Arrieta; name: Miriam Ugarte; name: Pablo Valle; name: José Antonio Parejo; name: Sergio Segura
Published: 2025-01-30 15:45:56

Abstract: The irruption of DeepSeek-R1 constitutes a turning point for the AI industry
in general and the LLMs in particular. Its capabilities have demonstrated
outstanding performance in several tasks, including creative thinking, code
generation, maths and automated program repair, at apparently lower execution
cost. However, LLMs must adhere to an important qualitative property, i.e.,
their alignment with safety and human values. A clear competitor of DeepSeek-R1
is its American counterpart, OpenAI's o3-mini model, which is expected to set
high standards in terms of performance, safety and cost. In this paper we
conduct a systematic assessment of the safety level of both, DeepSeek-R1 (70b
version) and OpenAI's o3-mini (beta version). To this end, we make use of our
recently released automated safety testing tool, named ASTRAL. By leveraging
this tool, we automatically and systematically generate and execute a total of
1260 unsafe test inputs on both models. After conducting a semi-automated
assessment of the outcomes provided by both LLMs, the results indicate that
DeepSeek-R1 is highly unsafe as compared to OpenAI's o3-mini. Based on our
evaluation, DeepSeek-R1 answered unsafely to 11.98% of the executed prompts
whereas o3-mini only to 1.19%.