UI-Level Evaluation of ALLaM 34B: Measuring an Arabic-Centric LLM via
  HUMAIN Chat
Authors: name: Omer Nacar
Published: 2025-08-24 14:32:15

Abstract: Large language models (LLMs) trained primarily on English corpora often
struggle to capture the linguistic and cultural nuances of Arabic. To address
this gap, the Saudi Data and AI Authority (SDAIA) introduced the ALLaM family
of Arabic-focused models. The most capable of these available to the public,
ALLaM-34B, was subsequently adopted by HUMAIN, who developed and deployed
HUMAIN Chat, a closed conversational web service built on this model. This
paper presents an expanded and refined UI-level evaluation of ALLaM-34B.
Using a prompt pack spanning modern standard Arabic, five regional dialects,
code-switching, factual knowledge, arithmetic and temporal reasoning, creative
generation, and adversarial safety, we collected 115 outputs (23 prompts times
5 runs) and scored each with three frontier LLM judges (GPT-5, Gemini 2.5 Pro,
Claude Sonnet-4). We compute category-level means with 95\% confidence
intervals, analyze score distributions, and visualize dialect-wise metric heat
maps. The updated analysis reveals consistently high performance on generation
and code-switching tasks (both averaging 4.92/5), alongside strong results in
MSA handling (4.74/5), solid reasoning ability (4.64/5), and improved dialect
fidelity (4.21/5). Safety-related prompts show stable, reliable performance of
(4.54/5). Taken together, these results position ALLaM-34B as a robust and
culturally grounded Arabic LLM, demonstrating both technical strength and
practical readiness for real-world deployment.