ICML 2025

Unnatural Languages Are Not Bugs but Features for LLMs
Keyu Duan * 1 Yiran Zhao * 1 Zhili Feng 2 Jinjie Ni 1 Tianyu Pang 3 Qian Liu 3 Tianle Cai 4 Longxu Dou 3
Kenji Kawaguchi 1 Anirudh Goyal 5 J. Zico Kolter 2 Michael Qizhe Shieh 1
Abstract
Large Language Models (LLMs) have been ob-
served to process non-human-readable text se-
quences, such as jailbreak prompts, often viewed
as a bug for aligned LLMs. In this work, we
present a systematic investigation challenging
this perception, demonstrating that unnatural lan-
guages - strings that appear incomprehensible
to humans but maintain semantic meanings for
LLMs - contain latent features usable by mod-
els. Notably, unnatural languages possess latent
features that can be generalized across differ-
ent models and tasks during inference. Further-
more, models fine-tuned on unnatural versions
of instruction datasets perform on-par with those
trained on natural language, achieving 49.71 win
rates in Length-controlled AlpacaEval 2.0 in av-
erage across various base models. In addition,
through comprehensive analysis, we demonstrate
that LLMs process unnatural languages by filter-
ing noise and inferring contextual meaning from
filtered words. Our code is publicly available
at https://github.com/John-AI-Lab/
Unnatural_Language.
1. Introduction
Large Language Models (LLMs) (OpenAI, 2023; Touvron
et al., 2023; Dubey et al., 2024; anthropic, 2024) have shown
remarkable capabilities in understanding and generating
human-readable text, achieving impressive performance
across tasks, spanning from question answering (Bisk et al.,
2020; Ni et al., 2024) and mathematical reasoning (Cobbe
et al., 2021; Hendrycks et al., 2021; Gao et al., 2024) to
open-ended dialogue (Li et al., 2023). Such abilities are
*Equal contribution
1National University of Singapore
2Carnegie Mellon University 3Sea AI Lab 4Princeton Univer-
sity 5Mila, University of Montreal. Correspondence to: Keyu
Duan <k.duan@u.nus.edu>, Yiran Zhao <zhaoyiran@u.nus.edu>,
Mi...