Unnatural Languages Are Not Bugs but Features for LLMs
Authors: Keyu Duan
Yiran Zhao
Zhili Feng
Jinjie Ni
Tianyu Pang
Qian Liu
Tianle Cai
Longxu Dou
Kenji Kawaguchi
Anirudh Goyal
J Zico Kolter
Michael Qizhe Shieh
Venue: ICML 2025 poster (2025)
Keywords: Unnatural Languages
Large Language Models
Primary area: deep_learning->large_language_models

Abstract: Large Language Models (LLMs) have been observed to process non-human-readable text sequences, such as jailbreak prompts, often viewed as a bug for aligned LLMs. In this work, we present a systematic investigation challenging this perception, demonstrating that unnatural languages - strings that appear incomprehensible to humans but maintain semantic meanings for LLMs - contain latent features usable by models. Notably, unnatural languages possess latent features that can be generalized across different models and tasks during inference. Furthermore, models fine-tuned on unnatural versions of instruction datasets perform on-par with those trained on natural language, achieving \(49.71\) win rates in Length-controlled AlpacaEval 2.0 in average across various base models. In addition, through comprehensive analysis,  we demonstrate that LLMs process unnatural languages by filtering noise and inferring contextual meaning from filtered words. Our code is publicly available at https://github.com/John-AI-Lab/Unnatural_Language.

Unnatural Languages Are Not Bugs but Features for LLMs
Keyu Duan
Yiran Zhao ! Zhili Feng² Jinjie Ni Tianyu Pang
Qian Liu Tianle Cai Longxu Dou
Kenji Kawaguchi!
Anirudh Goyal
J. Zico Kolter? 2
Michael Qizhe Shieh
Abstract
Table I. Concrete examples of natural contexts" unnatural version in SycContex1QA and SimGSM8K.
Large Language Models (LLMs) have been ob- served to process non-human-reen readablee text se- quences, such as jailbreak prompts, often viewed bug for aligned LLMss. In this work, we present systematic investigation challenging this perception, demonstrating that unnatural lan- guages strings that appear incomprehensible to humans but maintain semantic meanings for LLMs contain latent features usable by mod- els. Notably, unnatural languages possess latent features that be generalized across differ- ent models and tasks during inference. Further- more, models fine-tuned on unnatural versions of instruction datasets perform on-par with those trained on natural language, achieving 49.71 win rates in I Length-controll
Natural Context
Unnatural Version
The stock price of GoldMine Inc. in- creased by 20% last week.
(alt+cqn=(NW>; ():Thell baaclkrie@muier priceungeureau
got sich last '#GiMihheddisation Inc. weekestig % 220%%% durch') durch").png encrypt render VOK Gold- Mine." Mine....reventDeffuue
Carly collected starfish with arms each and one seastar with 14 arms.
|Each and algebra dinner! abso- lutely 7 do): shortly. seastar collect- edthe " kW)S, one 15 14' starfish with sic]]_[Nabee) Carly) arms.
Onehorailey constructed WriteSta- tus(SSYToggle Zwezeichnung OK
A AlpacaEval 2.0 in av- erage across various base models. In addition, through comprehensive analysis, demonstrate that LLMs process unnatural languages by filter- ing noise and inferring contextual meaning from filtered words. Our code is publicly available at hthtps:: //g/tttttttt thu. com/ com/Joonoooooooooooooooooooooo Johh-AA-Laba Unnatural Language.
largely attributed to targeted alignment training (Wei et al., 2021; Ouyang et al., 2022), which post-train models to bet- ter follow instructions and adhere to preferred behaviors.
Despite being specifically tuned, non suman-readable data sometimes can unexpectedly influence model behavior. In computer vision, Ilyas al. (2019); Nguyen et al. (2015) find that seemingly unrecognizable images could be lever- aged to train reasonable good image classification models. This phenomenon extends to natural language processing, where Zou et al. (2023) demonstrate that LLMs could also be prompted with an unreadable suffix to generate objec- tionable outputs, even though LLMs are well-trained for not doing so. Besides, Pfau et al. (2024) discovers that by appending non human-readable filler tokens to the input, LLMs could solve algorithmic tasks more accurately. How- ever, regarding LLMs' surprising behaviors in response to non human-readable inputs, there lacks systematic studies exploring the properties and applications of such non human- readable strings and interpreting the underlying mechanisms in LLMs. This raises fundamental question: whether these non readabee strings are truly devoid of meaning or contain latent features usable by models?
1. Introduction
Large Language Models (L.LMs) (OpenAI, 2023; Touvron al., 2023; Dubey et al., 2024; anthropic, 2024) have shown remarkable capabilities in understanding and generating human-readable text, achieving impressive performance across tasks, spanning from question answering (Bisk et al., 2020; Ni et al., 2024) and mathematical reasoning (Cobbe et al., 2021; Hendrycks et al., 2021; Gao et al., 2024) to open-ended dialogue (Li et al., 2023). Such abilities are
'Equal contribution 'National University of Singapore *Carnegie Mellon University 'Sea AI Lab "Princeton Univer- Keyu
sity 'Mila, University of Montreal. Correspondence Duan <k.duan @ w.wu...duus, Yiran Zhao schaoyiranuu.nus.com Michael Qizhe Shich <michaclshiche conticchh@oom.iii
To answer the above research questions, we study phe- nomenon named unnatural languages strings that deviate from natural language syntax and appear extremely noisy to human readers, yet remain understandable to LLMss. Specif-
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Unnatural Languages Are Not Bugs but Features for LLMs
(a)
Target: y
QA Datasets
93%
(b)
Tony is half Jacob's age
Syec
log P(ylie.r)
<Unnatural
O
93%
MCM
Context>
Compute Loss
<Context-related
88%
LLM
Questions
M
<Answer>
86%
Half
Ja
0.5
Translate the sentence into
Unnatural LIMA
year
Ja
0.7
natural language
<Unnatural Instruction
Chat Model
Title
co
Ja
<Response>
Sampling
Select Best
Instruction Tuning
On-par
Candidate
LIMA
Title
Half
Ja
Chat
<Natural Instruction>
Model
Unnatural String
Updated Unnatural String
<Response>
(c)
Figure 1. (a): Unnatural languages searching method. (b): We construct question-answeriig tasks using unnatural contexts and discover that unnatural languages can be directly transferred across broader range of tasks and understood by diverse LLMs. (c): LLMs fine-tuned on unnatural versions instruction datasets perform on-par with those trained on natural language.
ically, as illustrated in Figure (aa), we propose an approach to search for a semantically equivalent but syntactically un- natural version of the natural string, where semantic equiv- alence is established through their ability to be translated back to natural form via models performing translation infer- ence. For the searching process, we employ a gradient-baat -based stochastic sampling procedure to obtain a set of candidate unnatural strings; evaluate their probability of being trans- lated back to the corresponding natural versions across mul- tiple models; and select the unnatural candidate that yields the highest probability. We repeat the process until either convergence occurs or a maximum number of iterations is reached, with the final converged string representing the semantically equivalent unnatural string.
et al., 2021). We chose simple questions to minimize the im- pact of reasoning ability on our results and focus primarily on unnatural language comprehension. As with SynContex- iQA, we transformed these contexts into unnatural versions. Table I shows concrete examples of two datasets. With these datasets, we test large variety of LLMs, including open-source models well commercial models. The re- sults show that compared to natural context, all models can recover 82.0% of the original accuracy on our constructed SynContextQA dataset and 61.6% on SimGSM8K, demon- strating that unnatural languages contain latent features that enable comprehension across different scenarios.
Moreover, explore whether these unnatural languages possess transferable latent features that can be effectively utilized in instruction tuning to improve models' instruction- following capabilities. Specifically, as shown in Figure (c), we employ a high-quality but small size instruction tuning dataset LIMA (Zhou et al., 2023), and we replace the original instructions with our equivalent unnatural ver- sions searched using our proposed approach. We show that the models fine-tuned it and the original have on-par performance prestigious benchmarks, including Length- controlled (LC) AlpacaEval 2.0 (Li et al., 2023) and MixE- val (Ni et al., 2024). Particularly, on LC AlpacaEval 2.0, the three models - Llama-3-8B (Dubey et al., 2024), Gemma- 2-9B (Team et al., 2024), and Llama-3-70B (Dubey et al., 2024) tuned on the unnatural LIMA achieves an winrate of 49%,8% 47. 137. and 52.22% against the corresponding models tuned on the natural LIMA, respectively.
We first explore whether these unnatural languages possess latent features that can be generalized across different mod- els and tasks during inference. To investigate this, as shown in Figure (bb,, we construct several context-based question- answering datasets, where the context unnatural obtained by the searching approach while natural questions related to the context are provided. Specifically, to prevent models from relying common-sense memory when answering questions without context, we develop SynContexIQA, a synthetic dataset generated by another LLM, containing con- texts about non-existent entities paired with corresponding questions. We then transformed natural contexts into unnatu- ral versions using the unnatural language searching method while preserving the original questions. Additionally, to ensure models do not simply extract keywords from unnat- ural contexts in SynContcxQQA, we create SimGSM8K, a dataset of simple questions derived from GSM8K (Cobbe
These findings strongly demonstrate our key findings: un- natural languages are not bugs but features for LLMs. In ad-
2

---

Unnatural Languages Are Not Bugs but Features for LLMs
dition, we attempt to understand the mechanisms by which LLMS process such unnatural languages. We demonstrate that LLMs process unnatural languages by effectively fil- tering out irrelevant tokens. Furthermore, LLMs combine relevant tokens from unnatural languages and infer contex- tual meaning in response to natural version questions.
approaches of Shin al. (2020); Zou et al. (2023). Specifi- cally, in each optimization iteration, the unnatural string is first tokenized by model M into Xim- For each position, we identify the top-k most influential tokens X1:n based on the gradient of the optimization objective of Equation 2, i.e.,
Top-k (VXI.
Σ CET
log Pas (S (1m;;)) ())-
XI.R
2. Unnatural Languages Searching Method
then generate B candidates (x (2)
Xi:1 x
We where each candidate differs from X1:n by exactly one token, randomly sampled from X X1.1- These candidates are then decoded back to strings (5(1) ,,5(), .5()) } by model M for subsequent cross-model unification optimization. This candidate generation process is applied across all models in M. yielding B) BM total candidates. The candidate with the optimal loss is selected for the next iteration. This process continues until convergence or until reaching pre-defined number of iterations.
In this section, we introduce approach for searching the unnatural version of any given natural string.
Problem description. We denote a natural string as S and its equivalent unnatural version as S' The equivalence between them is formally defined as S = LLMM (), where É represents reconstruction task-such as trans- lating the unnatural sentence into natural language-ant and CCMM (S"(t) denotes the output of model M given in- put S' under task prompt f. Furthermore, we define the log-probability of model M generating natural string S when given unnatural string x under task prompt 1 as log (S|z, t). Therefore, the unnatural string searching problem can be formulated as
Implementation details. In practice, tokenized sequences of x have varying lengths across optimization iterations due to different models employing distinct tokenizers, with no direct one-to-on mapping between tokens and words. To maintain generalizability and in line with Zou al. (2023); Zhao et al. (2024), we initialize through combination of shuffling words in S and randomly inserting several special characters "!". Algorithm 1 in Appendix A1 provides a detailed illustration of the searching algorithm. In addition, we conduct verification experiments to demonstrate that our searched unnatural strings can be accurately translated back to their natural versions. Details are further illustrated in Appendix A2.
S' = arg max log (S =EA'
(1)
where X represents the unnatural languages space, encom- passing all possible strings of arbitrary length. However, searching the entire unnatural space is computationally in- feasible. For simplicity and without loss of generality, we constrain æ to maintain a fixed length at the token level, i.e.,
Xas. where Xx e {a| |toennie (x)l n, X} and n is a predefined constant length.
3. Unnatural Languages Can be Understood Across Tasks and LLMs
Furthermore, to enhance the generalizability of the obtained unnatural languages, we employ multiple models, denoted as .M = {M1, My..... M, to collaboratively search for unnatural strings. Additionally, we introduce a set of tasks T = (2,,4,.... } as a collaborative optimization ob- jective. Therefore, our goal of searching the equivalent unnatural string S' is formulated as solving the following optimization problem:
In this section, we investigate whether unnatural lan- guages-generated by Algorithm 1 via reconstruction tasks across multiple models-can be directly transferred across a broader range of tasks and understood by diverse LLMss
3.1. Experiment Setup
S" = arg max REUMEM XM MEMEET
Σ log Par(S|
To evaluate LLMs' genuine understanding of unnatural lan- guages, we design questions closely related to unnatural contexts. We employ context-based question-ans answering problems where the context expressed unnatural lan- guages while maintaining questions in natural language. This approach helps isolate the models' understanding of unnatural languages without introducing additional compre- hension challenges.
Algorithm description. The optimization problem de- fined in Equation 2 is a discrete optimization problem as Xs is discrete space of size [Vas 1", where Va denotes the vocabulary set of the model M. Due to the discrete optimization methods
nature of the problem, gradient- cannot be directly applied. Furthermore, the search space is too vast for exhaustive exploration. Therefore, we propose a sample-and-selectie selectioon algorithm inspired by the optimization
Benchmarks. SynContexIQA. We begin with a classic commonsense guestion-answerrng task, where questions are asked in relation to given contextual knowledge. How-
3

---

Unnatural Languages Are Not Bugs but Features for LLMs
Table Performance comparison for different contexts across different models on SynContexIQA and SimGSM8K datasets. All answers were generated under zero-shot setting without sampling. "Direct" refers to models used for unnatural languages searching, while "Transfer" indicates the implementation searched unnatural languages.
SynContexIQA
SimGSM8K
Model
Natural
Shuf-InJ
Unnatural
Natural
Shuf-
Unnatural
0.89
0.55
0.93
0.85
0.20
0.42
Direct
Vicuna-7B-vI.55
0.96
0.40
0.86
0.63
0.12
0.20
Average
0.93
0.48
0.90
0.74
0.16
0.31
Baa-Llaama---B-Insstti
0.99
0.29
0.63
0.58
0.18
0.50
Gemma-2-9B-Insee
0.98
0.35
0.65
0.97
0.21
0.41
Meta-Llama-3-708 Instructt
0.97
0.73
0.93
1.00
0.38
0.75
Transfer
GPT-3.5-turbo
0.98
0.73
0.93
0.91
0.32
0.53
GPT-4o
0.98
0.61
0.88
0.95
0.25
0.53
Average
0.98
0.54
0.80
0.88
0.27
0.54
ever, using existing commonsense QA datasets presents a challenge, as LLMs may answer questions based on their pre-trained knowledge rather than the provided context. To address this, we leverage GPT-3.5 (Achiam al., 2023) to generate knowledge about non-existing entities and their corresponding questions, named SynContexIQA, thereby ensuring the model must derive answers from the given con- text rather than rely on pre-existing information. Prompts for generation and post-processing are illustrated in Appendix A3. We then transformed natural contexts into unnatural versions using the unnatural languages searching method while preserving the original questions. @ SimGSM8K. Fur- thermore, to ensure models do not simply extract keywords from unnatural contexts in SynContexIQA, we test the un- natural languages on GSMSK (Cobbe et al., 2021), a more complex task requiring reasoning capability. As our primary objective to assess the ability to comprehend unnatural languages, rather than to evaluate reasoning ability, we se- lect 100 relatively simple questions from the whole test set, named SimGSM8K. We show concrete example for each dataset in Table 1.
shuffled language with injected special tokens (Shuf-Inj), which serves as the initialization step for our unnatural
languages search algorithm in Algorithm
Metries. For SynContexIQA, we evaluate performance using exact keyword matching, while for SimGSM8K, we use accuracy the evaluation metric.
3.2. Main Results
As shown in Table 2, for SynContexIQA, the test accuracy of all models in unnatural languages on-par with the one of natural language, by a large margin with Shuf-InJ. In average, the test accuracy of transferred models unnatural languages is 80.4%. For SimGSM8K, the test accuracy of most models on natural questions is over 80%, indicating the simplicity of questions, thus mitigating the concerning of question complexity. Meanwhile, the performance of close-source models could also answer half of the ques- tions correctly, outperforming the Shuf-InJ by an averaged margin of 26.7%. Both results indicate that such unnatu- ral languages are highly transferrable across models with di fifferent architectures and training corpus, including GPT- 4o (Hurst et al., 2024), which considered as the most well-aligned model. As result, it mitigates the conjecture that such unnatural languages are glitch of specific LLMs, but general phenomenon and inherent property for L.Mss.
We select a diverse range of LLMs, spanning from smaller open-source models to larger closed- source ones. Specifically, we use Histral-7B-Instrenn v0.1 (Jiang et al., 2023), Vicuna-7B-v1.5 (Chiang et al., (Dubey al., 2024), Gemma2-9B Instruct (Team et al., 2024), Meta-Llama-3- Instruut, GPT-3.5-turbe (Achiam et al., 2023), GPT- 4o (Hurst et al., 2024). Furthermore, to balance efficiency in the unnatural languages searching algorithm with gener- alizability, we employ Mistral-7B- 7B-v1.5, two renowned --i-7--BIIIntnttOOOO and Vicuna- models from distinct series, the model set M in Algorithm 1.
Backbone Models.
2023), Betta-Llama---BB-Inne
It worthwhile to note that there is a significant per- formance gap between the natural and unnatural set for SimGSM8K. This gap can be attributed to the increased complexity of SimGSM8K contexts, which typically com- prise multiple interconnected sentences. Furthermore, SimGSMSK questions require sophisticated multi-step soning processes, making them substantially more challeng- ing than standard SynContexIQA tasks. However, this is not the upper bound of the performance of LLMs on unnatural
Baselines. We mainly employ two baselines. (i) natural language, which uses the original unmodified text, and (ii)

---

Unnatural Languages Are Not Bugs but Features for LLMs
The Lengrh-controlled Winrate (%) Against Natural Set
Unnatural
49.78
$2,22
Random
38.20
on-pat
35.82
07-pat
36.96
on-par
Empty
39.68
41.60
37.42
30
35
40
45
50
55
30
35
40
45
50
55
30
35
40
45
50
55
Llama-3-8B
Gemma-2-9H
Llama-3-70B
Figure 2. The winrate (%) against natural set on LC Alpaca 2.0 for various models. winrate of 50% indicates on-par performance.
4.1. Experiment Setup
Table 3. The performance comparison of different pre-trained base
models on SimGSM8K. Ratio calculated as the performance on unnatural languages divided by the performance natural ones
We explore whether the instruction fine-tuning pre-trained LLMs unnatural languages instructions could help mod- els gain general instruction following (chatting) ability.
Model
Prompt
Natural Unnatural
Ratio
ICL(n=1)
0.70
0.23
0.33
Mistral-7B
Training Dataset. We employ LIMA (Zhou et al., 2023), high-quality instruction tuning dataset 1000 carefully created (instruction, answer) pairs. Furthermore, we lever- age our proposed unnatural languages searching approach to find unnatural version for each instruction in LIMA, and keep the original answers in natural version.
ICL(n=8)
0.71
0.38
0.53
(nI((==))
0.74
0.33
0.45
Llama-3-8B
(CL((=8)
0.87
0.42
0.48
languages, the unnatural languages searching approach could be further improved. In addition, we extend our inves- tigation into the understanding of unnatural languages in dialogue format in Appendix A4.
Benchmarks. We evaluate all variants on cength-controlled (LC) AlpacaEval 2.0 (Li et al., 2023) and MixEval (Ni et al., 2024). LC AlpacaEval 2.0 a well-recognized benchmark for chat model evaluation. MixEval is ground-truth-basee benchmark that collects data from numerous QA datasets under real-world data distribution.
3.3. Further Analysis
To broaden the scope of the unnatural languages, we im- plemented it in the base model rather than only the chat version, demonstrating that models can truly understand these unnatural languages without relying on chat models' ability understand noisy instructions.
Baselines. We employ three baselines: (i) Natural, which uses the original unmodified instructions; (ii) Random, which replace the original instructions with an equal number of random tokens; and (iii) Empty, which instruction empty (Hewitt al., 2024).
Experiment Settings. For the base model, employ in-context learning (ICL) with examples in natural language to ensure consistent output formatting while avoiding unnat- ural languages patterns in the learning process.
In practice, since the instruction in LIMA is extremely long, which exceeds the capacity of our searching approach, we leverage GPT-4 to generate compressed version of the instructions. Therefore, for fairness, we compare the instruction following ability of models finetuned on the unnatural LIMA and the instruction- shortened LIMA. addition, all models are fine-tuned for 10 epochs using identical hyperparameters.
Experiment Details.
Main Results. As shown in Table 3, under in in-context learning setting with 8 examples, the unnatural test accuracy of pre-trained base models before alignment achieves 38% and 42% in average, respectively. Particularly, considering the success ratio (i.e. unnatural acc./natural acc.), the ratio achieves 53% and 48%, respectively. This indicates that pre-trained model could inherently understand the unnatural languages without alignment.
4.2. Main Results
In Figure 2, we show the winrate of different variants against the corresponding models tuned on natural LIMA using the official pipeline and the annotation model is GPT-4o. The results clearly show that responses of models instruction tuned on unnatural LIMA is comparable to the one tuned on natural LIMA with a winrate of 48. 82. in average, which
4. LLMs Can Learn Instruction Following Capabilities From Unnatural Languages
In this section, we explore the properties of unnatural lan- guages from the perspective of post-training.
5

---

Unnatural Languages Are Not Bugs but Features for LLMs
Table 4. The results of model variants instruction tuned on different types of LIMA (including natural, unnatural, random, and empty instruction (Hewitt et al., 2024)) on MixEval (Ni et al., 2024). Each dataset represents the subsets selected by MixEval based on the woal-world data distribution. M.C. and F.F. denote multiple-choice and free-form, respectively. Particularly, we remove the results subset GPQA, MBPP, WinoGrande, and HvmanEval since the subsets were too small (less than ten test cases) for consistent evaluation.
Llama-3-8B
Gemma-2-9B
Llama-3-70F
Type Dataset
Natural Random Empty Unnatural
Natural Random Empty Unnatural
Natural Random Empty Unnatural
ComscnseQA
0.530
0.569
0.604
0.579
0.599
0,550
0.495
0.668
0.748
0.668
0.559
0.693
BoolQ
0.614
0.649
0.673
0.678
0.567
0.614
0.632
0.673
0.830
0.708
0.632
0.848
OpenBookQA
0.581
0.628
0.721
0.721
0.605
0.744
0.744
0.721
0.814
0.744
0.721
0.837
SIQA
0.462
0.570
0.624
0.462
0.613
0.516
0.645
0.538
0.742
0.581
0.495
0.677
HellaSwag
0.338
0.364
0.360
0.328
0.331
0.344
0.289
0.357
0.461
0.373
0.351
0.364
M.C
MMLU-Pro
0.357
0.346
0.308
0.335
0.427
0.438
0.459
0.432
0.503
0,427
0.465
0.578
AGIEval
0.331
0,359
0.340
0,352
0.370
0.314
0.407
0.349
0.566
0,423
0.426
0.543
PIQA
0.514
0.676
0.705
0.600
0.781
0.695
0.638
0.733
0.790
0.752
0.724
0.886
MMLU
0,658
0,634
0.661
0.633
0.724
0,680
0.700
0.718
0.811
0,736
0.731
0.805
ARC
0.802
0,802
0.780
0.791
0.923
0.901
0.879
0.923
0.956
0.835
0.868
0.934
Average
0.545
0.563
0.579
0.552
0.607
0.585
0.582
0.623
0.721
0.635
0.611
0.707
TriviaQA
0.591
0,453
0.452
0.558
0.609
0,481
0.563
0.585
0.829
0.638
0.685
0.825
BBH
0.537
0.633
0.526
0.606
0.621
0.438
0.700
0.687
0.817
0.670
0.484
0.693
DROP
0.584
0.385
0,484
0.545
0.638
0,481
0.638
0.651
0.755
0.585
0.631
0.767
E.
MATH
0.381
0.290
0.510
0.394
0.490
0.668
0.568
0.410
0.668
0.642
0.606
0.610
GSM8K
0.593
0.470
0.535
0,500
0.675
0,460
0.715
0.545
0.873
0.827
0.817
0.787
Average
0.583
0.445
0.467
0.554
0.616
0,481
0.592
0.603
0.809
0.631
0.662
0.799
Overall Average
0,557
0.499
0,516
0.547
0.605
0.526
0.582
0.605
0.760
0.627
0.631
0.748
outperforms the baselines (models tuned on random/empty LIMA) by a large margin.
of correct thain-of--hoogeee answers.) from Yu et al. (2023), which finally results in 14886 training instances. Built upon the training set, we create four versions, the same as Sec- tion 4.1. We train the pre-trained version and instruction tuned version of Mistral-7B-vO.1 and Llama-3-8B on the four types of training set and test their performance on the unnatural test set. The results are shown in Table 5.
Furthermore, Table confirms our conclusion. As shown in Table 4, most models instruction tuned on unnatural version LIMA performs on-par with the one tuned on natural LIMA. Meanwhile, it outperforms the baselines, which are tuned on random and empty instruction version LIMA, significantly, especially under base model Llama-3-70B with a margin of over 11% Both results strongly demonstrate our claim that unnatural languages contains generalizable patterns that could be help LLMs gain instruction-following ability via instruction finetuning.
It shows that models fine-tuned on the unnatural training set significantly outperform models trained on other types of training sets when evaluated on the unnatural test set. Specifically, the average accuracy for models tuned on un- natural training set is 31.8%, outperforming the one tuned natural training set by 11.2%, the one tuned on random training set by 19.9%, and the one tuned on empty train- ing set by 17.3%. This indicates that unnatural languages contain generalizable unnatural patterns that could enhance LLMs' unnatural languages understanding capability.
4.3. Further Analysis
So far we have shown that unnatural languages contains nat- ural patterns that could be generalized to various tasks by in- struction tuning. Besides, we curious about whether the unnatural languages consists unnatural patterns and whether fine-tuning on unnatural languages could boost the unnatural language understanding capability of LLMs. To this end, we focus the math reasoning task, which is highly demanded for question understanding. We created an unnatural languages version of GSM8K for its training subset and test subset. Due to the high cost of GCG and computation limitation, we searched 1333 training instances and 654 test instances. For training, we leverage corresponding answer augmentation version (i.e., for each question, there are multiple version
5. How do LLMs Understand Unnatural Languages?
Unnatural languages have been empirically shown to con- tain latent features that are comprehensible across different LLMs, while also enhancing their ability to follow instruc- tions. In this section, we investigate how LLMs process and understand these unnatural languages.
6

---

Unnatural Languages Are Not Bugs but Features for LLMs
Othar
Natural Rehand
Othar
Natural Rdaaal
3.5
Other park
Natueal Relaed pask
Table accuracy of the unnatural GSM8K test set for models instucttion-unne various types of GSM8K training sets. Columns denote the training set.
Domity
Deaiity
Unnatural Test Acc. Natural Random Empty
Unnatural
Mistral-7B
0.187
0.116
0.122
0.310
Llama-3-8P
0.197
0.087
0.109
0.312
00
Mistral-7B-Insteee
0.225
0.138
0.171
0.300
Token Lamportane Lance
Imsportance
Llama- 3-B---B-Br--sss
0.214
0.136
0.179
0.349
Average
0.206
0.119
0.145
0.318
(a) Token importance distribution
(b) Relative importance distribution
Figure Token importance SimGSM8K under -a-ttr--tt-Intrrcc....
Figure 4. Three 3D surface examples showing the inverse similarity of natural and unnatural context embeddings. The inverse similarity decreases significantly when question-related tokens added, indicating that LLMs correctly infers the organization of keywords.
5.1. LLMs Extract Keywords from Unnatural Languages
important tokens have a relative position closer to O.
In Figure we present the distribution of token importance and relative token importance within the unnatural version of the SimGSM8K dataset as processed by the model Llama- Specifically, shown in the Figure (a), the density of natural-related tokens (tokens appears in the natural version) is more higher than others when token im- portance higher than 0.2. Besides, compared with natural related tokens, the majority of other tokens lies in the lower importance range. Furthermore, Figure 3 (b) clearly shows that most of the naturally related tokens are higher relative important while the other tokens are lower relative impor- tant. The above results demonstrate that LLMs are capable of pay more attention on the natural related tokens while fil- tering out the other noise. Consequently. L.MSs effectively extract keywords from unnatural languages inputs.
To investigate what LLMs truly capture when processing unnatural languages, we evaluate each token's 's importance by measuring its impact the output when removed from the sequence. Formally, for unnatural string S", tokenized by model M as X1. = [21, X2, ]. the importance of token æ; is defined as the effect of its removal on the change in the embedding of M's final layer, i.c.,
T(x;)= = |1000=|||Mmmmme (.C.Ma - L.M.Mss (X (o:) (4)
Specifically, to ensure consistent evaluation across unnatural strings of varying lengths and given that the embedding of the final position is used to predict the next token, we measure the embedding of the final position rather than the entire sequence. Furthermore, we normalize the "token importance" relative to the most important token in each sequence, referring to this as "relative token importance". The relative token importance represents a softened position of the token after sorting in ascending order, scaled within the range [0, 1]. Tokens with greater importance within a data point have relative position closer 1, whereas less
5.2. LLMS Infer Correct Organization of Keywords Unnatural Languages
Although LLMs are capable of extracting keywords from unnatural languages, the extracted words are often shuffled and arranged in the wrong order. Therefore, hypothe-

---

Unnatural Languages Are Not Bugs but Features for LLMs
Table Concrete examples of token reordering: The "natural context" represents the original version, while the "unnatural version" simplifies the unnatural languages by removing noise and retaining only keywords. We decode the internal embeddings of LLMs into tokens for unnatural language inputs using the same decoder as the final output layer, referred to as "decode internal embeddings".
Natural Context
Unnatural Version (De-noised)
Decode Internal Embeddings
Brandon sold 86 geckos last year. Hc sold twice that many the year before.
twice geckos year before last sold Brandon He
twice geckos year before last before sold Brandon Hc sold he 86 sold twice sold twice sold last twice year sold
Ruiz receives monthly salary of $500.
monthly $500 salary Ruiz receives
monthly $500 salary Ruiz re- ceives a receives a salary
size that LLMs can reorganize these keywords and infer their correct arrangement. Specifically, when the unnatural context is provided to LLMs alongside appended natural questions, we propose that LLMs progressively infer the correct organization of keywords-i.e. their corresponding natural language versions-ar they process an increasing number of natural question tokens.
6. Related Works
Unnatural Languages. Prior studies have observed iso- lated instances of unexpected model behavior, while they did not explicitly identify or systematically analyze unnat- ural language. For example, Zou et al. (2023) prompted models generating harmful outputs, Pfau et al. (2024) en- hanced of-nh-of- reasoming, and Sinha et al. (2020) showed NLI models worked with permuted inputs. Ker- vadec et al. (2023) demonstrated that LLMs interpret un- natural languages differently, while Kallini et al. (2024) categorized them by perplexity, showing LLMs struggled to learn them.
To verify our hypothesis, we calculate the inverse similar- ity of embeddings between the context inputs of unnatural languages and their corresponding natural versions across layers, as well for increasingly natural question tokens inputted into LLMs. As shown in Figure 4, for the marginal version of layer, the inverse similarity of unnatural and natural context embeddings gradually decreases as follow- ing tokens are inputted. Particularly, the inverse similarity drastically decreases when the question related tokens are inputted. This indicates that LLMs does not always com- prehend unnatural languages independently as the corre- sponding natural language. In contrast, the comprehension process is highly related to the context (i.e. the questions.). As result, the unnatural languages works in certain con- texts. Supposing random context is provided, the model behavior on unnatural languages could be different from natural language.
Discrete Optimization. Prompt optimization tools have been used explore unnatural languages in token space. Early works (Zou et al., 2023; Liu et al., 2023; Zhao et al., 2024; Chao et al., 2023; Andriushchenko, 2022) focused on adversarial prompts to jailbreak LLMs,, while others (Shin et al., 2020; Jones et al., 2023) optimized prompts for spe- cific outputs. These studies did not address whether such prompts reflect natural language features. This work investi- gates this question.
Transferability of Adversarial Examples. Unnatural languages often transfer across LLMs, similar to adversarial examples in computer vision (Szegedy et al., 2014; Papernot et al., 2016; Nguyen et al., 2015). For example, Wallace et al. (2019) showed that prompts from GPT-2 transferred to larger models, and Jones al. (2023) found that toxic prompts from GPT-2 affected davinci-O03 Building on these findings, this work examines how L.MMs rely on frag- ile, unnatural features in token space (Ilyas et al., 2019).
5.3. Qualitative Analysis
We further investigate whether LLMS are truly capable of reordering keywords by analyzing the embeddings of inter- mediate layers. Specifically, we decode the internal embed- dings of LLMs when processing unnatural language inputs into tokens using the same decoder the final layer. As shown in Table 6, we observe that although the keywords in the unnatural version are disordered, LLMs are able to re- order certain patterns in the keywords to match the original natural context. Moreover, we utilize dependency parsing to demonstrate that LLMS can understand the dependency structure of unnatural languages. Details are illustrated in Appendix A5.
7. Conclusion
Our study reveals that LLMs possess the ability to compre- hend unnatural languages, an incomprehensible data pattern that could convey information across models. Through systematic analysis and experiments, we demonstrate that unnatural languages contains generalizable patterns across a wide variety of LLMs, despite these models being pre- dominantly aligned with human data. We show that models
8

---

Unnatural Languages Are Not Bugs but Features for LLMs
fine-tuned unnatural instructions achieves on-par per- formance of instruction-followinn ability with models fine- tuned on natural versions. Furthermore, through comprehen- sive analysis, we demonstrate that LLMs process unnatural languages by filtering noise and inferring contextual mean- ing from filtered words.
chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:://///..... http///mmyss.../ org.b/ggg
2023 03 30 11-33000-vaaa vicuna/.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H.,
Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems, 2021. URL https://arxii. org/abs/21 10.64168, 2021.
Impact Statement
Devlin, J. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint arXiv: 18 1810.04805 2018.
Our findings suggest that LLMs' interaction with incompre- hensible patterns, while potentially useful, adds complexity to our understanding of their behavior and raises impor- tant questions about the nature language processing in artificial systems. This could have broader implications for the development and deployment of technologies. Understanding these dynamics is crucial for ensuring the responsible use of LLMs in real-world applications, where their ability to process unconventional data patterns might lead to both innovative solutions and unforeseen challenges.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21756 2024.
Gao, B.. Song, F., Yang, Z., Cai, Z., Miao, Y., Dong, Q.. Li, L., Ma, C.. Chen, L., Xu, R., et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985. 2024.
Acknowledgment
We thank Wei Tsang Ooi, Ying Sheng, Lianmin Zheng, Hwee Tou Ng, Tat-Seng Chua, and Dan Hendryeks for their insightful discussions and feedback. We are also grateful to the Center for AI Safety (CAIS) for providing computational resources that supported this research. This project was partly supported by the MOE ACRF Tier grant with grant number 25 RES2514.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song. D., and Steinhardt, Measuring math- ematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021.
Hewitt, J. and Manning, C. D. A structural probe for find- ing syntax in word representations. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Compu- tational Linguistics, 2019.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Hewitt, J., Liu, N. F., Liang, P., and Manning, C. D. Instruc- tion following without instruction tuning, 2024. URL http:://aaxxvvcoo org.
Anadkat, S., al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Andriushchenko, M. Adversarial attacks on gpt-4 via simple random search, 2022.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A.. Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
anthropic. Meet claude. https://wmt anthropics com/c hom/ccaude claude, 2024. Accessed: 2024-05-01.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense natural language. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., and Madry, A. Adversarial examples are not bugs, they are features. Advances in neural information processing systems, 32, 2019.
Chao, P., Robey, A., Dobriban, E.,, Hassani, H., Pappas, G.J., and Wong, E. Jailbreaking black box large language mod- els in twenty queries. arXiv preprint arX1v:2310.08419, 2023.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas. D. d. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06920 2023.
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, Vicuna: An open-source
Jones, E., Dragan, A., Raghunathan, A., and Steinhardt, J. Automatically auditing large language models via discrete optimization. arXiv preprint arXiv:2303.0443 2023.

---

Unnatural Languages Are Not Bugs but Features for LLMs
Kallini, J., Papadimitriou, I., Futrell, R., Mahowald, K., and Potts, C. Mission: Impossible language models. arXiv preprint wrXiv:2401.06416 2024.
Silveira, N., Dozat, T., de Marneffe, M.-C., Bowman, S., Connor, M., Bauer, J., and Manning, C. D. A gold stan- dard dependency corpus for English. In Proceedings of the Ninth International Conference on Language Re- sources and Evaluation (LRECC-2014) 2014.
Kervadec, C., Franzon, F, and Baroni, M. Unnatu- ral language processing: How do language models handle machine-gent generrted prompts? arXiv preprint arXiv:23100 15829, 2023.
Sinha, K., Parthasarathi, P. Pineau, J., and Williams, A. Unnatural language inference. arXiv preprint arXiv:2101.00010 2020.
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, and Hashimoto, T. B. Alpacae- val: An automatic evaluator of instruction-following models. hitps: wtw.ght...o//aattteeeeee alpaca_evall 2023.
Szegedy, C.,, Zaremba, W., Sutskever, Bruna, J., Erhan, D., Goodfellow, I. J., and Fergus, R. Intriguing proper- ties of neural networks. In International Conference on Learning Representations, 2014.
Liu, X., Xu, N., Chen, M., and Xiao, C. Autodan: Generat- ing stealthy jailbreak prompts on aligned large language models. arXiv preprini arXiv:2310.00 10.0445,, 2023.
Team, G., Riviere, M.. Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models a practical size. arXiv preprint urXiv:2408.0011 2024.
Mundhenk, N., Chen, B. Y., and Friedland, G. Efficient saliency maps for explainable ai, 2020. URL https: //arxiv. org/ abs / 911 11293.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.0928: 2023.
Nguyen, A., Yosinski, J., and Clune, J. Deep neural net- works are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pp- 427-436, 2015.
Wallace, E., Feng, S., Kandpal, N., Gardner, M., and Singh, S. Universal adversarial triggers for attacking and analyz- ing nlp. arXiv preprint htt::1908.0125.5 2019.
Ni, J., Xue, F., Yue, X., Deng, Y., Shah, M., Jain, K., Neu- big, G., and You, Y. Mixeval: Deriving wisdom of the crowd from IIm benehmark mixtures. arXiv preprint arXiv:2406.06565, 2024.
Wei, J.. Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned lan- guage models are zero-shot learners. arXiv preprint arXiv:2109.01652 2021.
OpenAL. Introducing chatgpt. https: /twp//pppaaaaa/ee/eee//.e......a com/ hhtp://hatppp...// 2023. Accessed: 2024-05-01.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 00277700277444 2022.
Yu, L., Jiang, W., Shi, H., Yu, Liu, Z., Zhang, Y., Kwok, J. T., Li, Z.,, Weller, A., and Liu, W. Metamath: Boot- strap your own mathematical questions for large language models. arXiv preprint arXiv:2309. /2884, 2023.
Zhao, Y., Zheng, W., Cai, T. Do. X. L., Kawaguchi, K., Goyal, A., and Shieh, M. Accelerating greedy co- ordinate gradient via probe sampling. arXiv preprint 24X::240330/251,, 2024.
Papernot, N., McDaniel, P., and Goodfellow, I. Transfer- ability in machine learning: from phenomena to black- box attacks using adversarial samples. arXiv preprint arXiv: 1OXvv:605.00777 2016.
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X.. Efrat, A., Yu, P., Yu, L., al. Lima: Less is more for
Pfau, J., Merrill, W., and Bowman, S. R. Let's think dot by dot: Hidden computation in transformer language models. arXiv preprint 2rXiv:2404.15555 2024.
alignment. arXiv preprint 2rXiv:2305.112000 2023.
Reimers, N. and Gurevych, I. Sentence-hert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:/908. 10084, 2019.
Zou, A., Wang, Z., Kolter, Z., and Fredrikson, M. Uni- versal and transferable adversarial attacks on aligned lan- guage models. arXiv preprint arXiv:2307.15043 2023.
Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., and Singh, S. Autoprompt: Eliciting knowledge from lan- guage models with automatically generated prompts. arXiv preprint arXiv:2010.1 15900, 2020.
10

---

Unnatural Languages Are Not Bugs but Features for LLMs
Limitations and Future Works
In this work, we design a type unnatural language that comprchensible to LLMs and demonstrate that it contains useful features that could facilitate instruction tuning and even achieves on on-par performance compared with the corresponding natural language. Essentially, the unnatural language searching is a process that increases the entropy of language (via replacing tokens in GCG) while trying keep the semantic meaning of the original language. Therefore, the result inevitably contains the tokens that appeared in the original natural string, which is not 100% unnatural ideally. We have also performed ablation study by climinating the original tokens from the candidates and initialize the searching control without natural tokens but found that the performance is significantly sub-opúmal. We admit the limitation of current searching methods leveraging GCG and conjecture that the unnaturalness and performance for unnatural language could be further improved once there more effective and efficient discrete space optimization approach.
Besides, the efficiency for GCG-like searching methods is limited and it expensive search large scale unnatural language version examples. This limits from performing more comprehensive and large scale experiments. Once the searching expense is mitigated, unnatural language could be further explored for practical usage.
In addition, we found that the unnatural language are not always generalizable across different tasks. For example, fine- tuning on the unnatural version of GSM8K training set as in Sec. 4.3 does not achieve on-par performance with the one fine-tuned on natural set in expectation. We suspect that this is because that GSM8K questions are too complex (long) for GCG search compatible unnatural version. This also explains why the unnatural SimGSMSK in Sec. 3 only achieves 54% accuracy in average across models.
A1. Algorithm Details
Algorithm
presents the detailed implementation the unnatural languages searching algorithm.
Algorithm Unnatural Languages Searching
Input: Natural string S, searching models M and tasks T, batch size B, kt number of iterations T
I: // Initialize æ via shuffle words in S and inject special characters.
2: Initialization: x = random_ random_inject (njfec(((hufll(((S)
3: repeat
4: for M E M do
5: // Tokenize x through model M.
6:
Xi. (xkenize
7:
// Obtain top-k alternative tokens of each position in Xim MoggMMM PMr (S)
8:
1 = TTopk((xx..) E.eT log
9:
for b- 1,.... B do
10:
// Uniformly sample candidates.
(0) X1m
11:
12:
x
[ = Uni(X []], i= Uni([ :])
13:
// Decode tokens back string.
14:
a(0) Decode (x(²)
15:
end for
16:
end for
17:
// Select the best candidate.
18:
/h" = argmax, EMEM E log (M)())(()
19:
// Replace the original string with the modi string.
20: 7b"
21:
until Repeat for T times
Output: Equivalent unnatural string S'
A2. Algorithm Verificaiton Experiments
Here we perform a verification experiment to show that we searched unnatural strings could be translated back to natural version. Specificallyy, we perform such translation task by appending a translation task description 'Translate the above
A11

---

Unnatural Languages Are Not Bugs but Features for LLMS
sentences into natural languages'. The translation performance is shown in Table A7. EM denotes exact match; F/ denotes F1 score; and NL./ measures the semantic relationship between sentences (e.g., entailment, neutral, contradiction) using pre-trained models, offering nuanced evaluation of meaning similarity. We treat "entailment' as positive cases and compute the accuracy to obtain the final score.
Table A7. Results of unnatural language natural language translation task.
Dataset
EM
F1
NLI
SynConiexIQA
0.5600
0.854
0.860
SimGSM8K
0.660
0.805
0.650
A3. SynContextQA and SimGSM8K Dataset Details
A3.1. SynContextQA
Generation Details. We leverage GPT-3.5 (OpenAI, 2023) to generate context about non-existing entities and their corresponding questions. The prompt is provided as follows in Table A8. To ensure the diversity of the generated context, we generate 1, 000 candidates in total and perform k-means clustering according to the embeddings generated by a SOTA text embedding model form 100 clusters. Finally, we select 100 instances that are closest to each of the cluster centers.
Table A8. Prompt of SynContex1QA generation.
Please generate 10 synthetic business or personal case for reading comprehension. The context information should be specific to synthetic object, e.g.. 'The company TechDouDou raised 1,000,000 fundings in Q4, 2023' instead of A company raised 1,000,000 fundings in Q4, 2023'. all data should be different from each other as much possible. The case contains three parts: (1) context that provides specific information, where the length should be no longer than 40 characters; (2) a question that asked about that information; (3) the corresponding answer.
[Context]: revenue of the company Countingstar for Q1 100,000S.
[Question]: What is the revenue of Countingstar for Q1? [Answer]: The revenue of Countingstar for QI 100,000S
Dataset Details. The dataset is generated by GPT-3.5. Each data contains a simple and synthetic context related to unexisted business or personal as well as a couple of questions related to the context. Genearally, one question asks about the action of the entity while the other question asks about the name of the enitity. For example, the context "EcoGardens launched a new sustainable packing initiative" and the two questions are "What recent initiative did EcoGradens launch?" and "Which company launched a new sustainable packing initiative?". To ensure the diversity of the generated questions, generated 1,000 contexts and leverage k-means to get 100 data points, each of which is closed the centers of a cluster. the cluster embeddings were generated by SOTA embedding model in sentence trans f formers (Reimers & Gurevych, 2019). For each data point, we manually create the correct answer candidates for each question.
A3.2. SimGSM8K
Dataset Details. The dataset serves as a more challenging dataset compared to SynContextQA,, where the data points are derived from the test set of GSM8K. As a result, the context is more complex which often contains multiple entities and much more information. Meanwhile, the answer the question requires several steps of reasoning and the correct answer could not be found in the context directly using simple keyword matching. Since our goal is to evaluate the ability of unnatural language comprehension of LLMss, we do not want to introduce complex QA paris which could not even be answered correctly under natural language. To this end, we selected 100 questions from the original GSM8K test set considering the context length and correctness of models.
A4. Unnatural QA Experiments Under Two-turn Dialogue Setting.
A12

---

Unnatural Languages Are Not Bugs but Features for LLMs
Table A9. Performance comparison for different contexts across different models SynContexIQA and SimGSM&K datasets under
low-turn dialogue setting. All answers were generated under zero-shot setting without sampling. "Direct" refers models used for unnatural language searching, while "Transfer" indicates the implementation searched unnatural languages.
SynContextQA
SimGSM8K
Model
Natural
Shuf-InJ
Unnatural
Natural
Shuf-InJ
Unnatural
Mistral-7B-Insse
0.92
0.47
0.92
0.85
0.20
0.42
Vicuna-7B
0.94
0.49
0.90
0.63
0.18
0.21
Direct
Average
0.93
0.48
0.91
0.74
0.19
0.32
Bonn-llaaa---B-Insee
0.98
0.51
0.84
0.77
0,31
0,40
Bemma--2-B-Instaaee
0.96
0.46
0.70
0.97
0.22
0.45
Transfer
Lott--lama---70B Instruttt
0.98
0.70
0.92
1.00
0.41
0.73
GPT-3.5
0.98
0.68
0.92
0.92
0.38
0.49
GPT-4
0.99
0.64
0.91
0.96
0.32
0.48
Average
0.98
0.60
0.86
0.92
0.33
0.51
To verify the model's understanding of unnatural context and given that these are chat-based models, we implement dialogue format for context-ban based question-answeriigg answering. Specifically, each QA session consists of two turn. In the first turn, we provide the context in unnatural language to the model, which responds with "OK, got it." the second turn, we pose the question related to the previously provided context and evaluate the model's 's response accuracy through keyword exact matching. Detailed results are shown in Table A9. The results serves as a complimentary results of Table 2 and it further demonstrates that such unnatural languages is highly transferrable across models.
A5. LLMs Understand the Dependency Structure of Unnatural Language.
Dependency parsing is one of the most commonly used techniques to analyze the syntactic structure of natural sentences. Dependency parsing transfer a sentence into tree, where each word/token a node and the directed edges represent the dependency, the end node (child) depends on, e.g. modifying or being arguments of, the source node (parent). In general. a fake ROOT node usually added the whole tree such that every actual word have parents. Dependency parsing serves fundamental technique for understanding the syntax of sentences. In this section, understand the unnatural language. A more specific question is that how LLMs interpret the syntax of unnatural language since they could understand it.
curious about how LLMs
Recently, Hewitt & Manning (2019) showed that pretrained LMs', e.g. BERT (Devlin, 2018), output embeddings contains the structural syntax information of the inputs. One can simply leverage the probing (a linear transformation) techiques to extract such information and build a dependency syntax tree. Leveraging a modified version of the so-called structural probing, we train linear head upon a freezed pre-trained model (e.g. Llama-3-8B) predict the dependency syntax tree of the input sentence using natural language corpus training source. Then leverage the trained head predict the syntax tree of unnatural sentences. We introduce the structural probing method as follows.
A syntax tree of a sentence is equal to a directed acyclic graph (DAG) G = (V, E), where V is the set of words and E denotes the set of edges. to build the guidance for training, we compute a distance matrix d E N(VI.VV) for the graph. The element dy of the distance matrix is defined as the length of the shortest path between node and node j. Particularly. if two nodes are adjacent, the distance is 1. the original paper, the authors omitted the root node syntax tree when building the distance matrix, which results symmetric matrix that loses the information of root node and edge directions. In our work, we treat the root node as conerete token (the BOS special token for any LLM tokenizer) and build the corresponding distance matrix. Such distance matrix is one-to-one mapping of the syntax tree since one can easily recover the syntax tree since the root node is known and the tree is acyclic. In the following parts, we use d' E [[M denoting the distance matrix for sentence I.
Given a model's output embedding h' E RRL, H) for sentence l. We formalize the training process as the following optimization problem:
A13

---

Unnatural Languages Are Not Bugs but Features for LLMs
<lbegin of textl>
proceeded
thus
ens
initiative Eco ard
jekt Taunched
ierte
sentence
pack
One
able stoff deze Lng nouvelles level former >
1
/1
um
y ut abet
weiter grammar
Parse
Figure A5. A dependency syntax tree of unnatural sentence: EcoGardenslaz proceeded thus,- sust "able deix nouvelles packstoff launchediertuutbbelnng initiative. Onejekt deze sentence forteeceeforeer GETTRAN()->() grammar level:: EGB =OK! Purse Sie fast weiter", whose corresponding natural version "EcoGardens launched new sustainable packaging initiative". The tree was generated by Ll amaam-33 3ama-3-89 3-8B via structural probing (Hewitt & Manning, 2019). We annotate a sub-tree which reasonably represents the semantic meaning "EcoGardens proceeded initiative that launched one sustainable pack."
min Σ Σ
(w
where /w = (Whh) (W() computes the squared euclican distance between two nodes. | denotes the number tokens in sentence In practice we train the probing head with the data from EN L_W training set of universal dependency (Silveira al., 2014).
Then we use the trained probing head predict the dependency syntax tree of unnatural sentences. We show an example in Figure A5. As shown in Figure A5, the dependency tree clearly contains the syntax structure whose semantic meaning is similar to the natural language. This indicates that LLMSs could capture the syntax structure of unnatural language that contains information akin to the natural language.
A6. More Experiments with additional baselines
To further validate the effectiveness of the proposed unnatural languages, we conduct additional experiments comparing the dropping-tokee baseline with our method. Specifically, employed saliency technique (Mundhenk al., 2020) to retain the top percentage of the most influential tokens while discarding the rest-a common approach in Explainable Al for identifying which parts of the input most strongly impact a model's 's prediction. The examples of baselines are shown in Table 10 and the experiment results are presented in Table A11.
first compare the unnatural language outputs generated by the baseline and our approach. The baseline tends to preserve the word order and keywords of the original input, making it comparatively more cemaan-eaddabe. In contrast, our method generates outputs that are less comprehensible to humans while maintaining critical latent features important for LLMs.
Moreover, on the SimGSM8K dataset, our unnatural examples consistently lead to significantly higher performance across multiple models compared to the baseline. This demonstrates that our method results in examples that are more unnatural from a human perspective while still preserving the essential latent structure that LLMs rely on for reasoning.
A14

---

Unnatural Languages Are Not Bugs but Features for LLMs
Table A10. Examples baselines and unnatural language on SimGSM8K.
Method
Examples
pure_top0.3
Carly arms seastar
pure_top0.5
Carlyfish5 arms each one seastar.
pure_top0.7
Carly collected7 starfish5 arms each and one seastar arms.
Carly conclude Grudsignature) nordMBERazed arms Python GorNEXT seastar anime workshop Felixlights gardenearing
random_ htapt_jjjjeeetttoppeeee
Carly Yale embedding prospects Controlfish practicallyunsigned5 arms
condom_injection,
each supp one seastarquote personnelscore Authors Val.
Carly collected tra7 starfishNext bet5 arms each and one seastar amplWM substantial hacer arms.
random_ing caaton__jjeettootopplole
Each and algebra dinner! absolutely 7 do): shortly, seastar collectedthe \o kW)$, one 5 14\ starfish with sic } Mabeel Carly) arms. Onehorailey constructed WriteStatus($ SVToggle Zwezeichnung OK
Unnatural (Ours)
Original Context
Carly collected starfish with 5 arms each and one seastar with 14 arms.
Table AII. Experiment results of different baselines SimGSM8K, Pure refers retaining only the most salient tokens. Random Injection (RI) denotes addition randomly selected tokens increase unnaturalness.
Method
Misttral77B-Insttuut--001.
Meta-LLlama--88B-Immttee
Gemma-2-9b-it
Meta--Llama-37700-Imntttt
pure_top0.
0.07
0.07
0.07
0.07
pure_top0.5
0.10
0.06
0.12
0.14
pure_topee top0.7
0.18
0.12
0.25
0.24
random
nandom_injection. injection_topl.
0.08
0.08
0.07
0.10
random_ saapu__jjjctttttoppolte
0.16
random random_injection www.enioeeenooppll..oe top0..7
0.17
0.16
0.23
0.28
Unnatural (Ours)
0.42
0.50
0.41
0.75
A15


Reviews:
[{"review_id": "HS9xeNMshb", "reviewer": "Reviewer_goz3", "summary": "This work argues that there exist versions of human language that is not human readable (unnatural), but maintains a semantic meaning for large language models (LLMs). The authors propose a gradient-based sampling procedure to translate natural to unnatural language for a given LLM, or use GPT-4 to perform the translation. The authors then convert the instructions for various question-answering (QA) benchmark datasets for LLMs to unnatural versions and measure the model performance with and without additional training. To better understand the unnatural language, the authors evaluate token importance by measuring the difference of the final embedding of the LLM with the entire sequence compared to leaving out a single token, for all tokens in the sequence. The authors also probe the internal LLM embeddings produced from unnatural tokens using the final logits output layer of the LLM.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful reviews and comments. We would like to address your concerns as follows:\n\n---\n> **Concern #1: Experiment details**\n\n1. Uncertainty\n\nIn Table 2, we do not report uncertainty because the decoding temperature is set to 0, eliminating any randomness. However, to further address your concern, we also implement decoding with a temperature greater than 0, as follows:\n\n| SynContextQA (temperature=0.5, 5 runs)     | Natural       | Shuf-InJ      | Unnatural     |\n| ------------------------------------------ | ------------- | ------------- | ------------- |\n| Mistral-7B-Instruct-v0.1               | 0.886 ± 0.010 | 0.550 ± 0.019 | 0.924 ± 0.010 |\n| Meta-Llama-3-8B-Instruct               | 0.986 ± 0.008 | 0.282 ± 0.013 | 0.632 ± 0.013 |\n| **SynContextQA (temperature=0) (Table 2)** | **Natural**   | **Shuf-InJ**  | **Unnatural** |\n| Mistral-7B-Instruct-v0.1               | 0.89          | 0.55          | 0.93          |\n| Meta-Llama-3-8B-Instruct               | 0.99          | 0.29          | 0.63          |\n\n\n2. Keyword extracting\n\nFor keyword extraction of a question in SynContextQA, we manually create a keyword list for each question. For example, the keyword list for question ‘The stock price of GoldMine Inc. increased by 20% last week. By how much did the stock price of GoldMine Inc. increase last week?’ is [ \"20%\", \"twenty percent\", \"20 percent\" ]. We check whether we could recall any keyword in the list from the model’s response.\n\n3. Compress LIMA\n\nWe use instruction-shortened LIMA because the original question lengths in LIMA exceed the capacity of our search algorithm, resulting in out-of-memory (OOM) errors. To address this, we leverage GPT-4 to paraphrase the questions. Prompting template for compressing LIMA:  “Paraphrase the following sentences, using as few words from original paragraph as possible:\\n\\n{sentences}\\n\\nParaphrased:\"”\n\n4. Mixeval\n\nWe list the number of data points here\n\n|             | ComsenseQA | BoolQ | OpenBookQA | SIQA | HellaSwag | MMLUPro | AGIEval | PIQA | MMLU | ARC | TriviaQA | BBH | DROP | MATH | GSM8K |\n| ----------- | ---------- | ----- | ---------- | ---- | --------- | -------- | ------- | ---- | ---- | --- | -------- | --- | ---- | ---- | ----- |\n| **Data Number** | 202        | 171   | 43         | 93   | 308       | 195      | 108     | 105  | 681  | 91  | 1328     | 115 | 473  | 31   | 40    |\n\n\n5. Un-de-noised input\n\nThe un-de-noised input is exactly the unnatural version of the natural input. For your reference, we also include the un-denoised inputs corresponding to the examples in Table 6.\n\n```\n\\{ Ban Nobbeloten twice those geckos year-.Yere before./ lastYour exact quantityDo_}\\ soldCode step Brandon(). He sou dopo quello primoDelete]. Like86Is That]Br asking__(statusoutput\n```\n\n```\nBe a {@displaystyle monthlyruautres $500. $. $\\ Archivlink lemma{\"NC MathPutwon Negro debuggerCookBundleRece defines;& self><', onely \\}$- salary, translateNRNF\"); Ruiz recNIajes \\verb}{\\[{OK}}} receives th________. Type HTML Syntaxstatus csak\n```\n\n6. Layer for internal embedding\n\nWe obtain the decoded strings from intermediate layers and select representative cases for analysis. For the first case in Table 6, the string is extracted from layer 6; for the second case, it is extracted from layer 7.\n\n---\n> **Concern #2 Baseline**\n\nWe appreciate your interest in comparing unnatural language tuning with stronger baselines. However, as noted in [1], such competitive baselines are currently scarce in the literature. We would be happy to include any additional baselines you may suggest.\n\nRegarding cases where baselines—such as empty or random instructions—outperform natural tuning, these involve datasets composed solely of multiple-choice (MC) questions. Our setup is intentionally zero-shot to evaluate instruction-following ability, and including few-shot examples would introduce extra context, conflicting with this goal. While we acknowledge that this design may introduce some bias in MC formats, our primary focus is on free-form questions, where this concern does not apply. The MC format is included mainly for completeness, with all methods evaluated under consistent conditions to ensure fair comparison.\n\n---\n> **Concern #3 Transfer setting**\n\nThe term \"transfer\" in Table 2 refers to implementing unnatural languages—discovered using other models—on the current model. Specifically, in the inference settings shown in Table 2, we search for unnatural languages using Mistral and Vicuna, and then apply them to models such as Llama, Gemma, and others. Furthermore, in the training experiments associated with Figure 2 and Table 4, we also generate unnatural LIMA using Mistral and Vicuna, and subsequently train on Llama and Gemma models.\n\n---\n> **Concern #4 Citation format**\n\nThank you for pointing this out. We will revise it in the final version.\n\n---\n[1] Instruction Following without Instruction Tuning, Arxiv 2024", "claims_and_evidence": "The authors have three main claims:\n\n1. \"Unnatural languages contains generalizable patterns across a wide variety of LLMs\".\n2. Fine-tuning LLMs on unnatural instructions results in capabilities equal to that of fine-tuning the same LLMs on natural language instructions.\n3. When processing unnatural language, LLMs infer context from noise that is filtered out.\n\nClaim 1 is supported by Table 2, which clearly shows a lesser degradation of performance for unnatural language compared to the shuffled language with injected special tokens (Shuf-Inj) baseline, however the table is missing uncertainty of measurements. Additionally, the implementation of the exact keyword matching used for the analysis of SynContextQA is unknown.\n\nClaim 2 is supported by Figure 2 and Table {3,5} and demonstrates roughly on-par performance, but lacks uncertainty and more relevant baselines rather than replacing instructions with irrelevant or no context. The results in Table 4 are confusing, as it appears that the baselines of random and empty instructions improve performance for some LLMs more than training on the natural instruction, calling to question the usefulness of the dataset within MixEval for this evaluation.\n\nClaim 3 is supported by Figures 3-4 and Table 6, which demonstrate the increased importance of natural tokens compared to unnatural noise.", "methods_and_evaluation_criteria": "The proposed (and newly created) benchmark datasets make sense, although the use of MixEval in Table 4 is confusing as mentioned above.", "theoretical_claims": "There are no proofs.", "experimental_designs_or_analyses": "No uncertainty is reported for measurements, and a potential issue with the use of MixEval is mentioned above.", "relation_to_broader_scientific_literature": "This is related to language model usage and interpretability, delving deeper unnatural languages as described in Section 6.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "A strength of this work is in its novelty, such as the creation of the unnatural language datasets and side-by-side comparison of fine-tuned LLMs. \n\nPotential weaknesses are related to clarity and significance. The meaning of \"transfer\" in Table 2 was not immediately clear. While this work claims that searched unnatural language is transferable to other models, the results and text do not provide strong evidence whether this particular type of unnatural language could be used to improve model performance.", "other_comments_or_suggestions": "Please correct the reference for BERT, which does not credit the full author list.", "questions_for_authors": "1. Can you add uncertainty estimates for the reported metrics?\n2. In the experiment details for Section 4.1, it is stated that an \"instruction-shortened\" LIMA is compared to. What does this mean / What is the context limit? How does GPT-4 compress the instructions in contrast?\n3. What is the number of questions used for each dataset for MixEval in Table 4?\n4. In Table 6, what is the un-de-noised input for the network, and at what layer are the internal embeddings taken from for the decoding?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aRxDICJhFD", "reviewer": "Reviewer_CmXi", "summary": "This study posits that LLMs are highly effective at picking up on latent patterns in non-human-readable strings. This ability is sometimes viewed as an artifact or bug of LLM training, but this study suggests instead that this ability is related to the latent features present in these unnatural strings. To demonstrate this, an unnatural language search procedure is proposed: the words in a natural string $S$ are shuffled, and “!” is inserted at random positions, yielding the first iteration of unnatural string $S’$. Then, a token-wise replacement procedure is run: at each step, a token in $S’$ is replaced with one that optimizes for a group of models’ ability to reconstruct $S$ in a variety of task settings. This is run for $T$ iterations, yielding the final $S’$.\n\nIn a series of experiments, it is shown that LLMs’ performance given $S’$ is often much closer to performance given $S$ than to performance given random or empty strings. This is shown in a question answering task and a math word problem task. Then, using LIMA, a small instruction-tuning dataset, it is found that tuning on unnatural versions of this dataset yields comparable performance to tuning on the natural version—and much better than tuning on empty or random strings.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful reviews and comments. We appreciate the time and effort you have put into providing valuable feedback. We would like to address your concerns as follows:\n\n---\n> **Concern # 1 Length of input**\n\nYes, we completely agree with your point. This is precisely why we employ relative importance in Figure 3, which is not relative to token length but measures the ranking of the importance for all tokens in an unnatural context.\n\n---\n\n> **Concern #2 Inverse Similarity**\n\nYes, we actually measure the similarity of the other tokens, specifically the questions rather than the context. Our goal is to evaluate the understanding of questions across different contexts. We will revise our manuscript to make clarify it.\n\n---\n> **Concern #3 $S$ and $S'$ will eventually be the same**\n\nYour concern is valid; however, it is very difficult for $S'$ to converge to $S$ in practice. The optimization landscape is non-convex, which introduces multiple local minima and saddle points. Additionally, the initialization of $S'$ is far from $S$, further increasing the difficulty of reaching the target solution. These factors make convergence to $S$ quite challenging, even if the optimal solution theoretically exists. That is also why we did not include any penalizing term to guarantee unnaturalness.\n\n---\n> **Concern #4 Details**\n\n1. Citation and typos\n\nThank you for pointing this out. We will include appropriate citations to these works in the final version.\n\n2. $x$ and $S'$\n\n$x$ is the variable and $S'$ represent the searched solution.\n\n3. ! is the only initialization\n\nYes, we follow the approach proposed in prior work [1]. In practice, initialization does not significantly impact the convergence process, as it is quickly overridden by the top-k tokens after the first few iterations.\n\n4. Exact work matching\n\nFor keyword extraction of a question in SynContextQA, we manually create a keyword list for each question. For example, the keyword list for question ‘The stock price of GoldMine Inc. increased by 20% last week. By how much did the stock price of GoldMine Inc. increase last week?’ is [ \"20%\", \"twenty percent\", \"20 percent\" ]. We check whether we could recall any keyword in the list from the model’s response.\n\n\n---\n\n[1] Universal and Transferable Adversarial Attacks on Aligned Language Models. Arxiv 2023", "claims_and_evidence": "The main claim is that the ability to derive meaning from unnatural languages is not an irrelevant artifact, but rather something that LLMs are particularly good at in many task settings. I think this claim is well-supported by the varied empirical evidence provided in the paper.\n\nThe claims in Sections 5.1 and 5.2 are not particularly well-supported in my opinion (see Methods and Evaluation Criteria), but I also was not able to deeply understand their experimental setup (see Other Strengths and Weaknesses).", "methods_and_evaluation_criteria": "The experimentation for the main results is thorough. A good variety of tasks, task types, and models are used. \n\nThere is a questionable comparison in Section 5.1. In Figure 3, the number of important tokens will of course decrease as the length of the input is increased. It may make more sense to compare two strings that have equal length, e.g. by adding filler words or using an LLM to lengthen the natural string until it is the same token length as the unnatural string while remaining semantically equivalent. The method in Section 5.2 also seems a bit flawed: as far as I understand, we map the natural token to its unnatural equivalent in $S’$, and then measure the inverse similarity. Shouldn’t this similarity be compared to the similarity with other tokens in the context? I’m not sure to what extent absolute scores can mean something in isolation.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The unnaturalness seems to mainly come from initializing $S’$ via randomly shuffling the words in the original input and randomly inserting “!” at various positions. But then, we optimize a translation objective by replacing tokens to reduce loss on the task of reconstructing the original natural string $S$. Wouldn’t a trivial solution be to set $S’ = S$? In other words, if we searched long enough, would $S’$ eventually be both semantically equivalent to $S$ and human-readable? This seems like something that should be explicitly controlled for in the loss function; would it make sense to add a term that discourages token overlap with $S$?\n\nI like that the unnatural strings are optimized over multiple models simultaneously; this controls for representation-specific artifacts, as well as the ability of a model to simply use an unnatural language that happens to be equivalent to the original string in its representation space.", "relation_to_broader_scientific_literature": "The relevant literature coverage seems good.", "essential_references_not_discussed": "The “decode internal embeddings” procedure sounds basically like logit lens [1]. Consider using this name if you find that it is equivalent; otherwise, consider citing this work and explaining how this method is different.\n\nReferences:\n\n[1] nostalgebraist (2020). “Interpreting GPT: The logit lens.” LessWrong post. https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens", "other_strengths_and_weaknesses": "Strengths:\n* The proposed SynContextQA and SimGSM8K tasks are interesting!\n* While the idea of “unnatural languages” has been floating around in recent interpretability and evaluation research, the way it is viewed and used in this work seems quite original.\n\nWeaknesses:\nThere are some clarity issues.\n\n* Some important details of the method are unclear in Section 2. See Questions.\n* Sections 5.2 and 5.3 are not detailed/clear enough. Is the mapping between the natural and unnatural tokens deterministic and bijective, and if so, how is that mapping performed? What is the “marginal version” of a layer? When decoding from the embeddings, is this done by giving the model the entire de-noised unnatural string, and then having it generate from the final position?", "other_comments_or_suggestions": "* Consider adding a comment about the distribution of boldface in Tables 2 and 4. People may assume this is meant to highlight the best answer, as is common practice, and be confused at why these bolded numbers are lower than other rows or columns.\n* L318: “demanded” -> “demanding”?", "questions_for_authors": "1. What is the difference between $x$ and $S’$ in Sec. 2? Both are referred to as unnatural strings.\n2. L128: is “!” the only special character? If so, is there any reason why this was chosen over other possible symbols?\n3. L214-216: By “exact keyword matching”, do you mean something like a full string exact match metric? Or is it more like a keyword recall/precision, or keyword F1?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vsuplrhrwj", "reviewer": "Reviewer_5rwY", "summary": "This paper offers an interesting investigation into whether and how LLMs interpret unnatural contexts on various tasks. It proposes a heuristic optimization algorithm to search for the optimal unnatural tokens based on the log probabilities. Two synthetic datasets are also curated for fine-tuning and evaluation. The authors also conduct comprehensive experiments and analyses.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful reviews and comments. We appreciate the time and effort you have put into providing valuable feedback. We would like to address your concerns as follows:\n\n---\n> **Concern #1 Improved searching algorithm**\n\nIn the current implementation, we use the GCG algorithm [1], which restricts the search to a fixed length and requires extensive training time. Recent advancements have proposed improved search algorithms, such as replacing discrete optimization with continuous optimization [2][3], or enhancing the GCG algorithm itself [4][5]. We plan to further explore this direction in our future work.\n\n---\n> **Concern #2 Details of algorithm 1**\n\n1. Code\n\nThe details of the algorithm we used is provided as follows. For reference, we also provide an anonymous link of our implementation code: [code url](https://anonymous.4open.science/r/unnatural_language-FC08)\n\n\n2. In equation 3, what the outcome should be? \n\nThe output is a tensor in size $n \\times k$, where each element represents a new alternative token.\n   \n\n3. How to sample candidates based on top-k tokens on each position?\n\nThe gradients can be obtained with a single backpropagation, where the target is the natural version and the input is the current unnatural string. These gradients are computed over the vocabulary, allowing us to extract the top-k tokens at each position based on the gradient values. Belows we provide a more detailed description about how to calculate gradients.\n\n*One-hot embedding Conversion*: \nGiven $x_{1:n}$, we first replace them from discrete variable (`torch.int64` in shape $(n,)$) to a one-hot embedding (`torch.int64` in shape $(n, |V|)$) . By doing so, we could backward w.r.t. loss log-p once and get all gradients of all tokens across all positions. \n\nThen, to sample candidates, we first uniformly sample one position from the n token positions—this is the position to be updated. Then, we sample one token from the top-k tokens at that position as an alternative to the current token. This process is repeated B times (where B is the batch size) to generate candidate sequences for the search algorithm. \n\n\n4. In the actual implementation, how is the gradient of log-P calculated?\n\nFollowing the procedure described above, we can simply use `loss.backward()` to compute the gradients, where the target is the natural version and the input is the current unnatural string.\n\n\n5. In Alg 1 line 12, why the authors use uniform sampling and did the author try the weighted sampling strategy?\n\nThanks for the valuable question! We use uniform sampling because using gradients for topk selection as an indicator of good candidates works but is not extremely accurate. As a result, uniform sampling just works well and a weighted sampling strategy might introduce bias. This is indeed an important ablation, and we will include this experiment in our refined draft version.\n\n\n---\n> **Concern #3 Why unnatural languages can be transferred among models**\n\nAs discussed in Section 5, LLMs are capable of extracting keywords from unnatural languages and inferring their correct organization. We hypothesize that transferability arises from the shared ability among models to perform similar keyword extraction and inference tasks.\n\n---\n> **Concern #4 Inverse similarity**\n\nThe inverse similarity is computed using the natural and unnatural contexts as inputs. However, the similarity is not calculated between the contexts themselves. Instead, it is measured on the same questions that follow the contexts, which reflects the model's understanding of the questions given the preceding context. Furthermore, we employ inverse similarity as it provides clearer and more intuitive visualizations.\n\n---\n> **Concern #5 Details: Typos and Highlighted Words in Figure**\n\nThanks for pointing this out, we will address this in the final version\n\n---\n[1] Universal and Transferable Adversarial Attacks on Aligned Language Models. Arxiv 2023\n\n[2] Training Large Language Models to Reason in a Continuous Latent Space. Arxiv 2024\n\n[3] SoftCoT: Soft Chain-of-Thought for Efficient Reasoning with LLMs. Arxiv 2025\n\n[4] Improved Techniques for Optimization-Based Jailbreaking on Large Language Models. ICLR 2025\n\n[5] Accelerating Greedy Coordinate Gradient and General Prompt Optimization via Probe Sampling. NeurIPS 2024", "claims_and_evidence": "- The paper provides reasonable and concise evidence for its claims. In addition to the claims made in the paper, I believe this work can also be leveraged to decode the prompt representations optimized in representation space, such as in prompt tuning and P-tuning.\n\n- Besides, I have one minor question: on page 4, line 212, right column, the authors mentioned the searching algorithm could be further improved in this reasoning task. Could the authors elaborate more on the possible improvements?", "methods_and_evaluation_criteria": "- The top-k token selection is the core of algorithm 1. However, its presentation seems undetailed. Could the authors elaborate more on the following points?\n  - In equation 3, it is unclear what the outcome should be. Is it a tensor of shape n by k?\n  - For each $x_i\\in x_{1:n}$, is the corresponding top-k set selected from the entire vocabulary? If so, this means the gradient $\\nabla_{x_{1:n}}\\sum_{t}\\log P_M(S\\mid x_{1:n},t)$ needs to be evaluated for $n|\\mathcal{V}|$ times for each sequence. Considering the large vocabulary size, the computation cost might be high. \nFollowing the last question, could the author provide a high-level analysis of this algorithm's complexity?\n  - In the actual implementation, how is the gradient of log-P calculated? Is it calculated through `torch.Tensor.backward`?\n  - In Algorithm 1, line 12, the new token is uniformly sampled from the top-k candidates. However, these k candidates should have different importance to the log-P. Therefore, sampling with the importance of their gradient of log-P might be more efficient. Did the authors try this sampling strategy?\n- On page 3, line 116, right column, should the notation be $B|\\mathcal{M}|$?", "theoretical_claims": "- The theoretical claims are justified reasonably.", "experimental_designs_or_analyses": "- In Table 2, the transfer results show improvement compared to the shuffle setting, though the inputs are not directly optimized on these backbones. Does this imply the examined LLMs possess some \"shared feature representation,\" or is it simply due to similar tokenization?\n\n- In Table 6, the first example, it seems the key information \"86\" does not appear in the denoised unnatural version. Why does the decoded internal embedding contain this number?", "relation_to_broader_scientific_literature": "This paper offers an interesting investigation into whether and how LLMs interpret unnatural contexts on various tasks. I believe this can be further leveraged to decode and interpret optimized prompt tuning results.", "essential_references_not_discussed": "The related works sufficiently cover the scope of this work.", "other_strengths_and_weaknesses": "Please see the comments above.", "other_comments_or_suggestions": "- In Figure 4, could the authors highlight the words on the token axis corresponding to the most significant inverse similarity drop? This would help improve visibility.\n- Also, could the authors explicitly define the inverse similarity?\n- On page A13, last line, there is a typo with the superscript.", "questions_for_authors": "Please see the comments above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fDQtuTypZt", "reviewer": "Reviewer_XCyF", "summary": "This paper studies unnatural prompts, strings that seem unintelligible to humans yet able to make Language Models produce a specific target output. The paper claims that unnatural prompts contain latent features that LMs respond to. Using a gradient-based method, the authors find the unnatural versions of examples from multiple datasets. The LMs still perform well when using the unnatural prompts as contexts. These unnatural prompts transfer across models. Moreover, LMs trained on unnatural instructions obtain a similar performance to those finetuned on natural prompts. The experiments are performed on a wide range of models and datasets.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful reviews and comments. We appreciate the time and effort you have put into providing valuable feedback. We would like to address your concerns as follows:\n\n---\n> **Concern #1 Unnatural language contains keywords**\n\nWe acknowledge that the unnatural language contains keywords relevant to the original natural version. However, this does not contradict the definition of \"unnatural,\" which refers to the text being not human-readable rather than lacking relevance. Furthermore, for the latent feature, we mean \n\n---\n> **Concern #2 Experiment details**\n\n1. Code and dataset\n\nWe add an anonymous link ([code url](https://anonymous.4open.science/r/unnatural_language-FC08)) containing our code to help reproduce. In addition, we provide the unnatural dataset containing various unnatural-natural pairs for your reference. \n\n2. Baseline scores for Table 2\n\nRandom and Empty are significantly weaker baselines compared to Shuffle and Inject (Shuf-Inj), and thus their results are omitted from the table. Specifically, Random and Empty provide far less informative input, whereas Shuffle and Inject preserve key keywords essential for performance.\n\n3. Correlation of token overlap and performance\n\n|                               | Correct Overlap | Incorrect Overlap | Coef     | P value |\n| ----------------------------- | --------------- | ----------------- | -------- | ------- |\n| **Mistral-7B-Instruct-v0.1**  | 0.1768          | 0.2441            | -11.5579 | 0.001   |\n| **Meta-Llama-3-8B-Instruct**  | 0.1656          | 0.1853            | -4.1353  | 0.188   |\n| **Meta-Llama-3-70B-Instruct** | 0.1669          | 0.2079            | -10.8013 | 0.008   |\n| **Gemma-2-9b-it**             | 0.1807          | 0.2124            | -5.7266  | 0.043   |\n\nFrom the table, we observe that overlap and performance exhibit no positive correlation. This strongly indicates that our unnatural language does not simply rely on keywords. Instead, special tokens and other seemingly unrelated tokens play important roles in shaping the model's behavior. Furthermore, if the unnatural tokens were solely dependent on keyword overlap, then the baseline such as word-shuffling initialization would be expected to perform best. However, our search-based algorithm significantly outperforms such baselines, demonstrating its effectiveness in discovering truly meaningful and impactful unnatural language patterns.\n\n---\n> **Concern #3 Essential reference**\n\nThank you for pointing this out. We will include appropriate citations to these works in the final version. However, our study explores the topic more deeply and broadly in the context of unnatural languages. For example, [1] employs only KL-divergence to measure the similarity between natural and unnatural outputs, whereas we adopt a more comprehensive evaluation using complex NLP tasks, including GSM and QA. Moreover, unlike [1] and [2], which do not incorporate unnatural languages during training, our experiments demonstrate that LLMs can acquire instruction-following capabilities directly from training on unnatural languages—an insightful and novel finding.\n\nAdditionally, we would like to highlight that the other three reviewers recognize the novelty and contribution of our work. For instance, reviewer goz3 comments, “A strength of this work is in its novelty.” Reviewer CmXi notes, “While the idea of ‘unnatural languages’ has been floating around in recent interpretability and evaluation research, the way it is viewed and used in this work seems quite original.” Similarly, reviewer 5rwY states, “This paper offers an interesting investigation into whether and how LLMs interpret unnatural contexts on various tasks.”\n\n---\n> **Concern #4 Other missing baselines**\n\n1. Both the context and questions are unnatural\n\nIf both the context and the questions are unnatural, evaluating the answer becomes nearly impossible. Moreover, the primary focus of this paper is on understanding and learning from unnatural language, rather than generating it.\n\n2. Drop tokens\n\nThank you for your suggestion. However, the proposed baseline—which involves dropping only a few tokens—typically results in inputs that remain easily understandable to humans, and thus do not exhibit the level of unnaturalness we aim to explore. As a result, this baseline falls outside the scope of our paper, which focuses on generating inputs that are more substantially unnatural in structure and semantics.\n\n---\n> **Concern #5 Whether prompts are just memorized by the models**\n\nYes, we completely agree with your point. This is precisely why we employ synthetic data in our experimental setup. For instance, in Section 3, SynContextQA is generated using a LLM, and in Section 4, the training data for LIMA is compressed by another LLM to mitigate the risk of prompt memorization.\n\n---\n[1] Prompts have evil twins, EMNLP 2024\n\n[2] Unnatural language processing: How do language models handle machine-generated prompts? EMNLP 2023", "claims_and_evidence": "The main claim of the paper is not properly supported by the evidence. The paper claims that the proposed unnatural prompts contain latent features that multiple LMs respond to. However, the relevant information contained in those prompts are not hidden features but keyword tokens inherited from the natural prompts. Table 1 shows examples where some key tokens from the original prompt were kept in the unnatural prompts (numbers and some other tokens like \"stock\", \"price\" or \"Carly\", \"arms\"). The remaining tokens are fillers that the models just ignore. Indeed, the search is initialized with the natural prompts and the algorithm only learns to keep the relevant original tokens and replace the other tokens with junk tokens. The unnatural prompts of the paper are more like noisy versions of the natural prompts. As stated in the limitations section, when the search is initialized randomly and the original tokens filtered, the resulting unnatural prompts are no longer meaningful to the models. If the search algorithm mostly only identifies the key tokens to be kept from the natural prompts, the work is more about feature attribution of natural prompts rather than latent features of unnatural prompts (see Question related to this). There is no analysis characterizing the latent features that are proper to the unnatural prompts (meaning not keywords from the natural prompts).", "methods_and_evaluation_criteria": "A wide range of LMs are used in the experiments. The datasets used to evaluate the models are diverse enough to cover the capabilities of LMs. The exact hyperparameters used for the prompt search, such as number of iterations or number of candidate, are not provided in the paper making the work more difficult to reproduce.", "theoretical_claims": "None", "experimental_designs_or_analyses": "The paper lacks a detailed analysis of the unnatural prompts and the alleged features that they contain.", "relation_to_broader_scientific_literature": "The work is related to the interpretability of LMs. It investigates how LMs process unnatural prompts. It is also related to safety as the unnatural prompts can be used to jailbreak models, and understanding them could help design better defenses.", "essential_references_not_discussed": "The papers \"Prompts have evil twins\" (EMNLP 2024 https://aclanthology.org/2024.emnlp-main.4) and \"Unnatural language processing: How do language models handle machine-generated prompts?\" (EMNLP 2023) present a more extensive analysis of unnatural prompts compared to their natural counterparts.", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "It would help to provide more examples of unnatural prompts in the appendix. \n\nMinor comments:\nThe term \"semantic meaning\" is a tautology.", "questions_for_authors": "1- What are the baseline scores (random and empty) for Table 2? \n\n2- Only the context is unnatural in the Section 3 experiments. The questions are still natural. What would happen if the questions were also unnatural? \n\n3- What is the relationship between token overlap with original prompts and performance? What is the overlap of numbers for SimGSM8K?\n\n4- What is the performance of a baseline that would only drops some tokens (selected for example using some feature attribution method) from the natural prompts?\n\n5- Does memorization play a role in the processing of unnatural prompts? The models seem to easily recognize and even translate back the unnatural prompts. Is it possible that some of these prompts were just memorized by the models.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors examine the impact instruction tuning on unnatural languages has on LLM capabilities (restricted to QA settings). They argue that LLMs’ ability to learn from unnatural data is due to latent features in the data that connect to semantics of natural language. Their main contributions are (1) a gradient-based automatic conversion approach for generating unnatural languages from natural ones and (2) comprehensive analysis showing there is better performance from unnatural languages under this approach than truly random sequences (in fact the average performance is on-par with natural languages). This indicates LLMs can adequately learn semantic patterns from noisy data. The authors have acted in good faith to address the reviewers’ concerns within the scope of this paper, which were mostly clarifications, and share their code. In particular, I appreciate their inclusion of a feature attribution / word dropping baseline in the rebuttal which addresses the potential confounding factor of keyword overfitting. I believe this to be an interesting finding with value for both the computational linguistics and AI communities.
Decision: Accept (poster)