MIND SCRAMBLE: UNVEILING LARGE LANGUAGE MODEL PSYCHOLOGY VIA TYPOGLYCEMIA

## Abstract

Abstract Although still in its infancy, research into the external behaviors and internal mechanisms of large language models (LLMs) has shown significant promise in addressing complex tasks in the physical world. These studies suggest that powerful LLMs, such as GPT-4, are beginning to exhibit human-like cognitive abilities, including planning, reasoning, and reflection, among others. In this paper, we introduce an innovative research line and methodology named LLM Psychology, which leverages or extends human psychology experiments and theories to investigate cognitive behaviors and mechanisms of LLMs. Practically, we migrate the Typoglycemia phenomenon from psychology to explore the “mind” of LLMs. To comprehend scrambled text in Typoglycemia, human brains rely on context and word patterns, which reveals a fundamental difference from LLMs’ encoding and decoding processes. Through various Typoglycemia experiments at the character , word , and sentence levels, we observe the following: (I) LLMs demonstrate human-like behaviors on a macro scale, such as slightly lower task accuracy with consuming more tokens and time; (II) Different LLMs show varying degrees of robustness to scrambled input, making it a democratized benchmark for model evaluation without crafting new datasets; (III) The impact of different task types varies, with complex logical tasks ( e.g., math) in scrambled format being more challenging. Going beyond these, some misleadingly optimistic results suggest that LLMs are still primarily data-driven , and their human-like cognitive abilities may differ from what we perceive; (IV) Interestingly, each LLM exhibit its unique and consistent “cognitive pattern” across various tasks, unveiling a general mechanism in its psychology process. To conclude, we provide an in-depth analysis of hidden layers on a micro scale to explain these phenomena, paving the way for LLMs’ deeper interpretability and future research in LLM Psychology . 1 1 1 Our code is available at https://github.com/Ymm-cll/Typoglycemia

## 1 Introduction

“Typoglycemia refers to the pheonmneon where poeple can raed text even when the lettres in the midlde of wrods are scrambled, as long as the fisrt and last letters are in the crorect poistion.”

Do you notice that some words in the above explanation to Typoglycemia have letters in the wrong order? [pheonmneon, poeple, raed, …] These words contain certain misplaced letters, yet we can still recognize them. This phenomenon, known as Typoglycemia, is widespread in human reading and is used in psychology experiments to study human language cognition (Johnson et al., ; Rayner et al., ). With recent development of large language models (LLMs), they demonstrate “human-like” capabilities and open a potential path for the upcoming artificial general intelligence, excelling in complex tasks such as tool using (Yuan et al., ), reasoning (Hao et al., ), planning (Kalyanpur et al., ), and role-playing (Chen et al., ). However, research on the underlying cognitive mechanisms of LLMs remains in its infancy. Whether LLMs possess deep thinking and human-like cognition is an unsolved mystery that still looms over researchers (Binz & Schulz, ; Bender et al., ). Thus, we try to reveal this by exploring an intriguing question: “Does LLM possess human-like cognitive processes and mechanisms in reading and comprehending?”

To this end, this work aims to investigate the “human-like phenomena” demonstrated by LLMs and provide insights into whether these models truly possess cognitive capabilities or merely exhibit them in a statistical sense. To delve deeper into existing LLMs research, we categorize the off-the-shelf studies into three main research lines: (I) Single-LLM, where external human-like thought processes are applied to LLM through methods such as prompt engineering to achieve better performance (Liu et al., ). For instance, Wei et al. (); Yao et al. (); Besta et al. () simulate human-like reasoning by guiding LLMs through intermediate thought steps in the structure of chain/tree/graph. (II) Multi-LLMs (Agents), where interactions between multiple LLMs are used to explore their behavior and logic in complex communications, such as cooperative (Qian et al., ; Shen et al., ) and competitive (Zhao et al., ) scenarios, etc. (III) Notably, a small but growing body of work aims to investigate the intrinsic cognitive mechanisms of LLMs. Through cognitive science methods, Almeida et al. () investigates LLMs’ moral reasoning, while Zhang et al. () study collaboration mechanisms among LLM-based agents.

Figure: Figure 1: (Upper Left) The two-step process by which humans handle scrambled text. (Lower Left) The performance comparison among Human, Llama-3.1, GPT-4 on BoolQ dataset in both original and scrambled task description. We observe a widespread phenomenon of maintaining high accuracy, along with counter-intuitive improvements in certain cases. (Right) We draw the two-step process by which LLMs handle scrambled text to parallel with human. Scrambled text here is simple examples for better illustration. See our practical scrambled text cases for experiments in Appendix G.
Refer to caption: extracted/5950358/Figure/intro.png

However, Line (I) merely focuses on leveraging LLMs’ human-like abilities to solve real-world problems, while overlooking deeper investigations into why LLMs exhibit such capabilities. This preconceived notion of equating LLMs with humans may overlook their limitations and misuse risks, leading to unreliable outcomes. Line (II), constrained by specific parameters and settings, operates only within particular scenarios. This limitation results in reduced flexibility, as it prevents the agents from being adapted to diverse and unpredictable contexts beyond its predefined scope. Similarly, while Line (III) has begun to explore LLMs from a cognitive perspective, they remain limited to fixed and external scenarios such as moral reasoning or human simulation (Almeida et al., ; Petrov et al., ). What all of these studies lack is the systematical exploration of the generalized and intrinsic cognitive mechanisms of LLMs.

Insights. In this paper, we propose a new research line and concept: LLM Psychology, which follows and extends human psychology methods to explore and study LLMs. In practice, we use Typoglycemia as a lens to investigate the universal and underlying mechanisms of LLMs in comprehending. Psychologists, analyzing human behaviors in Typoglycemia scenarios, have explored human visual mechanisms, contextual reasoning, and language patterns (Agrawal et al., ; Caffarra et al., ). They discover that human reading relies on the overall shape of words and familiar patterns, enabling self-correction and holistic interpretation of scrambled text (Rayner et al., ). In a parallel vein, LLMs’ tokenization algorithms, such as Llama’s BPE (Sennrich, ; Touvron et al., ), shroud the inner mechanisms. Consequently, by applying Typoglycemia (not transcription errors) to LLMs, similar to what psychologists do with humans, we can explore whether LLMs demonstrate “human-like” performance and mechanisms from appearance to essence.

In practice, we first align humans with LLMs when processing scrambled text in Figure 1. We then naturally extend original Typoglycemia from character to word and sentence levels. To systematize subsequent study, we design the standardized experiment pipeline (referred as TypoPipe), which explores multi-dimensional performances through various tailor-made tasks in scrambled text(referred as TypoTasks). TypoPipe is deployed across 5 datasets on Llama-3.1, Gemma-2 and GPT families. Some interesting and counter-intuitive findings are as follows: ♣ LLMs exhibit human-like behavior in TypoTasks, demonstrating a retained ability to comprehend scrambled text, albeit at a higher computational cost. ♠ The emergent human-like abilities of LLMs are fundamentally statistical and data-driven, rather than genuinely resembling human cognition. As shown in the lower-left portion of Figure 1, GPT-4 displays an abnormal improvement ($0.2\sim 2.2\%\uparrow$) on scrambled text that is typically more challenging. ♥ Further experiments reveal a strong correlation between hidden layer semantics and model performance, indicating that transformers’ focus on certain displaced information in scrambled text may drive this unexpected improvement statistically. ♠ Each LLM exhibits its unique and consistent hidden layer semantics distribution across different Typoglycemia tasks. This mirrors how individual humans possess their own unique cognitive patterns.

In summary, our core contributions can be listed as follows:

- ❶
New Direction. We propose “LLM Psychology” as an interdisciplinary framework with significant research depth, offering novel methodologies, directions and insights for the future study of LLM’s human-like cognition. To the best of our knowledge, we are the pioneer to systematically transfer cognitive psychology methodologies and experiments to LLMs, assessing the similarities and differences between LLMs and humans from a cognitive psychological perspective.
- ❷
Comprehensive Experiments. We extend the original Typoglycemia experiments in psychology and adapt them to LLMs, using tailor-designed TypoPipe and TypoTask frameworks, we conduct extensive experiments on 8 models across 5 datasets, testing over 20 types of scrambled text at character, word, and sentence levels with distinct reordering, inserting, and deleting operations. Our results align LLMs’ performance and cost changes with human behaviors in these scenarios.
- ❸
Deep Analysis. We report LLMs’ unique “cognitive pattern” and anomalous behaviors. We explore the underlying causes through an analysis of hidden layer semantics in the encoder and decoder. Our findings demonstrate that LLMs’ emergent human-like abilities are driven by data and statistics, providing strong evidence that their “cognitive process” differ from that of humans.
- ❹
Democratized Benchmark. We present an innovative, implementable, yet effective benchmark based on the Typoglycemia method to evaluate LLMs’ capabilities based on existing datasets. Our experimental results reveal varying degrees of robustness across different LLMs, validating that our benchmark correlates well with commonly accepted assessments of their ability.

## 2 Related Work

Human-like Mechanisms of LLMs.
LLMs (Touvron et al., ; Achiam et al., ; Wang et al., ) have revolutionized both academic and industrial research paradigms, owing to their exceptional and human-like capabilities (Wei et al., ; Wang et al., ). Building on these foundational studies, numerous research efforts integrate mechanisms such as memory, role-playing and tool using to fully leverage these human-like capabilities (Wei et al., ; Bubeck et al., ; Hong et al., ; Li et al., ; Chen et al., ; ). Several studies explore the similarities between LLMs and human cognitive mechanisms. McCoy et al. () investigates model’s cognitive intuitions in reasoning tasks. Chowdhery et al. () analyzes PaLM’s memory mechanisms, highlighting its human-like processes in retrieval and question answering. Though promising, there is still a debate that whether LLMs truly understand language or merely rely on data-driven patterns (Bender et al., ). We use Typoglycemia as a psychological probe to uncover the superficial performance and underlying mechanisms of LLMs and present “LLM Psychology” with the first shot.

Cognitive Concepts in LLMs.
Due to the similarity between LLMs and humans, many studies have been inspired by human cognitive process to enhance LLMs (Bubeck et al., ; Wei et al., ). For example, SwiftSage (Lin et al., ) improves the task capabilities of LLM-based agents in dynamic environments by mimicking the dual-process theory of human cognition. Xie et al. () investigates whether LLMs can simulate human cognitive trust behaviors by employing a series of trust games. AvalonBench (Light et al., ) evaluates LLMs’ competency levels through the Resistance Avalon game, which involves cognitive strategies. PsySafe (Zhang et al., ) further explores the impact of cognitive states as prompts on the safety of LLM-agent. In this work, inspired by the Typoglycemia psychological experiment, we explore the underlying cognitive process of LLMs by comparing their micro and macro level performance with that of humans.

Human Reading and Typoglycemia.
An interesting phenomenon is that humans can maintain an understanding of the general meaning of scrambled text, a capability that some studies attribute to the brain’s mechanisms of holistic shape perception and pattern recognition (Miller, ; Rayner et al., ; Perea & Lupker, ; Shaywitz & Shaywitz, ; Rayner et al., ). As LLMs’ powerful understanding capabilities have been recognized, a few studies attempt to explore whether LLMs exhibit similar “human-like” phenomena. Cao et al. () investigates the exceptional performance of LLMs in reconstructing character-level scrambled text. Singh et al. () finds that LLMs can still maintain encoding consistency when confronted with such text. However, previous work has merely showcased related phenomena without delving deeply into the underlying mechanisms of LLMs. In our research, we systematically migrate the Typoglycemia phenomenon across multi-granularity to LLMs and provide a comprehensive explanation for its underlying causes.

Figure: Figure 2: TypoBench Overview. TypoPipe and TypoTask form the two components of our benchmark. The overall pipeline consists of 4 steps: Calibration, Navigation, Refabrication, and Refinement. TypoTask consists of two task categories: TypoC and TypoP which emphasize performance and perception, respectively.
Refer to caption: extracted/5950358/Figure/framework.png

## 3 Can LLMs recognize Tpyogylcmeia as Typoglycemia?

To apply core principles of LLM Psychology, we migrate and extend the Typoglycemia concept from psychology by proposing the calibrated benchmark (TypoBench), as shown in Figure 2 . Concretely, TypoBench consists of two components: (1) Typoglycemia Pipeline that provides standardized experiment workflow on LLMs (Sec 3.1) and (2) Typoglycemia Task that challenges LLMs’ all-around abilities to address scrambled text (Sec 3.2), with reasons for its specific design in Sec 3.3.

### 3.1 Typoglycemia Pipeline (TypoPipe)

In this section, we introduce the generalized framework TypoPipe to standardize the experimental process. TypoPipe divides the entire pipeline into the following 4 steps: ① Calibration aims to comprehensively select and calibrate datasets for a thorough evaluation of LLMs’ ability. ② Navigation targets to design reasonable functions (TypoFunc) to transform each data into various types of “Typoglycemia” text. ③ Refabrication. This process perform the original task on the dataset or design other scenarios to explore LLMs’ versatile performances. ④ Refinement consists of iteratively calculating metrics, analysing results and refining experiment settings for final conclusions.

Formulations. First, we provide denotations for further formulations. Let $\mathcal{C}$ be the character set, then the text set is $\mathcal{T}=\{c_{1}c_{2}\ldots c_{n}\mid c_{i}\in\mathcal{C},1\leq i\leq n\}$.
Then we denote a dataset with questions and answers as $\mathcal{D}=(\mathcal{Q},\mathcal{A})=\{(q_{i},a_{i})|q_{i}\in\mathcal{Q},a_{i
}\in\mathcal{A},1\leq i\leq m\}$, and LLM as a function $\mathrm{M}:\mathcal{T}\to\mathcal{T}$. For any finite set $\mathcal{X}$, we use $x_{i}(1\leq i\leq|\mathcal{X}|)$ to refer to its element for convenience.

① Calibration: We denote the family of datasets as $\mathscr{D}=\{\mathcal{D}_{1},\mathcal{D}_{2},\ldots,\mathcal{D}_{t}\}$. Calibration aims at:

$$ $\arg\max_{\mathscr{D}}\Bigg{[}\alpha\sum_{\scriptscriptstyle 1\leq i<j\leq t} \mathrm{Difference}(\mathcal{D}_{i},\mathcal{D}_{j})+\beta\sum_{i=1}\mathrm{ Quality}(\mathcal{D}_{i})\Big{]}$ (1) $$

Eq 1 seeks to select distinct and premium datasets to challenge LLMs from multi-aspects. In practice, we heuristically select tailored datasets from distinct fields. (See in Appendix B.1).

② Navigation: Indicate the binary set $(opt,gra)$ as $\Omega$, where $opt:\mathcal{T}\to\mathcal{T}$ is the text operation (reorder, insert, delete, etc.) and $gra\in\{\mathrm{character,word,sentence}\}$ is the smallest operational unit (granularity) for $opt$. We define TypoFunc $\mathcal{F}_{\Omega}:\mathcal{T}\to\mathcal{T}$, where $\forall t\in\mathcal{T}$,

$$ $\mathcal{F}_{\Omega}(t)=\mathrm{Recombine}\left(\{opt(s_{i})\mid s_{i}\in \mathrm{Split}_{gra}(t)\}\right),$ (2) $$

where function $\mathrm{Split}_{gra}:\mathcal{T}\rightarrow\mathcal{T}^{*}$ maps text into a set of tokens split at the specified granularity level and function $\mathrm{Recombine}:T^{*}\rightarrow T$ recombines tokens into text.

③ Refarbrication: Based on datasets and functions from previous two steps, we then apply them to get the Typoglycemia prompts. Concretely, let $\mathrm{P}:\mathcal{T}\rightarrow\mathcal{T}$ be the function that transforms data into prompts under certain task scenarios (See examples in Appendix C). For any function $f$ and set $\mathcal{X}=\{x_{1},x_{2},\ldots,x_{|\mathcal{X}|}\}$, denote $f(\mathcal{X})=\{f(x_{1}),\ldots,f(x_{|\mathcal{X}|})\}$ as applying $f$ to all individual elements in $\mathcal{X}$ respectively. For any dataset $\mathcal{D}=(\mathcal{Q},\mathcal{A})$, we define refarbrication step $\mathcal{T}\to\mathcal{T}$ as:

$$ $\mathcal{P}=\mathrm{P}\big{(}\mathcal{F}_{\Omega}(\mathcal{Q})\big{)}\;\; \mapsto\;\;\mathcal{A}_{\text{pred}}=\mathrm{M}(\mathcal{P})$ (3) $$

Eq 3 expresses the process of converting the original texts in dataset to the Typoglycemia text tasks and get corresponding responses from LLMs. $\mathcal{A}_{\text{pred}}$ is the LLMs’ solutions or answers to inputs.

④ Refinement: After step ③, we utilize evaluation function $\mathrm{E}^{\mathcal{D}\to\mathbb{R}}$ to quantify LLMs’ performance on corresponding tasks. The accuracy evaluation metrics is:

$$ $\mathrm{E}_{\text{acc}}(\mathcal{A}_{\text{pred}},\mathcal{A})=\frac{1}{| \mathcal{A}|}\sum\mathrm{I}(a_{\text{pred}},a),\quad\text{where}\quad\mathrm{I }(x,y)=\begin{cases}1,&\text{if }x=y\\ 0,&\text{otherwise}\end{cases}$ (4) $$

Here $\mathrm{E}_{\text{acc}}$ represents the accuracy between LLMs’ answers and correct answers. Since accuracy only evaluates the final results instead of intermediate thinking process of LLMs, we import a new metric to asses the semantic similarity of hidden states and representations in Transformers. Denote the embedding function as $\mathrm{V}:\mathcal{T}\to\mathbb{R}^{d}$. The semantic similarity evaluation metrics is:

$$ $\mathrm{E}_{\text{sim}}(\mathcal{A}_{\text{pred}},\mathcal{A})=\frac{1}{| \mathcal{A}|}\sum\frac{\mathrm{V}(a_{\mathrm{pred}})\cdot\mathrm{V}(a)}{|| \mathrm{V}(a_{\mathrm{pred}})||\cdot||\mathrm{V}(a)||},$ (5) $$

where $x\cdot y$ denotes the dot product of vectors and $||z||$ denotes the Euclidean norm of vector. $\mathrm{E}_{\text{sim}}$ assesses the cosine similarity of LLMs’ output with standard answers from a semantic view.

Finally, for any dataset $\mathcal{D}=(\mathcal{Q},\mathcal{A})$, a complete iteration of TypoPipe is represented as:

$$ $\mathrm{TP}(\mathcal{D},\mathcal{F}_{\Omega},\mathrm{M},\mathrm{P},\mathrm{E}) =\mathrm{E}\Big{[}\mathrm{M}\big{(}\mathcal{F}_{\Omega}(\mathrm{P}(\mathcal{Q} ))\big{)},\mathcal{A}\Big{]}$ (6) $$

$\mathrm{TP}$ is the function representing the whole TypoPipe. Going beyond this, we propose some metrics to evaluate LLMs’ ability from the Typoglycemia perspective:

$$ $\mathbb{T}_{\text{abs}}=\sum_{i=1}^{u}\alpha_{i}\cdot\mathrm{TP}(\mathcal{D}_{ i},\mathcal{F}_{\Omega},\mathrm{M},\mathrm{P},\mathrm{E})\text{ or }\sum_{j=1} ^{v}\alpha_{j}\cdot\mathrm{TP}(\mathcal{D},\mathcal{F}_{\Omega_{j}},\mathrm{M} ,\mathrm{P},\mathrm{E})$ (7) $$

$$ $\mathbb{T}_{\text{rel}}=\sum_{i=1}^{w}\alpha_{i}\cdot\frac{\mathrm{TP}( \mathcal{D}_{i},\mathcal{F}_{\Omega},\mathrm{M},\mathrm{P},\mathrm{E})}{ \mathrm{TP}(\mathcal{D}_{i},\mathcal{F}_{\ddagger},\mathrm{M},\mathrm{P}, \mathrm{E})},$ (8) $$

where $\sum\alpha=1,\alpha\in[0,1]$ and $\mathcal{F}_{\ddagger}$ refers to the identity transformation. $\mathbb{T}_{\text{abs}}$ and $\mathbb{T}_{\text{rel}}$ evaluate LLM’s absolute and relative performances on various datasets or TypoFuncs, respectively.

### 3.2 Typoglycemia Task = TypoC + TypoP

Building upon the standardized TypoPipe workflow, we have carefully designed TypoTask, which targets at assessing LLMs’ performance in specific Typoglycemia-related tasks, along with their ability to comprehend and correct scrambled text. Specifically, TypoTask consists of the following two categories of tasks: Typoglycemia Completion (TypoC) and Typoglycemia Perception (TypoP).

TypoC refers to performing native tasks on the dataset. For example, the native task of GSM8K (Hendrycks et al., ) is to solve mathematical problems. TypoC reflects LLMs’ ability to comprehend and follow scrambled text prompt when addressing problems in specific fields. To further explore the extent to which LLMs understand Typoglycemia text (scrambled text), we design TypoP consisting of Rectify, Summarize, and Translate. Rectify task aims to restoring Typoglycemia text back to its original form, assessing the model’s ability to locally identify and rectify such errors. Summarize and Translate tasks require summarizing and translating, respectively, which evaluates the model’s ability to understand the global context and detailed information in Typoglycemia text. See tailor-selected TypoC and TypoP tasks in Appendix B, C.1, and C.2.

### 3.3 Why Completion and Perception?

Methodologically, TypoC is designed to evaluate the behavioral performance of LLMs, while TypoP aims to assess their perception and understanding, drawing inspiration from behavioral psychology and cognitive psychology, respectively. In doing so, we provide a vivid example of how psychological principles can be applied to understand and evaluate LLMs via our proposed“LLM Psychology”. These two tasks explore the impact of Typoglycemia on LLMs from both fine-grained and coarse-grained perspectives, progressing from shallow to deep levels of analysis. To successfully complete these tasks, models must simultaneously grasp local (scrambled content) and global information (contextual semantics) in order to fully comprehend the task’s details and objectives. Based on the TypoBench framework, Eq 7 and Eq 8, we propose a more general and concise method for evaluating LLMs based on existing datasets, which reflects abilities not explored in previous research:

$$ $\mathbb{T}_{\text{gen}}=\frac{\mathrm{E}(\mathrm{M},\mathcal{F}(\mathcal{D}))} {\mathrm{E}(\mathrm{M},\mathcal{D})}$ (9) $$

Eq 9 means using metrics $\mathrm{E}$ to evaluate model $\mathrm{M}$ on dataset $\mathcal{D}$ before and after being applying function $\mathcal{F}$. We present division to quantify the impact of $\mathcal{F}$. In our work, $\mathcal{F}$ is Typoglycemia.

## 4 Experiment

We employ TypoPipe across various scenarios to comprehensively study the impact of Typoglycemia on LLMs. The experiments are designed to investigate the following research questions:

- •
RQ1: What is the impact of Typoglycemia on existing LLMs?
- •
RQ2: How do other Typoglycemia Functions (e.g., insertion and deletion) impact LLMs?
- •
RQ3: What are the effects of increasing the scrambling ratio of Typoglycemia?
- •
RQ4: Why do LLMs align with human performance under Typoglycemia?

### 4.1 Experimental Setups

Datasets. We aim to evaluate LLM Psychology across various task settings, including mathematics, code generation, situational question answering, and commonsense reasoning. Concretely, as for scenarios requiring strong logical reasoning, we select GSM8k (Hendrycks et al., ) for math and MBPP (Kocetkov et al., ) for code. Additionally, we explore the impact of Typoglycemia on LLMs’ emergent situational learning and knowledge capabilities. We select BoolQ (Clark et al., ) and SQuAD (Rajpurkar et al., ) dataset for situational question answering tasks. For commonsense reasoning, we use CSQA (Talmor et al., ) dataset, a multiple-choice commonsense dataset. More descriptions on dataset can be found in Figure 1, Appendix B, and C.

TypoFuncs ($\mathcal{F}_{\Omega}$) transform the above datasets into Typoglycemia texts. To extend psychological Typoglycemia, we execute $\mathcal{F}_{\Omega}$ at character, word, and sentence levels, allowing us to explore the sensitivity of LLMs to various text variations. Specific $\mathcal{F}_{\Omega}$ operations include reordering, inserting, and deleting (refer to as REO, INS, and DEL, respectively). Operation X can be applied in different positions or ways of the three levels, such as: all (X-ALL), internal (X-INT), adjacent (X-ADJ), beginning (X-BEG), ending (X-END), and reversing (X-REV). Utilizing well-designed $\mathcal{F}_{\Omega}$, our Typoglycemia experiment contains both mildly scrambled text and highly disordered text that is nearly unrecognizable to humans. The specific operations instances can be found in Appendix D.

Models and Metrics. We extensively evaluate our concept across diverse LLMs within zero-shot setting, including Gemma-2 (2B, 9B and 27B) (Team et al., ), Llama 3.1 (8B, 70B) (Touvron et al., ), GPT-3.5-Turbo(^2^22[https://platform.openai.com/docs/models/gpt-3-5-turbo](https://platform.openai.com/docs/models/gpt-3-5-turbo)), GPT-4o-mini(^3^33[https://platform.openai.com/docs/models/gpt-4o-mini](https://platform.openai.com/docs/models/gpt-4o-mini)) and GPT-4o(^4^44[https://platform.openai.com/docs/models/gpt-4o](https://platform.openai.com/docs/models/gpt-4o)). The selection of these models and their corresponding sizes provides a comprehensive “model zoom”. In our settings, we choose accuracy and cosine similarity as metrics. For accuracy, we consider a response correct only when the LLM’s output exactly matches the correct answer. For cosine similarity, we embed the reasoning processes into vectors using the text-embedding-3(^5^55[https://platform.openai.com/docs/models/embeddings](https://platform.openai.com/docs/models/embeddings)) and calculate cosine similarity with the standard process. The model parameter settings for reproducibility can be found in Appendix E.

### 4.2 Main Results (RQ1)

To answer RQ1, we compare different Typoglycemia concepts across various models and datasets. We apply random reordering and run the experiments multiple times, reporting the mean values. The experimental observations (Obs) are as follows and experiment discussion is placed in Appendix A:

Table: Table 1: Main results on the TypoC tasks when $\mathcal{F}_{\Omega}=$ REO on the character, word and sentence level. We evaluate the average task accuracy (over 3 runs) of various LLMs on the GSM8k, BoolQ, and CSQA datasets. BASE refers to the scenario where $\mathcal{F}_{\Omega}$ is not applied to the task description. With the same coefficient weights, $\mathbb{T}_{\text{abs}}$ (Eq 7) shows each row’s average accuracy, evaluating general performance across various TypoFuncs. $\mathbb{T}_{\text{rel}}$ calculates Eq 8, quantifying the ability retaining ratio compared with BASE. In each dataset, red (blue) marks the maximum value in each row (column), and green marks values that are the maximum in both. Gray marks the values that are higher than BASE in each row. Several TypoC cases are shown in Appendix G.

Figure: Figure 3: Token and time consumption ratio before and after being processed by TypoFunc when $\mathcal{F}_{\Omega}=$ REO-INT on character level for BoolQ dataset.
Refer to caption: extracted/5950358/Figure/usage.png

Obs.1. Typoglycemia generally leads to a decline in model performance, with more advanced models being less affected. As shown in Table 1, red markers predominantly appear in the BASE column, indicating that accuracy tends to decrease after applying $\mathcal{F}_{\Omega}$. The performance retention of models within the same series increases with model size. For instance, on GSM8k dataset, the Gemma-2 series exhibits an increase in average accuracy across scales, with retention rates of $41.7\%\rightarrow 70.2\%$. Furthermore, the SOTA model GPT-4o (more than 80% of the blue markers) retains an average of 87.8% of its capability, whereas the weakest model, Gemma-2-2B, retains only 41.7%. This aligns with that of humans (Rayner et al., ; Frost, ) in Typoglycemia scenarios and opens up a new avenue for evaluating model capabilities (more results are in Appendix F.1.1).

Obs.2. The degree to which LLMs’ performance is affected is positively correlated with the logical complexity of the TypoC task. In Table 1, gray markers are only seen in the BoolQ/CSQA (reasoning tasks), where LLMs retain 95.6% and 81.7% of their capabilities, respectively, compared to just 67.8% on the math (GSM8k) task, which demands more complex logical reasoning. Notably, for the yes/no BoolQ dataset, applying sentence-level $\mathcal{F}_{\Omega}$ results in an unusual slight average 0.7%$\uparrow$ in accuracy. However, for humans, reading scrambled text typically hampers comprehension (Ferreira et al., ). This performance improvement may be misleadingly optimistic, suggesting that LLMs might rely on the attention mechanism to capture certain representations from scrambled text that help derive correct results. This statistically-driven mechanism vastly differs from the micro-level processes of human reading and understanding.

Obs.3. The position of characters affects LLMs’ understanding differently. As shown in Table 1, the accuracy of character-level $\mathcal{F}_{\Omega}$ under the ALL setting is 100% lower than that of INT, while the accuracy of BEG is lower than that of END in 87.5% of the cases. This indicates that the importance of the first, last, and internal characters decreases in that order—which further reveals the similarity that both LLMs and humans pay more attention to the first and last characters (Perea & Lupker, ) (more results on another two datasets are placed in Appendix F.1.1).

Obs.4. Typoglycemia leads to an increased computational cost. As shown in Figure 3, the ratio of tokens and time before and after the $\mathcal{F}_{\Omega}$ transformation is greater than 1 in 100% of cases for all LLMs. For instance, GPT-3.5-Turbo exhibits a 46.5% $\uparrow$ in prompt tokens and an 18% $\uparrow$ in completion time. Similarly, (Rayner et al., ) finds that humans also require more eye fixations and longer fixation durations when reading Typoglycemia text. This finding reveals that both LLMs and humans struggle in Typoglycemia scenarios (more results are shown in Appendix F.3).

Obs.5. The results of TypoP are consistent with those of TypoC. As Table 2 shows, the performance ranking of 7 out of 8 LLMs closely mirrors that in Table 1. For example, Llama-3.1 shows the same accuracy ranking in both tables: $END>BEG>INT>ALL>REV$. This observation reveals that the robustness of LLMs in Typoglycemia scenarios is positively correlated with their ability to correct Typoglycemia text (See more results on another two TypoPs in Appendix F.2).

Table: Table 3: Results on the TypoC tasks when $\mathcal{F}_{\Omega}=$ INS and DEL on character levels. We apply $\mathcal{F}_{\Omega}$ at the begin and end of each word. We report the average accuracy (over 3 runs) of various LLMs on the GSM8k, BoolQ, and CSQA datasets. BASE means $\mathcal{F}_{\Omega}$ is not applied. $\mathbb{T}_{\text{abs}}$ shows each column’s average accuracy, while $\mathbb{T}_{\text{rel}}$ calculates our proposed metrics with equal weights. In each dataset, red marks the maximum in columns. Gray marks the values that are higher than BASE in columns. TypoC cases are in Appendix G.

### 4.3 Impact of Typoglycemia Functions (RQ2)

To answer RQ2, we conduct experiments using additional insertion and deletion Typoglycemia functions to verify the impact of other Typoglycemia concepts on LLMs. We list the results in Table 3 (more results are placed in Appendix F.1.2) and we can summarize the observations:

Obs.1. The impact of Insertion and Deletion on LLMs is generally similar to Reordering, but the magnitude of the impact is smaller. As shown in Table 3, red markers are primarily concentrated in the BASE row, and the SOTA model GPT-4o retains an average accuracy of 98.9% across the three datasets, while the weakest model, Gemma-2-2B, achieves 83.6%. In all cases, the average retained accuracy increases by $0.1\sim 26.8\%$ compared to Reordering. This indicates that LLMs are more robust to minor additions or deletions of characters than to character reordering.

Obs.2. Insertion and Deletion also result in an unusual increase in accuracy for tasks with weaker logic. As shown in Table 3, 90.9% of the gray markers appear in the BoolQ dataset, which is consistent with the pattern observed in Reordering. This observation further confirms that minor perturbations in the prompt can aid models in understanding simple logical problems.

Obs.3. LLMs exhibit sensitivity to character position for Deletion, but are less sensitive for Insertion. As shown in Table 3, in the case of the INS operation, $BEG<END$ occurs in 54.2% of cases, whereas for the DEL operation, this ratio rises to 83.3%, a 29.1% increase. This observation reinforces the finding in RQ1 that the first letter is more important than the last, and reveals that their relative importance can vary depending on the operation.

Figure: Figure 4: The line charts of accuracy for each model, as the number of operations increase from 1 to 4 when $\mathcal{F}_{\Omega}=$REO_INT, INS_INT, and DEL_INT at character level on GSM8k dataset.
Refer to caption: extracted/5950358/Figure/ratio_swap.png

### 4.4 Scrambling ratio of Typoglycemia (RQ3)

To address RQ3, we gradually increase the number of reordering, inserting, and deleting operations applied to each word’s internal characters to increase the scrambling ratio of texts. The corresponding results are shown in Figure 4 and Appendix F.4, of which we derive the following observations:

Obs.1. As the scrambling ratio increases, the TypoC task becomes more challenging for LLMs. As shown in Figure 4, with the increasing number of operations, the accuracy of LLMs generally shows a downward trend across all three cases, with a drop ranging from 0.3% to 14.2%. We highlighted the regions with the largest decreases in yellow. This observation aligns with human behavior (Just & Carpenter, ), indicating that as the internal structure of the text becomes more disordered, it becomes increasingly difficult for LLMs to understand the text.

Obs.2. Different models exhibit varying levels of resistance to scrambling text. As shown in Figure 4, Llama-3.1-8B demonstrates the weakest robustness, while GPT-4o shows the strongest anti-Typoglycemia ability. The absolute values of the average slope in accuracy for the Inserting operation are 10.9 and 1.2, respectively. This robustness can serve as a measure of LLMs’ ability to handle scrambled text, which may offer a new approach for evaluating LLMs.

**Table 4: Encoder Perspective: The cosine similarity between the embedding of normal text and text processed by $\mathcal{F}_{\Omega}$, using text-embedding-3 to get the vectors. BASE is the standard for similarity calculation.**
| Datasets/$\mathcal{F}_{\Omega}$ | Character Level | Word Level | Sentence Level |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | REO-ALL | REO-INT | REO-REV | INS-INT_3 | DEL-INT_3 | REO-ALL | REO-ADJ | REO-REV | REO-ALL | REO-ADJ | REO-REV |
| GSM8k | 0.755 | 0.891 | 0.594 | 0.865 | 0.830 | 0.930 | 0.962 | 0.915 | 0.978 | 0.987 | 0.967 |
| BoolQ | 0.912 | 0.944 | 0.863 | 0.944 | 0.933 | 0.965 | 0.978 | 0.960 | 0.986 | 0.993 | 0.980 |
| CSQA | 0.867 | 0.922 | 0.836 | 0.910 | 0.890 | 0.949 | 0.968 | 0.940 | 0.999 | 0.998 | 0.997 |

Figure: Figure 5: Decoder Perspective: The cosine similarity between the representations of normal text and the text processed by $\mathcal{F}_{\Omega}$ in the SQuAD and BoolQ for each layer of the Llama-3.1-8B model (which has 33 layers in total: 1 word embedding layer and 32 Transformer layers). BASE is the standard for similarity calculation.
Refer to caption: extracted/5950358/Figure/more_heatmap/heatmap_squad_llama-3.1-8b.png

### 4.5 Why do LLMs align with human performance (RQ4)

To answer RQ4, we embed the task text before and after being processed by $\mathcal{F}_{\Omega}$ using the text-embedding-3-large model and then calculate their semantic similarity. Additionally, We calculate the mean of representation tensors between Transformers, aggregating their semantics into vectors to calculate similarity with BASE. Finally, we derive the above Table 4, Figure 5 and observations:

Obs.1. The text before and after $\mathcal{F}_{\Omega}$ processing exhibits a high degree of semantic similarity, with the impact varying depending on the level of text granularity. As shown in Table 4, in the CSQA dataset, Typoglycemia text at the character, word, and sentence level retains an average semantic similarity of 0.885, 0.952, and 0.998, respectively, compared to the unprocessed text (BASE). This indicates that disturbances at the character level have the greatest impact on LLMs’ understanding of the text. This observation suggests that, from the encoder’s perspective, Typoglycemia text preserves a substantial amount of semantic information, which enables LLMs to exhibit robustness similar to humans in Typoglycemia scenarios (more results are placed in Appendix F.5).

Obs.2. The subsequent representations of Typoglycemia text by LLMs are critical to their task performance. As shown in Figure 5, the color of accuracy and representation is closely aligned. For instance, on SQuAD, for the 3 types of $\mathcal{F}_{\Omega}$ at the sentence level, the similarity scores of their representations are all yellow (high similarity), and corresponding accuracy is the highest at 74.4%, 73.2%, and 71.6%, respectively. In contrast, when $\mathcal{F}_{\Omega}=$ Char-REO-REV, the representation similarity score is the lowest (darkest color), with lowest accuracy at 22.0%. This observation demonstrates that, from the decoder’s perspective, the semantic information retained in the representations across the Transformer layers is crucial for LLMs to correctly understand and respond.

Obs.3. The hidden layer representations of the same LLM across different datasets exhibit similar “cognitive patterns.” As illustrated in Figure 5, the color distributions for SQuAD and BoolQ under various $\mathcal{F}_{\Omega}$ appear visually similar. Specifically, the cosine similarity between the concatenated and linearized heatmaps of these two datasets is 0.9994, indicating a high degree of similarity. Additionally, the color distributions vary across different models when evaluated on the same dataset (See more figures in Appendix F.6). Based on these observations, we posit that the heatmap can translate each model’s unique ”cognitive pattern” through our Typoglycemia experiments, much like how different human individuals exhibit distinct cognitive patterns.

## 5 Conclusion

In this paper, we explore the emerging field of LLM Psychology by investigating the behavior of LLMs through the lens of Typoglycemia. Our study reveals how LLMs handle scrambled text, providing insights into their cognitive-like abilities and limitations. Through systematic analysis, we observe that LLMs demonstrate human-like behaviors, such as reduced task accuracy and increased token and time consumption, when faced with text distortions. Additionally, the varying robustness across different LLMs suggests that scrambled text understanding serves as an accessible benchmark for evaluating model performance. Despite some promising results, our analysis of LLMs’ hidden layers reveals their reliance on data-driven mechanisms, with limited capacity for deep reasoning. By digging into the hidden layer semantics, we further reveal that each LLM demonstrates its unique and consistent cognitive pattern across different datasets in Typoglycemia.

## Appendix A Experimental Discussion

Through comprehensive and systematic experiments in Typoglycemia and its migrated scenarios, we discover both the alignment of LLMs with human cognition and their distinct behaviors. LLMs exhibit a decline in task accuracy, increased resource consumption, and many other human-like behaviors, such as placing greater emphasis on initial letters. This significantly advances research on aligning LLMs with human cognition and provides a solid and vivid case for our proposed ”LLM Psychology.” Furthermore, we observe LLMs’ counter-intuitive and counter-logical performance under certain settings, offering strong evidence for the argument that LLMs possess data-driven statistical reasoning abilities rather than human logic. Finally, we explore the underlying causes and observe different LLM’s unique cognitive pattern on these phenomena from the perspectives of encoder and decoder architectures, providing new insights into the cognitive mechanisms of LLMs.

## Appendix B Dataset Description

Table: Table 5: Dataset Details of TypoBench

### B.1 Dataset Selection Strategy

To systematically evaluate the performance of LLMs on TypoBench, we focus on three key capabilities when selecting datasets and task scenarios: logical reasoning, contextual learning, and knowledge acquisition.

#### B.1.1 logical reasoning

Strong Logic Tasks. We refer to tasks that involve multi-step reasoning, where an error in one step leads to errors in subsequent steps, as Strong Logic Tasks (SLTs). Representative scenarios we select include mathematical problem solving (GSM8k as dataset) and code generation (MBPP as dataset). SLTs pose stringent challenges to the logical reasoning capabilities of LLMs. From a data-driven perspective, Typoglycemia disrupts the morpheme order in normal natural language text, which, in turn, disturbs the inherent logic, leading to confused understanding and erroneous reasoning. For humans, the combination of SLT scenarios and Typoglycemia text makes task completion nearly impossible. In a certain sense, this implies that SLTs are effective in testing LLMs’ performance on TypoBench, thereby revealing their underlying cognitive mechanisms.

Weak Logic Tasks.
Conversely, tasks with less stringent requirements for logical correctness are referred to as Weak Logic Tasks (WLTs). These tasks typically challenge LLMs’ capabilities not only in simple logical reasoning but also in other areas. WLTs primarily serve as a platform for simultaneously evaluating multiple aspects of LLMs’ abilities. In our experimental strategy, WLTs are combined with contextual learning and knowledge acquisition.

#### B.1.2 contextual learning

Contextual learning refers to the ability of LLMs to perceive and learn the knowledge, patterns, and other elements within the context of a given prompt. We select task datasets for contextual learning at two levels of difficulty. Given a contextual passage and a related question, LLMs are instructed to answer with either ”yes/no” (BoolQ as dataset) or phrases (SQuAD as dataset), corresponding to easy and difficult settings, respectively. In the yes/no setting, the response is not directly tied to the context, allowing LLMs to rely on coarse-grained semantic understanding. However, in the phrases setting, LLMs are required to have a more localized understanding of the contextual content, posing a more severe challenge to their learning and perception capabilities. By combining these two scenarios with TypoBench, we can explore how Typoglycemia affects LLMs’ ability to perceive both local and global information.

#### B.1.3 knowledge acquisition

LLMs possess knowledge capabilities, which are embedded within their layer weights. Generally, a model activates and extracts the knowledge embedded in these weights through the input prompt, enabling it to generate responses. This raises an interesting question: does Typoglycemia affect this process? In our experimental strategy, we investigate whether Typoglycemia disrupts the extraction of knowledge from the model’s internal weights by evaluating its impact on answering context-independent common sense questions (CSQA as dataset). This approach allows us to explore how the perturbation of input text influences the model’s ability to retrieve knowledge.

### B.2 Dataset Example

#### B.2.1 GSM8K

#### B.2.2 MBPP

#### B.2.3 BoolQ

#### B.2.4 SQuAD

#### B.2.5 CSQA

## Appendix C Task Prompt

### C.1 Task Completion Prompt

#### C.1.1 Mathematical Problem Solving

#### C.1.2 Mathematical Problem Solving

#### C.1.3 Context Question Answering

#### C.1.4 Commonsense Reasoning

### C.2 Task Perception Prompt

#### C.2.1 Rectify

### C.3 Summarize

### C.4 Translate

## Appendix D TypoFunc Description

Psychological experiments on Typoglycemia typically involve transposing the letters at the beginning, end, and internal positions within words. We have extended this operation to a broader set of TypoFuncs at the letter, word, and sentence levels. Additionally, at the character level, we have designed TypoFuncs such as insertion and deletion.

### D.1 Character

At the character level, we treat characters as the smallest operational units. TypoFuncs operate on the letters within each word. The character-level TypoFuncs are divided into three categories: reordering , inserting, and deleting (denoted as R, I, D respectively). Each of these TypoFuncs includes the following specific operations:

- •
First Character R/I/D, which performs corresponding operation on the first letter of each word.
- •
Last Character R/I/D, which performs the respective operation on the last letter of each word.
- •
k Internal Characters R/I/D, which performs the respective operation on k randomly selected internal letters (excluding the first and last) within each word.

Additionally, the reordering category includes the following specific operations:

- •
All Characters Reordering: This operation shuffles all the letters within the word.
- •
Internal Characters Reordering: This operation shuffles the letters in the middle of the word (excluding the first and last letters).
- •
Characters Reversing: This operation reverses the order of all letters within the word.

### D.2 Word

On the word level, we consider words to be the basic operational units. Given the substantial effect of inserting and deleting words on the overall meaning, which can cause either nuanced or substantial semantic redundancy or loss, our primary emphasis is on the following reordering operations:

- •
All Words Reordering, which randomly shuffles the words within each sentence.
- •
Adjacent Words Reordering, which randomly swaps adjacent words within each sentence.
- •
Words Reversing, which reverses the order of words within each sentence.

### D.3 Sentence

On the sentence level, sentences are regarded as the basic operational units. Likewise, because of the considerable influence that sentence insertion and deletion have on textual meaning, our main focus is on the reordering operations detailed below:

- •
All Sentences Reordering, which randomly shuffles the sentences within the text.
- •
Adjacent Sentences Reordering, which randomly swaps adjacent sentences within the text.
- •
Sentences Reversing, which reverses the order of sentences within the text.

## Appendix E Parameter Settings

To ensure stability and consistency in the model outputs, we set $top_p=1$, $n=1$, $frequency_penalty=0$, and $presence_penalty=0$ for all models. The temperature is set to 0 0 for GPT series models, and to $10^{-6}$ for Llama and Gemma series models.

## Appendix F More Results

### F.1 TypoC

In this subsection, we further present the performance of various LLMs on two additional datasets, MBPP and SQuAD, in the TypoC task. The conclusions drawn from these results are consistent with those in the main text, further supporting the findings on the impact of the Typoglycemia scenario on LLMs.

#### F.1.1 Reordering

Table: Table 6: Results on the TypoC tasks when $\mathcal{F}_{\Omega}=$ REO at character, word, and sentence levels. We evaluate the average task accuracy (over 3 runs) of various LLMs on the MBPP and SQuAD datasets. BASE refers to the scenario where $\mathcal{F}_{\Omega}$ is not applied. In each dataset, red (blue) marks the maximum value in each row (column), and green marks values that are the maximum in both. Gray marks the values that are higher than BASE in each row. MBPP and SQuAD report the cosine similarity and accuracy, respectively. BASE of GPT-4o is the standard for similarity calculation. Gemma-2 series and Llama-3.1-8B fail to generate required format of code on MBPP dataset (See TypoC cases in Appendix G)

#### F.1.2 Insertion and Deletion

Table: Table 7: Results on the TypoC tasks when $\mathcal{F}_{\Omega}=$ INS and DEL at character level. We evaluate the average task accuracy (over 3 runs) of various LLMs on the MBPP and SQuAD datasets. BASE refers to the scenario where $\mathcal{F}_{\Omega}$ is not applied. MBPP and SQuAD report the cosine similarity and accuracy, respectively. BASE of GPT-4o is the standard for similarity calculation. In each dataset, red marks the maximum value in each column. Gray marks the values that are higher than BASE in columns. Gemma-2 series and Llama-3.1-8B fail to generate required format of code on MBPP dataset (See TypoC cases in Appendix G)

### F.2 TypoP

In this subsection, we present the performance of various LLMs on two additional tasks in TypoP: Summarize and Translate. The conclusions drawn from these results are consistent with those in the main text: the results of TypoP align with those of TypoC.

Table: Table 8: Results (Cosine Similarity) on TypoP-Summarize and Translate tasks when $\mathcal{F}_{\Omega}$ is set to REO on character level for BoolQ dataset. BASE is the standard for similarity calculation. In each TypoTask, red (blue) marks the maximum value in each row (column), and green marks values that are the maximum in both (See TypoP cases in Appendix H)

### F.3 Completion Time and Prompt Tokens

In this subsection, we present additional results on completion time and prompt token usage. First, we use the different datasets to comprehensively evaluate token and time consumption across different levels of scrambled input. The findings align with the conclusions drawn in the main text, further demonstrating strong parallels with human performance in scrambled reading scenarios.

Figure: Figure 6: Consumption ratio before and after being processed by TypoFunc when $\mathcal{F}_{\Omega}$ is set to REO-ALL on character level for CSQA.
Refer to caption: extracted/5950358/Figure/more_usage/usage_csqa_char_swap_all.png

Figure: Figure 12: Consumption ratio when before and after being processed by TypoFunc $\mathcal{F}_{\Omega}$ is set to DEL-END on character level for GSM8k.
Refer to caption: extracted/5950358/Figure/more_usage/usage_gsm8k_char_add_end.png

Figure: Figure 18: Consumption ratio when before and after being processed by TypoFunc $\mathcal{F}_{\Omega}$ is set to REO-ADJ on sentence level for MBPP.
Refer to caption: extracted/5950358/Figure/more_usage/usage_mbpp_sentence_swap_near.png

### F.4 Scrambling Ratio

In this subsection, we provide charts illustrating the number of Reordering, Inserting, and Deleting operations in various task scenarios across additional datasets, along with LLMs’ task performance. The results shown in these charts are similar to the findings in the main text, further validating the impact of the scrambling ratio on LLMs.

Figure: Figure 22: The line charts of accuracy for each model, as the number of operations increase from 1 to 4 when $\mathcal{F}_{\Omega}=$REO_INT, INS_INT, and DEL_INT on BoolQ datset.
Refer to caption: extracted/5950358/Figure/more_ratio/ratio_boolq_swap.png

Figure: Figure 23: The line charts of accuracy for each model, as the number of operations increase from 1 to 4 when $\mathcal{F}_{\Omega}=$REO_INT, INS_INT, and DEL_INT on CSQA datset.
Refer to caption: extracted/5950358/Figure/more_ratio/ratio_csqa_swap.png

Figure: Figure 24: The line charts of accuracy for each model, as the number of operations increase from 1 to 4 when $\mathcal{F}_{\Omega}=$REO_INT, INS_INT, and DEL_INT on SQuAD datset.
Refer to caption: extracted/5950358/Figure/more_ratio/ratio_squad_swap.png

### F.5 Encoder Embedding

In this subsection, we present the similarity of text embedding across additional datasets and the similarity of input text representations at each layer of the Transformer across more models and datasets. The results shown in these charts are consistent with those in the main text, providing further data to support the related conclusions.

Figure: Table 9: The cosine similarity between the embedding of normal text and text processed by $\mathcal{F}_{\Omega}$, using text-embedding-3 to get the vectors. BASE is the standard for similarity calculation.

### F.6 Decoder Representation

Figure: Figure 25: The cosine similarity between the representations of normal text and the text processed by $\mathcal{F}_{\Omega}$ in the GSM8k dataset for each layer of the Gemma-2-2B model (which has 27 layers in total: 1 word embedding layer and 26 Transformer layers). BASE is the standard for similarity calculation.
Refer to caption: extracted/5950358/Figure/more_heatmap/heatmap_gsm8k_gemma-2-2b.png

Figure: Figure 27: The cosine similarity between the representations of normal text and the text processed by $\mathcal{F}_{\Omega}$ in the BoolQ dataset for each layer of the Gemma-2-2B model (which has 27 layers in total: 1 word embedding layer and 26 Transformer layers). BASE is the standard for similarity calculation.
Refer to caption: extracted/5950358/Figure/more_heatmap/heatmap_boolq_gemma-2-2b.png

## Appendix G TypoC Cases

### G.1 Mathematical Problem Solving (GSM8k)

#### G.1.1 Good Mind

One case of GPT-4o’s correct responses when $\mathcal{F}_{\Omega}=$Char-REO-INT.
Typoglycemia Prompt:
Solve the math problem below:
Problem: Jluie wtnas to gvie her firoatve cusoin a $2345 mtinauon bkie for his btrihday. So far, she has saved $1500. Sncie the bdirthay is slitl a few wekes away, Jilue has time to save eevn more. She palns on miwong 20 lawns, dienliervg 600 npawsreeps, and wianklg 24 of her nhbriegos’ dgos. She is paid $20 for each lawn, 40 cnets per nepawespr, and $15 per dog. Afetr puhsicanrg the bike, how much menoy wlil Jilue have left?
Response in the following format without any other information:
process: {reasoning steps here}
answer_number: {final answer number here}
Response from GPT-4o:
process: 1. Calculate the total amount Julie will earn from mowing lawns: 20 lawns * $20/lawn = $400.
2. Calculate the total amount Julie will earn from delivering newspapers: 600 newspapers * $0.40/newspaper = $240.
3. Calculate the total amount Julie will earn from walking dogs: 24 dogs * $15/dog = $360.
4. Add the amounts from steps 1, 2, and 3 to find the total additional money Julie will earn: $400 + $240 + $360 = $1000.
5. Add the amount Julie has already saved to the total additional money: $1500 + $1000 = $2500.
6. Subtract the cost of the bike from the total money Julie will have: $2500 - $2345 = $155.
### 155
Let Sarah have $x$ cents. Then, Bigrde has $x+50$ cents. Together, they have $x+(x+50)=300$ cents (since $3 is 300 cents). Simplifying, we get $2x+50=300$. Subtract 50 from both sides to get $2x=250$. Divide both sides by 2 to get $x=125$.
answer_number: 155
Original Question:
Julie wants to give her favorite cousin a $2345 mountain bike for his birthday. So far, she has saved $1500. Since the birthday is still a few weeks away, Julie has time to save even more. She plans on mowing 20 lawns, delivering 600 newspapers, and walking 24 of her neighbors’ dogs. She is paid $20 for each lawn, 40 cents per newspaper, and $15 per dog. After purchasing the bike, how much money will Julie have left?
Answer from Dataset:
Mowing lawns will earn Julie 20 * 20 = $\langle\langle 20*20=400\rangle\rangle$ 400 dollars
Her earnings, in dollars, from delivering newspapers will be 600 * 40 / 100 = $\langle\langle 600*40/100=240\rangle\rangle$ 240 dollars
After walking 24 of her neighbor’s dogs, she will earn 24 * 15 = $\langle\langle 24*15=360\rangle\rangle$ 360 dollars.
She will therefore earn a total of 400 + 240 + 360 = $\langle\langle 400+240+360=1000\rangle\rangle$ 1000 dollars.
Combining earnings from her job with her savings will give Julie 1000 + 1500 = $\langle\langle 1000+1500=2500\rangle\rangle$ 2500 dollars.
Subtracting the cost of the mountain bike from the total will leave Julie with a balance of 2500 - 2345 = $\langle\langle 2500-2345=155\rangle\rangle$ 155 dollars.
#### 155

#### G.1.2 Bad Mind

One case of GPT-4o’s incorrect responses when $\mathcal{F}_{\Omega}=$Char-REO-ALL.
Typoglycemia Prompt:
Solve the math problem below:
Problem: iamrSa is the nassatsti oachc of a rccseo aetm gynlpai saingta one of the steb temsa in hreit ueealg. She has rfuo zneod etrwa blsotet dlefli thwi awrte in a box. In the itsfr kbrae of the chmta, the 11 apelsry on the fdlei ceah teka two tltoesb of rtawe from iamaSr’s box, and at the end of the eagm, kate one eorm tetbol chea. How ymna eosbttl of erwat are miinrange in Saimar’s box?
process: {reasoning steps here}
answer_number: {final answer number here}
Response from GPT-4o:
process: 1. Start with 4 bottles of water.
2. Each of the 11 players takes 2 bottles during the first break: 11 players * 2 bottles = 22 bottles.
3. Each of the 11 players takes 1 more bottle at the end of the game: 11 players * 1 bottle = 11 bottles.
4. Total bottles taken: 22 bottles + 11 bottles = 33 bottles.
5. Since there were only 4 bottles initially, and 33 bottles were taken, it implies that the problem is not feasible as described. Therefore, the remaining bottles would be 0.
answer_number: 0
Original Quetsion:
Samira is the assistant coach of a soccer team playing against one of the best teams in their league. She has four dozen water bottles filled with water in a box. In the first break of the match, the 11 players on the field each take two bottles of water from Samira’s box, and at the end of the game, take one more bottle each. How many bottles of water are remaining in Samira’s box?
Answer from Dataset:
If the box has four dozen bottles of water, there are 4 * 12 = $\langle\langle 4*12=48\rangle\rangle$ 48 bottles of water in the box
After the first half, the 11 players take 11 * 2 = $\langle\langle 11*2=22\rangle\rangle$ 22 bottles of water from the box.
If they take 11 more bottles of water at the end of the game, the number increases to 22 + 11 = 33 bottles of water taken.
Without the bottles taken, there are 48 - 33 = $\langle\langle 48-33=15\rangle\rangle$ 15 bottles of water remaining in the box.
#### 15

### G.2 Context Question Answering with yes/no (BoolQ)

#### G.2.1 Good Mind

One case of Gemma-27B’s correct responses when $\mathcal{F}_{\Omega}=$Word-REO-ALL.
Typoglycemia Prompt:
Answer the question with only ’yes’ or ’no’ based on the passage below:
Question: is it possible to have a score of 1 in football
Passage: rare not common American method occurrences since of are – football are but the 1932 scoring in Safeties least, has every a occurred 14 safety once. Football in (NFL) the games 31 League National, under scheduling about week or rules current a once. On 21 October, 1973, Fred Dryer, Rams Bay against the Angeles Packers Los for Green the playing, score only NFL game safeties to History a in single became the in player two. A occurrence is one-point safety rarer much the, offense which by two-point the on attempt scored an be extra can conversion point or; least in NCAA have Division occurred I since 1996 at those twice football, most Bowl at 2013 the Fiesta recently. in No conversion at 1940 least since the have NFL occurred safeties. by the possible conversion defense is A also safety, though highly unlikely; never has this although occurred, a finish an point team only the single way possible football game is a with it could American in.
Response in the following format without any other information:
reason: {reason for yes or no here}
answer: {’yes’ or ’no’ here}
Response from Gemma-27B:
reason: The passage states that it is possible for a team to finish a game with only one point.
answer: yes
Original Question:
question: is it possible to have a score of 1 in football
passage: Safeties are the least common method of scoring in American football but are not rare occurrences – since 1932, a safety has occurred once every 14.31 games in the National Football League (NFL), or about once a week under current scheduling rules. On October 21, 1973, Fred Dryer, playing for the Los Angeles Rams against the Green Bay Packers, became the only player in NFL History to score two safeties in a single game. A much rarer occurrence is the one-point safety, which can be scored by the offense on an extra point or two-point conversion attempt; those have occurred at least twice in NCAA Division I football since 1996, most recently at the 2013 Fiesta Bowl. No conversion safeties have occurred since at least 1940 in the NFL. A conversion safety by the defense is also possible, though highly unlikely; although this has never occurred, it is the only possible way a team could finish with a single point in an American football game.
Answer from Dataset:
True

#### G.2.2 Bad Mind

One case of Gemma-27B’s incorrect responses when $\mathcal{F}_{\Omega}=$Word-REO-REV.
Typoglycemia Prompt:
Answer the question with only ’yes’ or ’no’ based on the passage below:
Question: is there a train station at burnham on sea
Passage: closure station’s After, demolished were shed goods and platform the. Drive Marine into developed was eastwards trackbed former The, town the into flow traffic ease helped has which road a. Street Abingdon and Approach Station Old of junction the on is itself station the of location The, park car small a now is yard goods former the and. Dorset & Somerset the renamed subsequently was Street High the on pub adjacent An. building station main former the of east and south the to land occupy yard and station RNLI new The, platform excursion former the of site the to adjacent. side seaward-west the to built was supermarket Somerfield specifically-designed ever first The, Morrisons to sold then was which.
Response in the following format without any other information:
reason: {reason for yes or no here}
answer: {’yes’ or ’no’ here}
Response from Gemma-27B:
reason: The passage states that ”itself station the of location The, park car small a now is yard goods former the and.
answer: yes
Original Quetsion:
question: is there a train station at burnham on sea
passage: After station’s closure, the platform and goods shed were demolished. The former trackbed eastwards was developed into Marine Drive, a road which has helped ease traffic flow into the town. The location of the station itself is on the junction of Old Station Approach and Abingdon Street, and the former goods yard is now a small car park. An adjacent pub on the High Street was subsequently renamed the Somerset & Dorset. The new RNLI station and yard occupy land to the south and east of the former main station building, adjacent to the site of the former excursion platform. The first ever specifically-designed Somerfield supermarket was built to the seaward-west side, which was then sold to Morrisons.
Answer from Dataset:
False

### G.3 Context Question Answering with phrases (SQuAD)

#### G.3.1 Good Mind

One case of Llama-3.1-70B’s correct responses when $\mathcal{F}_{\Omega}=$Char-DEL-BEG.
Typoglycemia Prompt:
Answer the question with word or phrase based on the context below:
Question: What was the trade name of amphetamine as a nasal decongestant?
Context: _hile _ighly _ffective, the _equirement for _njection _imited the use of _orepinephrine[_larification _eeded] and _rally _ctive _erivatives _ere _ought. A _tructurally _imilar _ompound, _phedrine, was _dentified by _apanese _hemists in the Ma _uang _lant and _arketed by Eli _illy as an _ral _reatment for _sthma. _ollowing the _ork of _enry _ale and _eorge _arger at _urroughs-_ellcome, _cademic _hemist _ordon _lles _ynthesized _mphetamine and _ested it in _sthma _atients in 1929. The _rug _roved to _ave _nly _odest _nti-_sthma _ffects, but _roduced _ensations of _xhilaration and _alpitations. _mphetamine was _eveloped by _mith, _line and _rench as a _asal _econgestant _nder the _rade _ame _enzedrine _nhaler. _mphetamine was _ventually _eveloped for the _reatment of _arcolepsy, _ost-_ncepheletic _arkinsonism, and _ood _levation in _epression and _ther _sychiatric _ndications. It _eceived _pproval as a New and _onofficial _emedy _rom the _merican _edical _ssociation for _hese _ses in 1937 and _emained in _ommon use for _epression _ntil the _evelopment of _ricyclic _ntidepressants in the1960s.
Response in the following format without any other information:
reason: {reason for the answer here}
answer: {answer here}
Response from Llama-3.1-70B:
reason: The context mentions that amphetamine was developed as a nasal decongestant under a specific trade name.
answer: Benzedrine Inhaler
Original Question:
context: While highly effective, the requirement for injection limited the use of norepinephrine[clarification needed] and orally active derivatives were sought. A structurally similar compound, ephedrine, was identified by Japanese chemists in the Ma Huang plant and marketed by Eli Lilly as an oral treatment for asthma. Following the work of Henry Dale and George Barger at Burroughs-Wellcome, academic chemist Gordon Alles synthesized amphetamine and tested it in asthma patients in 1929. The drug proved to have only modest anti-asthma effects, but produced sensations of exhilaration and palpitations. Amphetamine was developed by Smith, Kline and French as a nasal decongestant under the trade name Benzedrine Inhaler. Amphetamine was eventually developed for the treatment of narcolepsy, post-encepheletic parkinsonism, and mood elevation in depression and other psychiatric indications. It received approval as a New and Nonofficial Remedy from the American Medical Association for these uses in 1937 and remained in common use for depression until the development of tricyclic antidepressants in the 1960s.
question: What was the trade name of amphetamine as a nasal decongestant?
Answer from Dataset:
Benzedrine Inhaler

#### G.3.2 Bad Mind

One case of Llama-3.1-70B’s incorrect responses when $\mathcal{F}_{\Omega}=$Char-REO-INT_4.
Typoglycemia Prompt:
Answer the question with word or phrase based on the context below:
Question: What types of tracks from popular artists did adult contemporary radio play?
Context: Atnoher big itpeums for the evultooin of the AC rdiao fmorat was the papultoriy of easy ltinensig or ”beuafiutl msiuc” stiotnas, satitons with music saplificecly dneigesd to be purely anmebit. Wehaers most easy lniteinsg music was inttauemsnrl, caerted by rvltaieely unonwkn atstris, and rerlay pchuaesrd, AC was an aettmpt to cearte a sliimar ”lite” fmorat by csooihng ceitran tcraks (both hit sneligs and album cuts) of poalupr astrits.
Response in the following format without any other information:
reason: reason for the answer here
answer: answer here
Response from Llama-3.1-70B:
reason: The context states that AC radio played certain tracks from popular artists to create a ”lite” format.
answer: Lite tracks from popular artists
Original Quetsion:
context: Another big impetus for the evolution of the AC radio format was the popularity of easy listening or ”beautiful music” stations, stations with music specifically designed to be purely ambient. Whereas most easy listening music was instrumental, created by relatively unknown artists, and rarely purchased, AC was an attempt to create a similar ”lite” format by choosing certain tracks (both hit singles and album cuts) of popular artists.
question: What types of tracks from popular artists did adult contemporary radio play?
Answer from Dataset:
hit singles and album cuts

### G.4 Commonsense Reasoning (CSQA)

#### G.4.1 Good Mind

One case of GPT-4o-mini’s correct responses when $\mathcal{F}_{\Omega}=$Char-INS-INT_2.
Typoglycemia Prompt:
Choose one choice that best answers the commonsense question below:
Question: WnheJre dsqoes a dqooYctor wgfork silZowtly?
Choices: [emergency room, nursing home, medical school, dentist, golf course]
Response in the following format without any other information:
reason: {reason for the choice here}
answer: {one choice from the choices list here}
Response from GPT-4o-mini:
reason: A nursing home typically involves providing ongoing care for residents, which can be a slower-paced environment compared to the fast-paced nature of an emergency room or the more specialized settings like a dentist’s office.
answer: nursing home
Original Question:
question: Where does a doctor work slowly?
choices: [emergency room, nursing home, medical school, dentist, golf course]
Answer from Dataset:
nursing home

#### G.4.2 Bad Mind

One case of GPT-4o-mini’s incorrect responses when $\mathcal{F}_{\Omega}=$Char-REO-REV.
Typoglycemia Prompt:
Choose one choice that best answers the commonsense question below:
Question: nhoJ sehctaw eht llew desserd elpoep morf a klawtac evoba eht egats. eH snetsil ot meht kaeps desraeher senil elihw eht ecneidua snetsil. erehW si eh/
Choices: [theatre, new york city, fashion show, construction site, school play]
Response in the following format without any other information:
reason: {reason for the choice here}
answer: {one choice from the choices list here}
Response from GPT-4o-mini:
reason: A ”rotcod” is a doctor spelled backward, and doctors typically work in places where they provide medical care, such as an emergency room or a nursing home. However, the emergency room is specifically where doctors work in urgent situations, making it the most fitting choice.
answer: emergency room
Original Quetsion:
question: John watches the well dressed people from a catwalk above the stage. He listens to them speak rehearsed lines while the audience listens. Where is he?
choices: [theatre, new york city, fashion show, construction site, school play]
Answer from Dataset:
catwalk

### G.5 Code Generation (MBPP)

#### G.5.1 Bad Mind

One case of GPT-3.5-Turbo’s incorrect responses when $\mathcal{F}_{\Omega}=$Char-DEL-INT_2.
Typoglycemia Prompt:
Solve the code problem below in Python:
Problem: W_i_e a f_n_tion to f__d e__erian n_m_er a(n, m).
Response in the following format without any other information:
code: {Python code here}
Response from GPT-3.5-Turbo:
code:
def find_eleven_number(a, b):
for i in range(a, b+1):
if ’11’ in str(i):
print(i)
find_eleven
_number(1, 100)"
Original Quetsion:
Write a function to find eulerian number a(n, m).
choices: [theatre, new york city, fashion show, construction site, school play]
Answer from Dataset:
def eulerian_num(n, m):
if (m >= n or n == 0):
treturn 0
if (m == 0):
return 1
return ((n - m) *
eulerian_num(n - 1,
m - 1) + (m + 1)*
eulerian_num(n - 1, m))

## Appendix H TypoP Cases

### H.1 Rectify

One case of GPT-4o’s performance on Rectify TypoP task when $\mathcal{F}_{\Omega}=$Char-REO-ALL.

### H.2 Summarize

One case of Gemma-2-2B’s performance on Summarize TypoP task when $\mathcal{F}_{\Omega}=$Char-DEL-INT_1.

### H.3 Translate

One case of Gemma-2-9B’s performance on Summarize TypoP task when $\mathcal{F}_{\Omega}=$Char-REO-INT.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. In Obs.2., the authors claim that LLMs' performance is affected more with tasks that require stronger reasoning capabilities. However, would manipulating, especially on the word and sentence level change the original task? How do you ensure that after TypoC and TypoP changes, the task and answers remain the same?
2. Why would prompt tokens increase after TypoFunc? The computation time should be highly relevant to the generated tokens, rather than the prompt. Furthermore, is Figure 3 showing the ratio between completion time / prompt tokens, or such ratio before and after processing the TypoFunc? The legend and the figure comments are confusing.
3. Is cosine similarity only used to compare the difference of embedding before and after the text processing step? It is very confusing to consider this as a "metric" in explanation how you run your experiments.
soundness: 3
strengths: 1. This paper studies an interesting research question and aims to understand why LLMs are not robust to deep reasoning from a psychology perspective. This could inspire more researchers in interdisciplinary fields to explore how to evaluate and interpret language models from their expertise.
2. The paper evaluates several language models across different setups, and shows insightful observations and findings.
summary: This paper proposes to evaluate language models from the Typoglycemia perspective to verify how much alignment there is with humans in the emerging field of LLM Psychology. Specifically, the authors design experiments by formatting the original text such as changing character order (or word, and sentence order). Results across several LLMs including GPT and Gemma suggest that similar to humans, LLMs perform worse and may take longer time on tasks such as GSM8k, BoolQ, and CSQA.
weaknesses: 1. Although the studied research question can be inspiring, the proposed method and conclusion has been widely studied in the NLP field, More importantly, some of the findings seems to be a bit superficial. For instance, it is not convincing that showing the hidden layer representation (especially from an embedding API) would reveal that relying on data-driven mechanisms would limit the capacity for deep reasoning. This is certainly an important question for the whole LLM development, but without considering tokenization (which is a critical part in the paper's method but is not discussed), training data, and other aspects, the experiments and results would not be sufficient. Therefore, this paper may not benefit a lot of the researchers in the ICLR community.
2. Although the illustrates with a clear example, and is generally well organized, adding all these equations (from 1 - 9) to explain very simple and well adopted concepts (such as accuracy metrics) makes it unnecessarily complicated to follow.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1) I do not understand whether the authors are moving toward testing the (adversarial) robustness of LLMs with psychologically grounded tests. In that case, I think the idea is interesting, but I would ask them to elaborate on this.

2) How does their work connect with the large amount of literature on adversarial robustness for NLP? What is new in their approach that differs, for example, from the many works on character level perturbation of NLP classifiers?

3) How do the authors measure when a text is not intelligible anymore for humans? How do LLMs relate to that “lower bound”?
soundness: 2
strengths: The paper is well written. Math, though a bit too prolonged given the simplicity of the concepts expressed (which is not bad; the opposite), makes it clear what the authors' relevant research questions are.
The authors report many interesting results on how LLMs fail when prompted with perturbed inputs and draw an interesting analogy between humans and LLMs as cognitive systems. If I understand their idea correctly, humans are their baseline to understand whether and how LLMs differ in their reasoning process when prompted with ‘scrambled’ inputs. I applaud this idea as that may attract psychologists into an area that requires interdisciplinary expertise, with the capabilities of these models that grow every day and, for some tasks, surpass that of humans.
summary: The authors study the concept of typoglicemia for LLMs, i.e., how LLMs perform when prompted with inputs whose words’ characters (but they go beyond to sentence level) are perturbed by shuffling, deletion, insertion or addition.
The motivating idea is that humans are quite robust to these perturbations (an attack in the machine learning robustness community); such experiments are intended to show whether LLMs have similar biases to human cognition.
The authors provide a mathematical framework to explain typoglycemia, standardise the benchmark, and run experiments on several LLMs.
weaknesses: The paper lacks relevant connections to related works on the adversarial robustness of NLP models. Between 2017 and 2022, a large number of influential articles were published on methods to attack NLP classifiers and then LLMs [1, 2] (at that time, BERT and GPT-2, for example). Please notice that the second article I cite is a survey that reviews tens of works.
The authors also missed some recent articles on the robustness of LLMs against text-level perturbations [3].

The authors claim they are the first to pioneer LLM Psychology, i.e., “systematically transfer cognitive psychology methodologies and experiments to LLMs.” That is not true. There are some works in this area, e.g., [4] and related works in that paper (though I state it clearly here, that article addresses a different problem).

Apart from that, my opinion is that the authors do not elaborate enough on tokenization and a model’s internal representation. At the same time, they over-focus on the analogy between the psychology of humans and machines (but I reckon that is their intent from the beginning and the general narrative of the paper). While they measure the impact of typoglycemia on the input length (e.g., Obs. 4 (4.2)), they do not conduct an in-depth analysis at the tokenization level, i.e., where the inputs are pre-processed to be fed to a model. I understand the core idea is to measure the output of an LLM as if it were a human, but with LLMs, we need to be careful first to understand whether some phenomena are the consequence of design choices and training (e.g., positional embedding and tokenization), or they “emerge” spontaneously (in that case, your approach is interesting).

Obs 2. (4.3) Should be further elaborated. While insertion may increase a model performance as the input sequence becomes longer (e.g., an LLM has more computational time as it processes more tokens (see the paper “Think dot-by-dot” for an extreme case)), the case of deletion is interesting but no rationale is given behind this performance boost.

Obs 3. (4.4) should be further elaborated. It would have been interesting to understand the reasons behind GPT-4o's increased robustness, which seems to be common in many tasks (it can be the training data as well as a more noise-resistant tokenizer/encoding).

Minor:
Figure 1 is difficult to read (it is too small; I would either simplify it a bit or enlarge it), but it is quite illustrative!


[1] Bad Characters: Imperceptible NLP Attacks, Nicholas Boucher et al.
[2] A Survey of Adversarial Defences and Robustness in NLP, Shreya Goyal et al.
[3] Are Large Language Models Really Robust to Word-Level Perturbations?, Haoyu Wang et al.
[4] Cognitive Effects in Large Language Models, Jonathan Shaki et al.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: Same as the weakness section. If authors can address all my concerns, I'm happy to raise my score.
soundness: 2
strengths: - This paper takes an interesting angle -- From the psychology and CogSic angle and comprehensively evaluate the Typoglycemia phenomenon in LLMs.
- It tests multiple open-source and closed-source models to ensure a more thorough and comprehensive evaluation.
summary: This paper introduces a new interdisciplinary framework called “LLM Psychology”, which applies and extends human psychology methods to investigate the Typoglycemia phenomena of large language models.
weaknesses: - LLM Psychology is not a new direction; there have been many recent papers in this area. Therefore, I don't think you can claim to be the pioneer in systematically transferring cognitive psychology methodologies and experiments to LLMs in your paper. There are several related works (e.g., https://arxiv.org/abs/2409.11733, https://arxiv.org/abs/2402.07282, https://arxiv.org/abs/2406.17055, https://arxiv.org/abs/2407.06004) and many more, so the authors need to have a better literature review for this paper. If you could specifically highlight your contribution regarding Typoglycemia, I would consider this a novel and valuable contribution.
- The paper includes a lot of observations, which are interesting but need to be better compressed and emphasized. Currently, it feels more like a description of the table results rather than a focused analysis.
- The paper requires a more relevant and comprehensive related works section. The authors appear to have overlooked some recent papers, such as those addressing human-like mechanisms of LLMs. For example, the similarities between LLMs and human cognitive mechanisms are not cited, even though there is a substantial body of recent work on this topic (e.g.,https://arxiv.org/abs/2402.04559). Additionally, citations are missing in discussions about the model’s cognitive intuitions in reasoning tasks. Overall, the related works section suggests that the authors may not be sufficiently familiar with the latest research in this field.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: I did not have questions about the paper.
soundness: 2
strengths: The approach is creative and makes contact between psychology and the analysis of large language models.
summary: This paper introduces a new approach to constructing a benchmark for LLMs based on scrambling letters words and sentences in existing benchmarks. The approach is motivated by research in psychology on reading.
weaknesses: 1. The authors argue they are introducing “LLM psychology” as an approach despite the fact that there are a number of existing works taking this approach — the Binz and Schulz paper that is cited, their follow up work on CogBench, and more than 20 papers at the Annual Conference of the Cognitive Science Society to start with. This claim should be removed and related work should be cited more prominently.

2. The connection to psychology is very superficial. There are no direct comparisons between models and people on the same tasks or direct replications of psychological tasks.

3. As a result, it is not clear what we learn from the analyses presented in the paper. The main finding is that models are surprisingly robust to perturbation of characters, words, and sentences in existing benchmarks, but it is not clear what the consequences of this are for understanding LLMs or using them in practice.

4. The formalism introduced in Section 3.1 largely seems unnecessary. As an illustration of this, it is not used anywhere else in the paper.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: Question 1: Could the authors provide more discussions on the logical connection between the observation to human cognitive patterns, as well as more psychological research citations for support in Lines 527-52 below?
> Based on these observations, we posit that the heatmap can translate each model’s unique ”cognitive pattern” through our Typoglycemia experiments, much like how different human individuals exhibit distinct cognitive patterns.

Question 2: Instead of “scrambling ratio”, why not just use the text edit distance as an indicator and visualize this pattern? Currently, the number of changes is between 1-4, what if we scale this up to large numbers of edits?

Question 3: As the authors also mentioned (line 92), LLMs’ tokenization algorithms have a lot to do with misspelling understanding and robustness under typos. In attributing these behaviors directly to the LLMs, I wondered to what extent the observations might actually stem from the tokenization process. Could the authors provide examples comparing the tokenization of correctly written versus scrambled text?
soundness: 1
strengths: 1. This work is one of the first to introduce Typoglycemia experiments (both correction and task completion) to realistic LLM benchmarks. 
2. This work highlights a lot of interesting behavioral observations in LLMs dealing with misspelled input. I particularly enjoy observation 4, RQ 1, Section 4.2 (computation cost). I think a lot of the findings can give a lot of hints for future inference time reasoning research. 
3. Studying LLMs using psychological methods is a timely direction to explore.
summary: This work introduces a set of Typoglycemia experiments as an example for comparing LLM behaviors to human cognitive patterns. The authors studied the scrambled text correction task (aka the known misspelling correction task) and evaluated the model on existing benchmarks while typos are there. A number of findings on how different text scrambling affects LLM performance and computational cost have been provided.
weaknesses: ### Weakness 1: Overclaiming. 

The most concerning problem of this work is overclaiming throughout this paper.

The authors claim that they contributed a “New Direction”. 
> We propose “LLM Psychology” as an interdisciplinary framework with significant research depth, offering novel methodologies, directions and insights for the future study of LLM’s human-like cognition. To the best of our knowledge, we are the pioneer to systematically transfer cognitive psychology methodologies and experiments to LLMs, assessing the similarities and differences between LLMs and humans from a cognitive psychological perspective (Lines 112-116).

This is clearly overclaiming to me, with so much research being done in this field [1-3] ignored. I can understand that the authors try to position this work as an “example of how psychological principles can be applied to understand and evaluate LLMs”, but still, the authors also claimed that “What all of these studies lack is the systematical exploration of the generalized and intrinsic cognitive mechanisms of LLMs” (Lines 83-84). While I agree with this statement, neither did this work provide a satisfying answer for (1) a systematical/generalized study, nor “intrinsic cognitive mechanisms”, which is ill-defined and requires significant studies on mechanistic interpretability and neuroscience.

**I strongly encourage the authors to really scope down and identify their own unique angle of contributions to the community, rather than trying to convey a sensational headline to hype the community.**

---
### Weakness 2: Weak Logical Connections
There is a critical lack of logical connection between behavioral results to mechanistic cognitive principles. 
Now, ignoring all the psychological stories, this paper essentially studied the scrambled text correction task (aka the known misspelling correction task) and evaluated the model on existing benchmarks while typos are there. 

Some behavioral observations are interesting, though most of them are not surprising. I particularly enjoy observation 4, RQ 1, Section 4.2 (computation cost). This can give a lot of hints for future inference time reasoning research.

Some parts of the experiments can be better executed, for example, instead of “scrambling ratio”, why not just use the well-known text edit distance as an indicator?

Some parts of the experiments lack logical soundness. For example, 
> Obs.3. The hidden layer representations of the same LLM across different datasets exhibit similar “cognitive patterns.” As illustrated in Figure 5, the color distributions for SQuAD and BoolQ under various FΩ appear visually similar. (Lines 522-524)

Let’s forget about the fact that the color bars in Figure 5 have different value scales for a moment. The logical connection between the “... appear visually similar” and “similar cognitive patterns” has not been discussed.

---
### Weakness 3: Missing Related Work. 
The authors should discuss more on the critically related lines of work like misspelling detection, typographic attacks, typo-correction, and typo-robustness in LLMs [4-5, inter alia].

---
### Weakness 4: Communication. 
Figures 1/2 certainly draw the eye with a charming simplicity reminiscent of classic, child-friendly visuals. While delightfully approachable, they might benefit from enhanced clarity and a touch more alignment with typical scientific presentation standards, as I am completely lost and need help understanding the whole pipeline and method from it. I encourage the authors to consider alternative ways of making the main figures.


[1] Hagendorff, Thilo, Thilo Hagendorff, Ishita Dasgupta, Marcel Binz, Stephanie C.Y. Chan, Andrew Lampinen, Jane X. Wang, Zeynep Akata, Eric Schulz. "Machine psychology: Investigating emergent capabilities and behavior in large language models using psychological methods." arXiv preprint arXiv:2303.13988 (2023).

[2] Demszky, Dorottya, Diyi Yang, David S. Yeager, Christopher J. Bryan, Margarett Clapper, Susannah Chandhok, Johannes C. Eichstaedt et al. "Using large language models in psychology." Nature Reviews Psychology 2, no. 11 (2023): 688-701.

[3] Li, Yuan, Yue Huang, Hongyi Wang, Xiangliang Zhang, James Zou, and Lichao Sun. "Quantifying ai psychology: A psychometrics benchmark for large language models." arXiv preprint arXiv:2406.17675 (2024).

[4] Evertz, Jonathan, Merlin Chlosta, Lea Schönherr, and Thorsten Eisenhofer. "Whispers in the Machine: Confidentiality in LLM-integrated Systems." arXiv preprint arXiv:2402.06922 (2024).

[5] Song, Gan, Zelin Wu, Golan Pundak, Angad Chandorkar, Kandarp Joshi, Xavier Velez, Diamantino Caseiro et al. "Contextual Spelling Correction with Large Language Models." In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1-8. IEEE, 2023.

Meta-review: The contributions of this paper were not judged as novel and well grounded in literature by initial reviews -- authors did not engage in rebuttal or discussion.

Decision: Reject