Unnatural Error Correction: GPT-4 Can Almost Perfectly Handle Unnatural Scrambled Text
Authors: Qi Cao
Takeshi Kojima
Yutaka Matsuo
Yusuke Iwasawa
Venue: EMNLP 2023 Main (2023)
Keywords: Large Language Models
Emergent Ability
Scrambled Text
GPT-4

Abstract: While Large Language Models (LLMs) have achieved remarkable performance in many tasks, much about their inner workings remains unclear. In this study, we present novel experimental insights into the resilience of LLMs, particularly GPT-4, when subjected to extensive character-level permutations. To investigate this, we first propose the Scrambled Bench, a suite designed to measure the capacity of LLMs to handle scrambled input, in terms of both recovering scrambled sentences and answering questions given scrambled context. The experimental results indicate that multiple advanced LLMs demonstrate the capability akin to typoglycemia, a phenomenon where humans can understand the meaning of words even when the letters within those words are scrambled, as long as the first and last letters remain in place. More surprisingly, we found that only GPT-4 nearly flawlessly processes inputs with unnatural errors, a task that poses significant challenges for other LLMs and often even for humans. Specifically, GPT-4 can almost perfectly reconstruct the original sentences from scrambled ones, decreasing the edit distance by 95%, even when all letters within each word are entirely scrambled. It is counter-intuitive that LLMs can exhibit such resilience despite severe disruption to input tokenization caused by scrambled text.

Unnatural Error Correction: GPT-4 Can Almost Perfectly Handle Unnatural Scrambled Text
Qi Cao, Takeshi Kojima, Yutaka Matsuo and Yusuke Iwasawa
The University of Tokyo, Japan
{qi. cao, t. koj.kojim,aattttsoo jaaaslalaewebaa t. u-tokyo. ac. jp
Abstract
following sentence contains words scrambled letters. Please recover original from Serambled sentence: wno
While Large Language Models (LLMS)) have achieved remarkable performance in many tasks, much about their inner workings remains unclear. In this study, we present novel experi- mental insights into the resilience LLMs, par- ticularly GPT-4, when subjected extensive character- permutations. To investigate this, we first propose the Scrambled Bench, suite designed to measure the capacity of LLMs to handle scrambled input, in terms of both re- covering scrambled sentences and answering questions given scrambled context. The experi- mental results indicate that multiple advanced LLMs demonstrate the capability akin typo- glycemia! ', phenomenon where humans can understand the meaning of words even when the letters within those words scrambled, as long as the first and last letters remain in found that only
Mentsrs ermtno no duySan atgsuAu ntaaNloi Gflo bClu, gneleinhi ish nereg ecatkj nad ucedos rascer jroam.
Recovered sentence:
Rahm the 2023 Tournament Sunday Augusta National Golf clinccinn his first green juckel
second career major;
Figure GPT-4 can recover the original sentence from the scrambled sentence, even if the tokenization drastically changes. (The colors indicate the division of sub-words during the tokenization.)
become more "intelligent". many new benchmarks have been proposed (Liang et al., 2022; Qin et al., 2023) to investigate the ability of LLMs. Nonethe- less, the internal mechanisms underlying the capa- bilities of LLMS remain enigmatic.
place. More surprisingly, GPT-4 nearly flawlessly processes inputs with unnatural errors, a task that poses significant challenges for other LLMs and often even for humans. Specifically, GPT-4 can almost per- fectly reconstruct the original sentences from scrambled ones, decreasing the edit distance by 95%, even when all letters within each word are entirely scrambled. It is counter-intuitive that LLMs exhibit such resilience despite severe disruption to input tokenization caused by scrambled text."
Several studies investigate the behavior of LLMs given some input perturbations. For example, Sinha et al. (202la,b); Abdou et al. (2022) investigate the influence of word-level permutations and show that models are insensitive to permutations of word order that corrupt the original syntax, in some downstream tasks (e.g.. natural language inference). These results particularly interesting because they challenge the common assumption of the inner workings of LLMs, i.e., LLMs understand human- like syntax to some extent and use to understand sentences.
1 Introduction
Large language models (LLMs) demonstrate im- pressive proficiency across range of tasks, with certain capabilities emerging the models scale up in size-a phenomenon commonly known as emer- gent abilities. (Wei al., 2022a). As the LLMs
Going beyond the effect of word-level permu- tations, we investigate the ability of LLMs under character-level permutations. LLMs are supposed to rely on the tokenizers to turn natural language into the form that LLMs can perceive. It would be counter-intuitive if LLMS could effectively handle text containing unnatural permutations that sig- nificantly alter tokenization. In other words, we propose the following research question: Deas the
'Typoglycemia internet meme without specific sci- entifie backing. Nonetheless, people do exhibit the ability to recognize scrambled common words some extent. Our focus not on examining this ability of humans, we use the concept for illustrative purposes.
"Code will be available at https://github..cm com/ ccag7//unntural-errg perror-correction

---

oredr of ltteers in wrods milaer for LLMs? Note that the above sentence contains scrambled words, but humans can somehow recognize and under- stand such a sentence, as several cognitive stud- ies have explored (Rawlinson, 2007; Mason, 1982; Johnson and Eisler, 2012). Analyzing the robust- ness of LLMs against such character-level permu- tations can shed light on their word comprehension capabilities and reveal differences between various LLMs and human understanding.
bled text along with the task accomplishment given scrambled context. It differs with typographical er- ror correction (Shah and de Melo, 2020; Sun et al., 2022), as (i) we do not train models to correct er- rors, i.e., we measure the ability of LLMs, and (ii) we add much more severe noises than natural ty- pographical errors. The word unscrambling task in BigBench (Srivastava et al., 2023) is similar to our recovery task. However, it is more akin to wordplay puzzle rather than task for comprehend- ing scrambled text, since it includes only single common words and no context is given.
To this end, this paper first constructs Scrambled Bench, which converts existing benchmarks into a test suite to measure the ability of LLMs to handle scrambled text. We designed two types of tasks: (1) Scrambled Sentence Recovery, which tests the capability of LLMs to reconstruct the original sen- tences from scrambled ones, and (2) Scrambled Question Answering, which measures how well LLMs can answer questions when some context is scrambled. Note that since the slight change in letter-order within word drastically changes the tokenized output (see Figure 1), it is questionable whether LLMs recognize the scrambled words a sentence. Counter-intuitively, we show that the most powerful LLMs are able handle scrambled sentences to varying degrees, when we scramble words while keeping the first and last letters un- changed. More surprisingly, we found that only GPT-4 can almost flawlessly process inputs with unnatural errors, even under extreme conditions. That even when we scramble all letters in words, GPT-4 manages to handle such input-- signifi- cantly challenging task for other models and even humans. For instance, GPT-4 can reconstruct the original sentences to near-pert near-perfect recovery rate in the extreme scenario, as in Figure 1.
3 Scrambled Bench
We propose two tasks to evaluate the capability of LLMs to handle scrambled text.
(1) Scrambled Sentence Recovery (ScrRec). In this task, we provide sentence containing scram- bled words to LLMs and then ask them to recover the original sentence from it. This task can be uti- lized to directly measure the capability of LLMs recognize and reconstruct the scrambled words in sentence.
(2) Scrambled Question Answering (ScrQA). While ScrRec can directly measure the capabil- ity to comprehend and process scrambled text, it is an "unusual" task for LLMs. If model does not perform well on the recovery task, there are two possible reasons: (i) having difficulty following the instructions and (ii) not being able to recover sentences. To distinguish them, we measure the ability to accomplish a standard task (i.e., QA) given scrambled context. Specifically, we scramble the content that contains essential information for answering questions and then assess the models based on the variations in their performances.
2 Related Works
3.1 Dataset Creation
The most related works are the studies investigat- ing the effects of word or sub-word level pertur- bations (Sinha et al., 2021a,b; Pham et al., 2021; Abdou et al., 2022) and the studies evaluating the robustness of LLMS (Wang et al., 2023; Zhu et al., 2023). To the best of our knowledge, no existing studies have investigated LLMS'' ability to handle character-level permutations, particularly those of extremely high level that drastically change tok- enization. Our study aims fill this gap. Table in Appendix categorizes the prior studies and demonstrates the position of our study. Besides, we directly evaluate the ability to recover the scram-
In this study, we primarily utilize a scrambled ver- sion of RealtimeQA (Kasai et al., 2022) for eval- uation. A common issue in evaluating LLMs is data contamination, which occurs when the test data of downstream tasks is present in the train- ing data. It particularly affects our experiments because the analysis would be useless if some models memorized the original contents. Real- timeQA dynamic question answering dataset that weekly announces questions about recent news that are unlikely to be memorized by the current LLMs. Specifically, we collect the most recent

---

ScrRec (zero-shot)
ScrRec (few-shot)
ScrQA
100
Performance
80
60
0
GPT-4
GPt-3.0-turbo
text-davinG-0033
Falcon-180b
Llama-2-700
Figure Experimental results on scrambled RealtimeQA dataset. Performance on SerRec with zero-shot setting (left). Metric: RR(%). Performance SerRec with a few-shot setting (middle). Metrie: RR(%), Performance on ScrQA with zero-shot setting (right). Metric: RPG(%). The missing bars represent values less than zero.
data (2023/03/1722023/08// from RealtimeQA (totally 419 samples) and process the evidence sen- tences to construct samples for SerRee and ScrQA. Finally, 418 samples are selected for SerRee (re- moving a duplicate sentence), and 346 samples are selected for ScrQA (manually eliminating 73 sam- ples when the provided evidence does not provide sufficient information answer the corresponding question).
different scramble types in word recognition for LLMs.
3.2 Metries
The average Edit Distance (ED) (Levenshtein, 1966) between the original sentences and the re- covered sentences a natural metric to quantify the performance on ScrRec. Besides, we define Recovery Rate (RR) to measure the proportion ED reduced in recovered sentences, which makes the performance comparison different settings more straightforward as:
Besides, we also introduce two additional datasets: DREAM (Sun et al., 2019) and AQuA- RAT (Ling et al., 2017). DREAM is dialogue- based multiple-choice reading comprehension dataset. AQuA-RAT dataset of math word problems necessitating multi-step reasoning for their resolution. For DREAM dataset, we con- structed the dataset by selecting 1025 samples with annotated categories from the development and test sets and then scrambling the dialogue part of each question. For AQuA-RAT dataset, we adopt the few-shot Chain of Thought (CoT) setting as in Wei al. 2022b and evaluate LLMs with serambled questions in samples and demonstrations.
RR = Σn (D rec,) Σ. (Droi scr;)
Σ. ED(ori,, scr;)
where ori, SeT,, rec, mean the original sentence, the scrambled sentence, and the recovered sentence of sample i, respectively. If a model is capable of recovering scrambled text to some extent, its RR should range between 0 and 100%.
For ScrQA, accuracy is natural metric to mea- sure performance. But varying capabilities of mod- els on original questions make hard to compare the performance among models. So, Relative Per- formance Gain (RPG) is defined to mitigate the differences and make evaluations focus on how well models can extract information from scram- bled text in comparison to original text
For each dataset, we generate scrambled text with various scramble types and rates. (1) Ran- domly Scramble (RS). For each sentence, we randomly select certain percentage (20%, 50%, 100% in case²) of words and randomly shuf- fle the positions of letters in each selected word (Arabic numerals are kept invariant).
ACCser ACCsub
(2) Keep First (KF). We keep the first letter in each word unchanged and randomly shuffle the letters in other positions. (3) Keep First and Last (KFL). We keep the first and last letter in each word unchanged and randomly shuffle the letters in other positions. We use KF and KFL to investigate the effects of
RPG
(2)
Accori ACCsub
where Accori. Accaer, Acc,uh are defined as ac- curacy when using original evidences, scrambled evidences, substituted evidences (in which all let- ters of words that be scrambled have been ran- domly substituted), respectively. Accoris Accsub are served as the higher bound and the lower bound of accuracy, respectively. Normally, RPG ranges between and 100%.
IIn the latter part of the paper, RS denotes 100% randomly scrambling, unless a different pereentage is specified.

---

4 Experiments
100
90
Models and Settings. In the experiments, we evaluate the most powerful closed-source LLMs. including text-davinci-003 (Brown et al., 2020),
GPT-3.5-turbo and GPT-4 (OpenAI, 2023) and the open-source models from Falcon series (Penedo al., 2023), Llama-2 series (Touvron al., 2023), MPT series (Team, 2023), UL2 series (Tay al., 2022), T5 series (Raffel et al., 2020; Chung et al., 2022; Xue et al.. 2022). In scrambled RealtimeQA dataset, we adopt zero-shot setting and a few- shot setting with 3-shot exemplars from the wik- iQA dataset (Yang et al., 2015) for SerRee, while we only conduct experiments on zero-shot ting (since the task rather straightforward) for ScrQA. In scrambled DREAM dataset, the set- ting is also zero-shot ScrQA. In scrambled AQuA dataset, we adopt a few-shot CoT setting with scrambled demonstrations (in the question part). Note that we are showcasing the results of the top five most proficient LLMs (i.e., GPT-4, GPT- 3.5-turbo, text-davinci-003, Falcon-180b, Llama-2- 70b) in this section, but comprehensive results can be found in Appendix C.
RR (%)
50
40
Model
30
GPT-4
20
daxtidavinci-Ooe 003
Setting
Falcon-
zero shot
Uama-2-70b
few-shot
20%
100%
Scramble Rate
Figure 3: The effect of scramble rates on the perfor- mance ScrRee-RealtimeO. The missing points represent values less than zero.
100
80
RPG (%)
60
Model
50
GPT-4
GPT-3 5-turbb
text-d avin-iovin
Falcon-180b
30
Jaam-22-70b
Results 1: Effect of different scramble types. Figure 2 show the results on zero-shot ScrRec, few- shot SerRec, and ScrQA, with three scramble types: randomly scramble (RS), keep first (KF), and keep first and last (KFL). The results show the perfor- mance gaps among models are not large in KFL setup. However, except for GPT-4, performance significantly decreases as the difficulty of scramble types increases (KFL, KF, and RS in order). In con- trast, the performance of GPT-4 remains constantly high regardless the scramble types. For ScrRec, RR GPT-4 is constantly above 95% for all setups. For ScrQA, GPT-4 also constantly performs best with very limited accuracy drop, as the difficulty of scramble types increases.
20%
50%
100%
Figure 4: The effect of scramble rates the perfor- mance on
Similarly, Figure 4 plots RPG against the scram- ble rates for different models on ScrQA with seram- bled Realtime RPG of all models drops scramble percentage increases. But GPT-4 main- tains 87.8% of original performance even with 100% scrambled evidence. The gap between the performance of different models becomes larger when the scramble rates increases. Table 3, Table 4 and Table 5 in Appendix C show full results on scrambled RealtimeQA dataset, with additional evaluation metrics (ED for ScrRec and Acc for ScrQA).
Results 2: Effect of different scramble rates. Figure 3 illustrates the relationship between the scramble rates (i.e., the percentages of randomly scrambled words in text) and the performance on SerRec with scrambled RealtimeQA. As the scram- ble rates increases, RR decreases for text-davinci- 003, Falcon-1 80b and Llama-2-70b. RR of GPT- 3.5-turbo and GPT-4 does not change significantly. GPT-4 outperforms other models by wide margin, with higher than 95% RR for most setups (except
Results 3: Results of additional datasets. Fi-
nally, we test the generality of the finding across datasets by two additional datasets for ScrQA. For scrambled DREAM dataset, evaluate perfor- mance not only overall but also on different cat- egories of questions, using the annotations. The performance disparities between GPT-4 and other models are more pronounced than those observed on RealtimeQA, possibly since DREAM requires
for 20% scramble rate).

---

All
Arithmetic
Commonsense
100
RPG (%)
60
40
Logic
Matching
Summary
100
80
RPG (%)
20
0
GPT-4
GPT-3.5-ture
text-davinci-003
Falcon-180b
Llama-2-70b
Figure 5: Experimental results of ScrQA on scrambled DREAM dataset. Overall performance and performance on different categories of questions. The missing bars represent values less than or equal zero.
Exemplars
Original
%
50 %
(00
Question
Original
46.85 52.76 66.93
47.24131.97
70.87
45.67 T 54.77447....0
46.46 14.72 70.08
20 %
40.94 49.61 66.54
40.16 51.57 74.02
42.52
52.36 68.50
45.28
50.79 71.26
50 %
40.94 40.94 67.72
41.34
44.88
65.75
40.16
147. 64 64.96
44.09
42.91 67.32
100) %
26.77 29.92 66417
27.17 31.89 62.60
33.47 27.17 1 64.96
28.35 35.04 60.24
Table 1: Experimental results ScrQA on scrambled AQuA-RAT dataset with a few-shot CoT setting. LLMs
assessed in various scenarios by adjusting scramble rate of questions in examplars and the main question. (left: text-davinci-003; middle: GPT-3.5-turbo; right: GPT-4). Metrie: Acc (%).
higher-level comprehension of longer texts. Per- formance on arithmetic questions tends be more susceptible to scrambled text compared to other categories, even for GPT-4. Table demonstrates experimental results with 4-shot CoT setting on scrambled AQuA-RAT dataset (we only test the performance of three closed-source models here because the original questions in AQuA-RAT are too challenging for most open-source models). We evaluate models with several different scenar- ios by changing scramble rates both questions in exemplars and main question. The results show the influence of scrambled exemplars is relatively small. And the performance of GPT-3.5-turkee and text-davinci-003 drop significantly when the scramble rate of main questions increases 100%, of its original perfor-
Conclusion
In this study, we propose Scrambled Bench, test suite to measure the ability LLMs to handle scrambled text, including two tasks (i.e., scram- bled sentence recovery and serambled question an- swering) and construct scrambled datasets based on RealtimeQA, DREAM and AQuA-RAT. De- spite the serambled text drastically changes the tokenization, we demonstrate that advanced LLMs are capable of processing scrambled text to varying degrees. However, most LLMs have difficulty han- dling text that scrambled to an extreme degree (i.e., 100% randomly scrambling). Surprisingly, for both tasks, GPT-4 shows good results and out- performs other models by a large margin. For the scrambled sentence recovery task, GPT-4 can cover sentences by 95% edit distance reduction even in 100% randomly scrambling settings. For the scrambled question answering task, GPT-4 can maintain a very high proportion of its original ac- curacy using scrambled context.
while GPT-4 remains mance. Table 6, Table 7 and Table 8 in Appendix C show full results on scrambled DREAM dataset.

---

Limitations
Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling instruction finttion-- language mod- els.
For L.Mss, there are various ways to disrupt the tokenization of words (e.g., inserting letters, substi- tuting letters). In this study, we only investigate the influence of scrambling the letter-order in words. Investigating the performance of LLMs to handle other situations would be an interesting topic.
In addition, we have conducted our experiments using only three datasets, RealtimeQA, DREAM and AQuA-RAT. Experiments datasets could be another future work. Note that the two tasks can be applicable for diverse datasets, and it easy to extend the analysis.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Effficient finetuning of quantized Ilms. arXiv preprint arXiv:2305. 14314.
more diverse
Rebecca L Johnson and Morgan E Eisler. 2012. The importance of the first and last letter in words during sentence reading. Acta psychologica, 141 141(3)3336 351.
We investigate the capability of different LLMs to handle scrambled text in different settings. How- ever, it is difficult to conclude the reason why (some) LLMs capable to these tasks. Espe- cially, the reason why GPT-4 can perform almost perfectly would be an interesting topic worth fur- ther investigation. We can not access the closed- source models directly and are aware of little in- formation about them (even the exact model size of GPT-4). These situation make investigating the reason difficult. An hypothesis is that this capabil- ity might be related to training methods, such as incorporating tasks similar to denoising in the train- ing objectives, or using a vast amount of text data containing various errors in the training process. Another hypothesis is that this capability emerges as LLMs scale. Validating these hypotheses could potentially enhance our understanding of the inner workings of LLMs, thereby enabling us to reverse- engineer and recreate more sophisticated models like GPT-4.
Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, and Kentaro Inui. 2022. Realtime qa: What's s the answer right now? arXiv preprint arXiv:2207. 13332..
Vladimir I Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. Soviet physics doklady, 10(8):707-710.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayaman, Yuhuai Wu, Ananya Ku- mar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.00 (iV02211,,01111
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blun- som. 2017. Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems. ACL.
Mildred Mason. 1982. Recognition time for letters and nonletters: effects of serial position, array size, and processing order. Journal of Experimental Psychol- ogy: Humun Perception and Performance, 8(5):724.
OpenAL. 2023. Gpt-4 technical report.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint 200.013366.01166.
References
Mostafa Abdou, Vinit Ravishankar, Artur Kulmizev, and Anders Søgaard. 2022. Word order does matter and shuffled language models know it. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 6907-6911 Dublin, Ireland. Association for Computational Linguistics.
Thang Pham, Trung Bui, Long Mai, and Anh Nguyen. 2021. Out of order: How important is the sequen- tial order of words in a sentence in natural language understanding tasks? In Findings of the Association for Computational Linguistics: ACL-IICNLP 2021. pages 1145-1166 Online. Association for Computa- tional Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:7777-1
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt general-purpose natural language process- ing task solver? arXiv preprint 2tl.:2302.06777
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang,

---

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-5511
MosaicML NLP Team. 2023. Introducing mpt-30b: Raising the bar for open-source foundation models. Accessed: 2023-06-22.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0922 009888.
Graham Rawlinson. 2007. The significance of letter position in word recognition. IEEE Aerospace and Electronic Systems Magazine, 22(1):26-21 27.
Jindong Wang. HU Xixu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang. Wei Ye, Haojun Huang, Xiubo Geng, et al. 2023. On the robustness of chatgpt: An adversarial and out-of- distribution perspective. In ICLR 2023 Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models.
Kshitij Shah and Gerard de Melo. 2020. Correcting the autocorrect: Context-aware typographical error correction via training data augmentation. In Pro- ceedings of the Twelfth Language Resources and Evaluation Conference, pages 6930-6936, Marseille, France. European Language Resources Association.
Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 2021a. Masked language modeling and the distributional hy- pothesis: Order word matters pre-training for little. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 2888 2888-2913 2993, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph. Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022a. Emer- gent abilities of large language models. Transactions on Machine Learning Research. Survey Certifica- tion.
Koustuv Sinha, Prasanma Parthasarathi, Joelle Pineau, and Adina Williams. 2021b. UnNatural Language Inference. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 7329-7346, Online. Association for Computa- tional Linguistics.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. 2022b. Chain-of-thought prompting elicits rea- soning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837.
Linting Xue, Aditya Barua, Noah Constant, Rami Al- Rfou, Sharan Narang, Mihir Kale, Adam Roberts. and Colin Raffel. 2022. ByT5: Towards token-free future with pre-trained byte-to-byte models. Transac- tions of the Association for Computational Linguis- tics, 10:291-300
Aarohi Srivastava, Abbinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2023, Beyond the imitation game: Quantifying and extrapolating the capabili- ties of language models. Transactions on Machine Learning Research.
Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. Wikiqa: A challenge dataset for open-domain ques- tion answering. In Proceedings of the 2015 con- ference on empiricul methods in natural language processing, pages 2013-2018.
Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019. DREAM: challenge data set and models for hallguu-bbssed reading compre- hension. Transactions of the Association for Compu- tational Linguistics. 7:217-231.
Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Neil Zhenqiang Gong, Yue Zhang, al. 2023. Promptbench: Towards evaluating the robustness of large language models on adversarial prompts. arXiv preprint arXiv:2306.0455
Rui Sun, Xiuyu Wu, and Yunfang Wu. 2022. An error- guided correction model for Chinese spelling error correction. In Findings of the Association for Com- putational Linguistics: EMNLP 2022, pages 3800- 3810, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Gar- cia. Jason Wei, Xuezhi Wang, Hyung Won Chung. Dara Bahri, Tal Schuster, Steven Zheng, et al. 2022. U12: Unifying language learning paradigms. In The Eleventh International Conference on Learning Rep- rexentations.

---

A Examples of different tasks (by GPT-4)
bsipioes ionmanantitco, grnoadice eth ADE heT pyomacn noniacrggue rplsean ckhee yna poduesrt yhte evah mhoe nda cdisadr sehot taht aym eb ecaeftdf.
A.1 Zero-shot ScrRec on RealtimeQA
Input:
The following sentence contains words with scram- bled letters. Please recover the original sentence from it.
Recovered sentence:
Oulput: A Gerber baby formula was distributed to stores despite recall over possible contamination, cording to the FDA. The company is encouraging parents check any products they have at home and discard those that may be affected.
Scrambled sentence: A reGebr byba ulfaorm wsa titbudiserd ot soetsr epdstie a Iclera eovr bslpi- oes ionmananticco, grnoadice ot eth ADF he'I py- omacn si noniacrggue rptsean of ckhec yna poduc- srt yhte evah mhoe nda edisadr sehot taht aym eb ecaeftdf.
Recovered sentence:
A.3 Zero-shot ScrQA on RealtimeQA
Output:
Input:
A Gerber baby formula was distributed to stores despite recall over possible contamination, ac- cording to the FDA. The company is encouraging parents to check any products they have at home and discard those that may be affected.
Question: Which type of product was recently dis- tributed to some US stores despite recall notice?
Choices: (A)Salad dressing (B (B)Baby formula (C)Ground beef (D)Whole milk
Evidence: A reGebr byba ulfaorm titbudiserd ot soetsr epdstie a Iclera eovr bslpioes ionmanan- titco, grnoadice eth ADE. heT pyomacn noni- acrggue rptsean ot ckhec yna poduesrt yhte evah ta mhoe nda cdisadr sehot taht aym eb ecaeftdf.
A.2 Few-shot ScrRec on RealtimeQA
Input:
Scrambled Sentence: eTh camp continued to fictinuo this ayw ilntu the rwa needd.
Answer: Based on the evidence, among A through D, the answer is Output:
Recovered Sentence: The camp continued to function this way until the war ended.
BB)Baby formula
Scrambled Sentence: It swa first developed ni the 1980s yb oAern Computers tdL ot crowp their pstodke nmecisah and subsequently supn off separate paocnmy, now ARM Holdings.
A.4 Zero-shot ScrQA DREAM
Input:
Dialogue:
Recovered Sentence: was first developed in the 1980s by Acorn Computers Ltd to power their desktop machines and subsequently spun off as a separate company, now ARM Holdings.
M: owH goin veah uyo ehen ehctnaig ni iths edmdli
colosh?
W: Fro ent esray, oT eb akfmn, I'm edirt fo higneact het mesa bokoextt rof os ongl hghuto od jyneo gebin aterech. I'm idenoncisrg gntriy minseghot
Scrambled Sentence: According to the CIA kcb- Fotoa, the United States is fo eethr iusecnort (het etrhos nebgi Liberia nda mBuar//yanmarr) that sha not adopted eth International System fo Utins (SI) rmtcei symset as iethr ffliicao system fo gswheit dna measures.
Question: What's the woman probably going do?
Choices: (A)To teach a different textbook. (B)To change her job. (C)To leam a different textbook.
Answer: Based on the dialogue, among A through
Recovered Sentence: According to the CIA Fact- book, the United States is one of three countries (the others being Liberia and Burma/Myanmar) that has not adopted the International System of Units (SI) metric system their official system of weights and measures.
C, the answer is
Output:
(B) change her job.
A.5 Few-shot CoT ScrQA on AQuA-RAT
(1)
Input:
Scrambled sentence: A reGebr byba ulfaorm wsa titbudiserd ot soetsr epdstie a Iclera eovr
Question: hJno nfodu ttah teh gaeerav fo 15 msbmnue si 40. fT 10 si edadd ot ceha umenbr enht

---

Example (Original: How does the change affect tokenization?)
Natural errors
Change of tokenization
Level
Severity
Studies
Sinha al. 2021a.
How the affect does tokenization change?
Sinha al. 202
Word
No
Order
Low-High
Pham et al. 2021,
Ahdou et al. 2022
How change token does affectization the?
Sub-word
No
Order
Low-High
Ahdou et al. 2022
How does the chagne affect tokeniztion?
Wang al. 2023, Zhu 2023
Character
Yes
Tokens
Low
wHo deso teh chenga afetfe aotznoenitki?
Character
No
Tokens
Middle-EEttremlly High
Ours
Table 2: Categorizing the prior works and demonstrating the position of this work.
teh naem fo het rbenmsu si?
75 licsm/uhor. Wthh si het otalt orjeyun iroatdnu?
Choices: (A)50 (B)45 (C)65 (D)78 (E)64 Answer: If 10 added to each number, then the mean of the numbers also increases by 10. So the new mean would be 50. The answer is (A).
Choices: (A)10 (B) (C)12.5 (D)13.5 (E)15 Answer:
Output:
The train covers a quarter the distance at 100 miles/hour, so it travels for 2.5 hours (10 hours 4) at this speed. The remaining distance is 3/4 of the total distance, which takes the train 7.5 hours (10 hours 2.5 hours) at 100 miles/hour. When the train slows down to 75 miles/hour, it takes 1.5 times longer to cover the same distance (100 miles/hour 75 miles/hour 1.33). So the train takes 11.25 hours (7.5 hours 1.33) to cover the remaining distance at 75 miles/hour. The total journey duration is 13.75 hours (2.5 hours + 1 1.25 hours). The answer is (D).
Question: fl a b = 3/4 dan 8a + 5b = 22. htne nfid het evual fo a.
Choices: (A)1/2 (B)3/2 (C)5/2 (D)4/2 (E)7/2
Answer: If a b 3/4, then b 4a 33. So 8a +
5(4a / 3) = 22. This simplifies 8a 20a/ 22,
which means 44a 3 22. So equal 3/2. The answer (B).
Question: A nprseo si gtlanrvie ta 20 mk/rh nda hdrecae ihs ynedits ni 2.5 rh neth fdin eth dicatsen?
Choices: (A)53 km (B)55 km (C)52 km (D)60 km (E)50 km
(2)
Input:
Answer: The distance that the person traveled would have been 20 km/hr * 2.5 hrs 50 km. The answer is (E).
Question: John found that the average of 15 numbers is 40. If 10 added to each number then the mean of the numbers is?
Choices: (A) (B)45 (C)65 (D)78 (E)64
Question: wHo aymn etosreyksk ear deeend ot peyt het nsrbmeu rfmo I ot 500?
Answer: If 10 is added to each number, then the mean of the numbers also increases by 10. So the new mean would be 50. The answer is (A).
Choices: (A)1156 (B)1392 (C)1480 (D)1562
1E)
Answer: There are 9 one-digit numbers from to
Question: If / b 3/4 and 8a 5b 22.
9. There are 90 two-digit numbers from 10 99.
then find the value of
There are 401 three-digit numbers from 100 to 500. 9 90(2) + 431 1392. The answer is (B).
Choices: (A)1/2 (B)3/2 (C)5/2 (D))/2 (E)7/2
Answer: / = 3/4, then b = 4a / 3. So 8a +
5(4a / 3) = 22. This simplifies to 8a + 20a / = 22,
Question: A ritna inunrgn ta a eepsd fo 100 mlesi/uhro, eakts 10 hurso ot cerah ist dneitiotsan. trefA ovgeime etarqru fo het entsdaic, ti tstrsa grniina nda eth rniat ash ot eb dlseow ot seedp fo
which means 44a / 22. So a is equal to 3/2. The answer is (B).
Question: A person is traveling 20 km/hr

---

and reached his destiny in 2.5 hr then find the distance?
Llama-2 series, MPT series, UL2 series, and T5 rics. The open-souree model covers diverse model architectures (decoder only and encoder-decoder), model size (from 7b to 180b), training objectives (e.g., with or without further finetuning) and tok- enizers (c.g.. tokenizer-free: ByT5-xxI).
Choices: (A)53 km (B)55 km (C)52 km (D)60 km (E)50 km
Answer: The distance that the person traveled would have been 20 km/hr 2.5 hrs = 50 km. The answer is (E).
For GPT-4, the version GPT-4-0314 is used. For GPT-3.5-turbo, the version GPT-3.5-turbo-0301 is used. For Falcon-1 80b and Falcon-180b-char the quantized method (Dettmers al., 2023) is used load the model and run the experiments. It probably affects their performance to some extent.
Question: How many keystrokes needed type the numbers from to 500?
Choices: (A)1156 (B)1392 (C)1480 (D)1562 (E)1788
Answer: There are 9 one-digit numbers from to
C.1 Full experimental results on scrambled RealtimeQA dataset
9. There are 90 two-digit numbers from 10 to 99, There are 401 three-digit numbers from 100 to 500. 9 90(2) + (3)(3) 1392. The answer is (B).
Table 3 and Table 4 illustrates the full experimental results on scrambled Realtime dataset for few- shot ScrRec and zero-shot ScrRec, respectively. Table 5 illustrates the full experimental results on scrambled RealtimeQA dataset for zero-shot ScrQA.
Question: ieahMnc A psut otu a oy-oy ryeve nsimtue. enicMah pstu uto oy-oy veeyr nmusite. tAefr hwo aynm smtnuei Iwil yeth hvea eurdpeod 10 oy-soy?
Choices: (A)24 minutes (B)32 minutes (C)36
C.2
Full experimental results scrambled DREAM dataset
minutes (D)64 minutes (E)72 minutes
Answer:
Output:
Table illustrates the full experimental results on scrambled DREAM dataset for zero-shot ScrQA. Table and Table 8 illustrates the experimental results on different question types of the top five models (like in Figure 5) with accuracy and RPG as the metrics, respectively.
First, let's rewrite the question with the words unserambled:
"Machine puts out a yo-yo every 6 minutes. Machine puts out yo-yo every minutes. After how many minutes will they have produced 10 yo-yos?"
To solve this problem, we can find the least common multiple (LCM) of 6 and 9, which is 18. This means that every 18 minutes, both machines will have produced yo-yo the same time. In 18 minutes, Machine A will have produced yo-yos (18/6) and Machine will have produced 2 yo-yos (18/9), for a total of 5 yo-yos. Since we need 10 yo-yos, we can multiply 18 minutes by 2 to get 36 minutes. The answer is (C) 36 minutes.
B Summary of related work
Table 2 categorizes the related work and demon- strates the position of our study.
Full experimental results
We conduct experiments using the most powerful closed-source LLMs,, including text-davinci-003: GPT-3.5-turis and GPT-4 and various soures LLMs, including the models from Falcon series,

---

Model
20%
50%
100% %
KF
KFL
1.49
2.40
4.79
3.79
2.24
GPT-4
94.51
96.46
96.45
96.37
97.04
5.80
15.38
36.87
20.76
7.40
GPT-3.5-turbo
78.57
77.29
72.63
80.14
90.22
5.38
18.89
84.39
22.21
8.36
text-davinci-003
80.13
72.11
37.35
78.75
88.95
4.18
21.84
116.73
28.77
9.40
Falcon-1888 180b
84.55
67.76
13.34
72.47
87.58
11.61
38.19
186.65
63.51
21.73
Falcon-180b-chat
57.12
43.62
<0
39.24
71.28
17.27
69.86
185.26
119.00
66.03
Falcon-40b
36.18
<0
<0
<0
12.71
18.79
74.26
192.86
130.82
62.39
Falcon-40b-instruet
30.56
<0
<0
<0
17.53
7.85
37.30
169.85
56.25
18.99
Llama-2-70b
71.0
44.93
<0
46.19
74.9
11.11
54.49
196.44
106.31
43.18
Llama-2-7bb-chat
58.97
19.56
<0
<0
42.93
14.18
59.13
192.67
103.50
42.67
Hlama-2-13b
47.62
12.71
<0
0.99
43.59
20.75
73.04
201.50
119.52
56.03
Llaha-2-13b-chat
23.35
<0 0
<0
<0
25.93
20.90
79.83
194.48
135.50
68.18
Llama-2-7h
22.77
<0
<0
9.88
25.14
83.13
196.88
132.44
73.84
Llama-2-7b-cha
7.11
0
<0
< <0
2.39
14.36
54.81
181.43
90.92
36.14
MPT-30b
46.96
19.08
<0
13.02
52.23
19.36
66.05
170.88
99.38
50.44
MPT-30b-instruct
28.46
2.49
<
4.92
33.33
18.12
68.79
153.53
109.47
71.70
UL2
33.03
0
5.22
20.51
62.91
137.15
105.67
74.35
Flan-UUL
24.21
7.13
<0
<0
1.72
16.60
59.34
136.41
105.44
73.97
Flan-T5-xxl
38.68
12.4
<0
<0
2.22
79.27
104.28
160.17
123.97
102.50
ByT5-xxl
<0
<
<
<0
<0
Table 3: The experimental results the few-shot ScrRec using RealtimeQA dataset (upper: ED: lower: RR).

---

Model
20%
50%
100%
KF
KFL
1.69
2.78
5.47
4.19
2.53
GPT-4
93.76
95.89
95.94
95.99
96.66
4.68
13.00
29.80
14.22
6.94
GPT-3.5-turbo
82.72
80.82
77.88
86.39
90.82
15.02
31.86
109.63
30.88
13.30
text-davinci-003
44.52
52.96
18.61
70.45
82.41
7.18
26.53
159.36
43.63
13.08
Falcon-1888 180b
73.48
60.84
<0
58.25
82.72
15.92
50.50
180.13
76.31
29.29
Falcon-180b-chat
41.17
25.46
< 0
27.0
61.29
20.59
66.33
177.33
92.45
39.04
Falcon-40b
23.94
2.08
<0
11.56
48.39
26.42
78.94
178.55
104.46
48.42
Falcon-40b-nnstruct
2.4
<0
<0
0.07
36.0
8.11
41.10
180.82
60.16
21.68
Llama-2-70b
70.04
39.32
<0
42.45
71.34
12.65
60.23
181.30
118.13
51.73
Llama-2-70b-chat
53.26
11.08
<0
<0
31.62
20.21
75.54
231.41
127.63
59.77
Hlama-2-13b
25.32
<
<0
< 0
20.99
26.79
84.09
191.85
120.51
63.82
Llaha-2-13b-chat
1.03
<0
<0
<0
15.64
29.81
106.38
192.05
165.56
88.65
Llama-2-7h
<0
<0
<
<0
<0
48.92
123.66
202.16
167.49
112.03
Llama-2-7b-cha
<
<0
<0
<0
<
20.54
70.69
226.34
135.58
62.16
MPT-30b
24.11
<0
<0
<0 0
17.84
36.01
84.98
233.60
132.77
76.60
MPT-30b-instruct
<0
<0
<0
<0
<0
72.39
139.14
205.86
188.57
163.67
UL2
<
<
<0
<0
33.10
71.76
136.94
106.48
78.55
1lan-UL2
<0
<0
<0
<0
<0
27.07
68.02
136.47
106.08
76.69
Flan-T5-xxl
<0
<
<0
<0
<0
85.83
111.53
155.27
136.99
117.80
ByT5-xxl
<0
<0
<
<0
<
Table 4: The experimental results on the zero-shot ScrRec using RealtimeQA dataset (upper: ED; lower: RR).

---

Model
Ori
20%
50%
100%
KF
KFL
Sub
97.69
97.40
96.82
93.35
93.35
95.66
62.14
GPT-4
100.0
99.19
97.56
87.80
87.80
94.31
0.0
97.11
94.80
91.33
81.79
88.44
93.06
52.60
GPT-3.5-turbo
100.0
94.81
87.01
65.58
80.52
90.91
0.0
93.93
91.62
87.28
77.46
84.10
88.15
54.91
text-davinci-003
100.0
94.07
82.96
57.78
74.81
85.19
0.0
84.68
82.37
78.61
65.03
70.52
74.28
57.23
Falcon-180b
100.0
91.58
77.89
28.42
48.42
62.11
0.0
75.43
72.54
71.39
58.09
63.87
67.34
54.62
Falcon-180b-chat
100.0
86.11
80.56
16.67
44.44
61.11
0.0
58.96
59.25
54.62
44.51
44.51
50.29
39.60
Falcon-40b
100.0
101.49
77.61
25.37
25.37
55.22
0.0
65.90
63.87
61.27
48.55
53.47
60.69
46.82
40bbin--0b-
100.0
89.39
75.76
9.09
34.85
72.73
0.0
88.44
86.13
80.35
67.92
75.43
80.64
57.23
Llama-2-70b
100.0
92.59
74.07
34.26
58.33
75.00
0.0
96.24
92.49
83.82
64.74
75.43
82.66
54.91
Llama-2-70b-chat
100.0
90.91
69.93
23.78
49.65
67.13
0.0
66.18
65.03
57.51
47.40
52.89
57.80
44.80
Llama-2-13b
100.0
94.59
59.46
12.16
37.84
60.81
0.0
84.68
78.90
66.18
47.98
55.20
59.83
44.51
Jlama-2-13b-chaa
100.0
85.61
53.96
8.63
26.62
38.13
0.0
47.40
45.09
42.77
38.73
40.75
43.64
39.31
Llama-2-7b
100.0
71.43
42.86
0
17.86
53.57
0.0
82.08
75.14
64.16
45.09
52.31
56.94
43.93
1lama-2-7b-chan
100.0
81.82
53.03
3.03
21.97
34.09
0.0
62.72
56.36
50.87
37.86
47.69
53.47
37.28
MPT-30b
100.0
75.00
53.41
2.27
40.91
63.64
0.0
65.03
66.76
58.67
47.11
54.62
61.27
39.88
MPT-30b-instruct
100.0
106.90
74.71
28.74
58.62
85.06
0.0
87.57
84.68
76.30
45.95
60.69
71.10
38.44
UL2
100.0
94.12
77.06
15.29
45.29
66.47
0.0
98.55
95.95
89.31
49.71
65.61
76.88
46.24
Flan-UL2
100.0
95.03
82.32
6.63
37.02
58.56
0.0
97.40
96.24
87.86
49.42
66.47
82.66
40.75
Flan-T5-xxl
100.0
97.96
83.16
15.31
45.41
73.98
0.0
73.12
57.23
46.24
26.30
35.84
41.91
25.14
ByT5-xxl
100.0
66.87
43.98
2.41
22.29
34.94
0.0
Table 5: The experimental results on the zero-shot ScrQA using RealtimeQA dataset (upper: Acc; lower: RPG).

---

Model
Ori
20%
50%
100%
KF
KFL
Sub
96.68
96.29
94.43
87.40
93.65
96.39
46.48
GPT-4
100.0
99.22
95.53
81.52
93.97
99.42
0.0
92.48
87.30
77.83
45.61
71.78
83.89
26.07
GPT-3.5-turbo
100.0
92.21
77.94
29.41
68.82
87.06
0.0
92.29
89.45
80.27
52.44
76.66
87.50
43.85
text-davinci-003
100.0
94.15
75.20
17.74
67.74
90.12
0.0
93.65
89.65
82.23
55.37
75.39
84.47
51.17
Falcon-180b
100.0
90.57
73.10
9.89
57.01
78.39
0.0
92.68
89.16
78.32
49.90
70.51
80.76
45.31
Valcon-180b-chat
100.0
92.58
69.69
9.69
53.20
74.85
0.0
70.12
65.04
56.45
44.34
50.78
58.30
43.26
Falcon-40b
100.0
81.09
49.09
4.00
28.00
56.00
0.0
75.20
67.48
58.59
43.65
50.00
60.06
42.87
Falcon-40b-instrue
100.0
76.13
48.64
2.42
22.05
53.17
0.0
93.26
89.75
80.96
54.30
73.34
84.38
47.85
Llama-2-70b
100.0
92.26
72.90
14.19
56.13
80.43
0.0
91.11
85.74
75.10
48.24
59.67
74.02
45.12
Llama-2-70b-chat
100.0
88.32
65.18
6.79
31.63
62.85
0.0
69.53
63.87
53.32
38.67
44.63
52.25
37.21
Llama-2-133
100.0
82.48
49.85
4.53
22.96
46.53
0.0
78.61
71.78
59.77
41.80
48.44
57.42
40.92
Llama-2-13b-cha
100.0
81.87
50.00
2.33
19.95
43.78
0.0
60.55
55.08
47.46
37.60
43.07
46.39
37.30
Llama-2-7b
100.0
76.47
43.70
1.26
24.79
39.08
0.0
73.63
64.45
56.25
40.04
45.41
53.81
39.84
Llama-2-7b-dhe
100.0
72.83
48.55
0.58
16.47
41.33
0.0
74.02
69.24
59.47
47.17
50.00
57.71
43.46
MPT-30b
100.0
84.35
52.40
12.14
21.41
46.65
0.0
83.50
75.68
65.43
44.82
54.30
60.64
41.41
MPP-30b-instruct
100.0
81.44
57.08
8.12
30.63
45.71
0.0
43.36
39.36
29.88
14.55
17.38
25.20
12.79
UL2
100.0
86.90
55.91
5.75
15.02
40.58
0.0
94.24
91.21
80.96
49.80
62.99
75.39
48.14 14
Flan-ULX
100.0
93.43
71.19
3.60
32.20
59.11
0.0
95.02
92.87
84.67
51.17
67.09
81.54
51.07
Flan-T5-xxl
100.0
95.11
76.44
0.22
36.44
69.33
0.0
43.16
40.23
36.23
31.93
33.50
36.62
31.84
ByT5-xxl
100.0
74.14
38.79
0.86
14.66
42.24
0.0
Table 6: The experimental results on the zero-shot ScrQA using DREAM dataset (upper: Ace; lower: RPG).

---

Model GPT44
Type
Ori
20%
50%
100% %
KF
KFL
Sub
all
96.68
96.29
94.43
87.40
93.65
96.39
46.48
arithmetic
76.09
78.26
67.39
45.65
58.70
71.74
32.61
commonsense
94.69
94.41
93.02
82.40
91.90
94.41
42.46
logic
95.97
95.84
93.42
85.77
92.62
95.97
47.65
matching
99.07
97.22
98.15
92.59
97.22
99.07
47.22
summary
98.47
97.71
96.18
90.08
95.42
96.95
36.64
all
92.48
87.30
77.83
45.61
71.78
83.89
26.07
arithmetic
60.87
60.87
52.17
21.74
41.30
50.00
15.22
commonsense
89.11
84.36
76.54
39.39
63.69
79.61
25,42
GPT-3.5-turbo
logic
91.01
85.64
76.11
44.83
71.28
82.95
25.10
matching
99.07
91.67
81.48
56.48
78.70
89.81
30.56
summary
95.42
93.13
82.44
41.22
70.99
83.97
24.43
all
92.29
89.45
80.27
52.44
76.66
87.50
43.85
arithmetic
47.83
45.65
41.30
21.74
41.30
45.65
17.39
commonsense
90.78
87.99
77.37
46.37
71.79
84.92
45.25
text-davinci-003
logic
91.01
87.92
78.39
52.89
75.03
86.71
41.34
matching
99.07
94.44
86.11
57.41
87.96
93.52
48.15
summary
94.66
94.66
86.26
48.09
77.86
87.79
49.62
all
93.65
89.65
82.23
55.37
75.39
84.47
51.17
arithmetic
52.17
47.83
45.65
34.78
43.48
47.83
34.78
commonsense
91.34
88.55
79.33
56.15
74.58
83.24
52.23
Falcon-180b
logic
92.62
88.59
79.87
52.35
73.02
82.55
50.47
matching
98.15
90.74
87.04
62.96
78.70
89.81
51.85
summary
96.18
93.89
89.31
67.18
87.02
91.60
56.49
all
93.26
89.75
80.96
54.30
73.34
84.38
47.85
arithmetic
58.70
47.83
50.00
36.96
47.83
43.48
36.96
commonsense
91.34
89.39
80.73
53.63
73.18
84.36
44.97
Llama-2-70b
logic
92.62
88.99
79.73
54.50
73.02
84.16
48.05
matching
94.44
89.81
79.63
50.93
73.15
79.63
50.93
summary
96.18
94.66
86.26
58.02
79.39
90.84
50.38
Table 7: The experimental results of different categories on the zero-shot ScrQA using DREAM dataset (Acc).

---

Model
Type
Ori
20%
50%
100 %
KF
KFL
Sub
GPT-4
all
100.0
99.22
95.53
81.52
93.97
99.42
0.0
arithmetic
100.0
105.00
80.00
30.00
60.00
90.00
0.0
commonsense
100.0
99.47
96.79
76.47
94.65
99.47
0.0
logic
100.0
99.72
94.72
78.89
93.06
100.00
0.0
matching
100.0
96.43
98.21
87.50
96.43
100.00
0.0
summary
100.0
98.77
96.30
86.42
95.06
97.53
all
100.0
92.21
77.94
29.41
68.82
87.06
0.0
arithmetic
100.0
100.00
80.95
14.29
57.14
76.19
0.0
commonsense
100.0
92.54
80.26
21.93
60.09
85.09
0.0
GPT-3.5-turbo
logic
100.0
91.85
77.39
29.94
70.06
87.78
0.0
matching
100.0
89.19
74.32
37.84
70.27
86.49
0.0
summary
100.0
96.77
81.72
23.66
65.59
83.87
0.0
all
100.0
94.15
75.20
17.74
67.74
90.12
0.0
arithmetic
100.0
92.86
78.57
14.29
78.57
92.86
0.0
commonsense
100.0
93.87
70.55
2.45
58.28
87.12
0.0
text-davinei-003
logic
100.0
93.78
74.59
23.24
67.84
91.35
0.0
matching
100.0
90.91
74.55
18.18
78.18
89.09
0.0
summary
100.0
100.00
81.36
<
62.71
84.75
0.0
all
100.0
90.57
73.10
9.89
57.01
78.39
0.0
arithmetic
100.0
75.00
62.50
0.00
50.00
75.00
0.0
commonsense
100.0
92.86
69.29
10.00
57.14
79.29
0.0
Falcon-180b
logic
100.0
90.45
69.75
4.46
53.50
76.11
0.0
matching
100.0
84.00
76.00
24.00
58.00
82.00
0.0
summary
100.0
94.23
82.69
26.92
76.92
88.46
0.0
all
100.0
92.26
72.90
14.19
56.13
80.43
0.0
arithmetic
100.0
50.00
60.00
0.00
50.00
30.00
0.0
commonsense
100.0
95.78
77.11
18.67
60.84
84.94
0.0
Llama-2-70b
logic
100.0
91.87
71.08
14.46
56.02
81.02
0.0
matching
100.0
89.36
65.96
0.00
51.06
65.96
0.0
summary
100.0
96.67
78.33
16.67
63.33
88.33
0.0
Table 8: The experimental results of different categories on the zero-shot ScrQA using DREAM dataset (RPG).


Reviews:
[{"review_id": "Fc5CtISBQZ", "reviewer": "Reviewer_NrTg", "summary": "", "questions": "Why GPT-4 performs better than text-davinci-003 and GPT-3,5-turbo?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer NrTg:\n\nThank you very much for your review.\n\n> 1. Did not test the performance of character-based/tokenizer-free models such as T5. \n\nWe add T5 series models, including byt5-xxl (tokenizer-free). The original T5 is not included in results, since it is completely not capable in this setting.\n\n> 2. Only test on models that are not public. It would be interesting to test the benchmark on public models such as LLaMa and see the distribution, tokenization results, and other statistics for further analysis.\n> 3. No further analysis.\n\nFollowing the reviewers’ comments, we conducted experiments using various open-source LLMs, including (llama-2, falcon, mpt, ul2, flan-ul2, flan-t5, and byt5). The model covers diverse model architectures (decoder only and encoder-decoder), model sizes (from 7b to 70b), and training objectives (e.g., with or without further finetuning) and tokenizers (e.g., tokenizer-free: byt5-xxl). The results are shown below:\n\n**Scrambled Sentence Recovery on RealtimeQA (edit distance)**\n|                     |  20percent | 50percent | 100percent | keepfirst | keepfirstlast |\n|---------------------|------------|----------:|-----------:|----------:|--------------:|\n|          gpt-4-0314 |       1.49 |      2.40 |       4.79 |      3.79 |          2.24 |\n|  gpt-3.5-turbo-0301 |       5.80 |     15.38 |      36.87 |     20.76 |          7.40 |\n|    text-davinci-003 |       5.38 |     18.89 |      84.39 |     22.21 |          8.36 |\n|      Llama-2-70b-hf |       7.85 |     37.30 |     169.85 |     56.25 |         18.99 |\n| Llama-2-70b-chat-hf |      11.11 |     54.49 |     196.44 |    106.31 |         43.18 |\n|      Llama-2-13b-hf |      14.18 |     59.13 |     192.67 |    103.50 |         42.67 |\n| Llama-2-13b-chat-hf |      20.75 |     73.04 |     201.50 |    119.52 |         56.03 |\n|       Llama-2-7b-hf |      20.90 |     79.83 |     194.48 |    135.50 |         68.18 |\n|  Llama-2-7b-chat-hf |      25.14 |     83.13 |     196.88 |    132.44 |         73.84 |\n|          falcon-40b |      17.27 |     69.86 |     185.26 |    119.00 |         66.03 |\n| falcon-40b-instruct |      18.79 |     74.26 |     192.86 |    130.82 |         62.39 |\n|             mpt-30b |      14.36 |     54.81 |     181.43 |     90.92 |         36.14 |\n|    mpt-30b-instruct |      19.36 |     66.05 |     170.88 |     99.38 |         50.44 |\n|                 ul2 |      18.12 |     68.79 |     153.53 |    109.47 |         71.70 |\n|            flan-ul2 |      20.51 |     62.91 |     137.15 |    105.67 |         74.35 |\n|         flan-t5-xxl |      16.60 |     59.34 |     136.41 |    105.44 |         73.97 |\n|            byt5-xxl |      79.27 |    104.28 |     148.44 |    123.97 |        102.50 |\n\n\n**Scrambled Sentence Recovery on RealtimeQA(recovery rate)**\n|                     |  20percent | 50percent | 100percent | keepfirst | keepfirstlast |\n|---------------------|------------|----------:|-----------:|----------:|--------------:|\n|          gpt-4-0314 |      94.51 |     96.46 |      96.45 |     96.37 |         97.04 |\n|  gpt-3.5-turbo-0301 |      78.57 |     77.29 |      72.63 |     80.14 |         90.22 |\n|    text-davinci-003 |      80.13 |     72.11 |      37.35 |     78.75 |         88.95 |\n|      Llama-2-70b-hf |       71.0 |     44.93 |         <0 |     46.19 |          74.9 |\n| Llama-2-70b-chat-hf |      58.97 |     19.56 |         <0 |        <0 |         42.93 |\n|      Llama-2-13b-hf |      47.62 |     12.71 |         <0 |      0.99 |         43.59 |\n| Llama-2-13b-chat-hf |      23.35 |        <0 |         <0 |        <0 |         25.93 |\n|       Llama-2-7b-hf |      22.77 |        <0 |         <0 |        <0 |          9.88 |\n|  Llama-2-7b-chat-hf |       7.11 |        <0 |         <0 |        <0 |          2.39 |\n|          falcon-40b |      36.18 |        <0 |         <0 |        <0 |         12.71 |\n| falcon-40b-instruct |      30.56 |        <0 |         <0 |        <0 |         17.53 |\n|             mpt-30b |      46.96 |     19.08 |         <0 |     13.02 |         52.23 |\n|    mpt-30b-instruct |      28.46 |      2.49 |         <0 |      4.92 |         33.33 |\n|                 ul2 |      33.03 |        <0 |         <0 |        <0 |          5.22 |\n|            flan-ul2 |      24.21 |      7.13 |         <0 |        <0 |          1.72 |\n|         flan-t5-xxl |      38.68 |      12.4 |         <0 |        <0 |          2.22 |\n|            byt5-xxl |         <0 |        <0 |         <0 |        <0 |            <0 |\n\n\n**Scrambled QA on RealtimeQA (accuracy)**\n|                     |  original | 20percent | 50percent | 100percent | substituted | keepfirst | keepfirstlast |\n|---------------------|-----------|----------:|----------:|-----------:|------------:|----------:|--------------:|\n|          gpt-4-0314 |     98.18 |     97.26 |     96.05 |      93.01 |       62.92 |     93.92 |         95.74 |\n|  gpt-3.5-turbo-0301 |     97.26 |     96.66 |     91.19 |      84.50 |       55.62 |     86.93 |         91.49 |\n|    text-davinci-003 |     94.22 |     92.10 |     87.84 |      78.72 |       55.62 |     82.98 |         91.49 |\n|      Llama-2-70b-hf |     88.75 |     86.32 |     82.98 |      65.65 |       59.88 |     78.42 |         82.67 |\n| Llama-2-70b-chat-hf |     96.05 |     92.71 |     85.71 |      64.74 |       55.62 |     75.68 |         84.19 |\n|      Llama-2-13b-hf |     66.87 |     62.92 |     57.75 |      46.81 |       46.20 |     55.02 |         55.62 |\n| Llama-2-13b-chat-hf |     85.41 |     75.99 |     66.87 |      48.02 |       45.90 |     55.32 |         61.40 |\n|       Llama-2-7b-hf |     46.81 |     44.38 |     41.64 |      40.73 |       38.30 |     40.12 |         44.07 |\n|  Llama-2-7b-chat-hf |     82.67 |     77.51 |     65.05 |      46.20 |       46.20 |     50.76 |         60.79 |\n|          falcon-40b |     49.85 |     47.42 |     41.03 |      29.79 |       28.88 |     31.00 |         34.65 |\n| falcon-40b-instruct |     57.14 |     56.23 |     48.94 |      35.56 |       33.13 |     39.21 |         41.03 |\n|             mpt-30b |     62.31 |     56.53 |     50.15 |      41.64 |       39.51 |     47.42 |         50.15 |\n|    mpt-30b-instruct |     64.13 |     63.83 |     58.97 |      48.33 |       42.86 |     56.23 |         62.92 |\n|                 ul2 |     87.23 |     84.19 |     75.68 |      48.94 |       38.60 |     62.92 |         70.52 |\n|         flan-t5-xxl |     97.26 |     94.53 |     88.15 |      50.76 |       38.30 |     65.35 |         79.64 |\n|            byt5-xxl |     72.04 |     56.84 |     46.81 |      27.36 |       24.62 |     34.35 |         42.55 |\n\n\n**Scrambled QA on RealtimeQA (relative accuracy gain)**\n|                     |  original | 20percent | 50percent | 100percent | substituted | keepfirst | keepfirstlast |\n|---------------------|-----------|----------:|----------:|-----------:|------------:|----------:|--------------:|\n|          gpt-4-0314 |       1.0 |      0.97 |      0.94 |       0.85 |         0.0 |      0.88 |          0.93 |\n|  gpt-3.5-turbo-0301 |       1.0 |      0.99 |      0.85 |       0.69 |         0.0 |      0.75 |          0.86 |\n|    text-davinci-003 |       1.0 |      0.94 |      0.83 |       0.60 |         0.0 |      0.71 |          0.93 |\n|      Llama-2-70b-hf |       1.0 |      0.92 |      0.80 |       0.20 |         0.0 |      0.64 |          0.79 |\n| Llama-2-70b-chat-hf |       1.0 |      0.92 |      0.74 |       0.23 |         0.0 |      0.50 |          0.71 |\n|      Llama-2-13b-hf |       1.0 |      0.81 |      0.56 |       0.03 |         0.0 |      0.43 |          0.46 |\n| Llama-2-13b-chat-hf |       1.0 |      0.76 |      0.53 |       0.05 |         0.0 |      0.24 |          0.39 |\n|       Llama-2-7b-hf |       1.0 |      0.71 |      0.39 |       0.29 |         0.0 |      0.21 |          0.68 |\n|  Llama-2-7b-chat-hf |       1.0 |      0.86 |      0.52 |       0.00 |         0.0 |      0.13 |          0.40 |\n|          falcon-40b |       1.0 |      0.88 |      0.58 |       0.04 |         0.0 |      0.10 |          0.28 |\n| falcon-40b-instruct |       1.0 |      0.96 |      0.66 |       0.10 |         0.0 |      0.25 |          0.33 |\n|             mpt-30b |       1.0 |      0.75 |      0.47 |       0.09 |         0.0 |      0.35 |          0.47 |\n|    mpt-30b-instruct |       1.0 |      0.99 |      0.76 |       0.26 |         0.0 |      0.63 |          0.94 |\n|                 ul2 |       1.0 |      0.94 |      0.76 |       0.21 |         0.0 |      0.50 |          0.66 |\n|         flan-t5-xxl |       1.0 |      0.95 |      0.85 |       0.21 |         0.0 |      0.46 |          0.70 |\n|            byt5-xxl |       1.0 |      0.68 |      0.47 |       0.06 |         0.0 |      0.21 |          0.38 |\n\n\n**Scrambled QA on DREAM (accuracy)**\n|                     |  original | 20percent | 50percent | 100percent | substituted | keepfirst | keepfirstlast |\n|---------------------|-----------|----------:|----------:|-----------:|------------:|----------:|--------------:|\n|          gpt-4-0314 |     96.68 |     96.29 |     94.73 |      88.48 |       45.70 |     93.95 |         96.48 |\n|  gpt-3.5-turbo-0301 |     94.92 |     90.23 |     81.05 |      44.73 |       21.29 |     73.44 |         85.74 |\n|    text-davinci-003 |     93.36 |     91.02 |     81.84 |      54.69 |       45.12 |     79.30 |         88.48 |\n|      Llama-2-70b-hf |     92.97 |     90.62 |     83.40 |      57.23 |       50.20 |     74.02 |         85.16 |\n| Llama-2-70b-chat-hf |     91.02 |     86.13 |     76.95 |      48.44 |       45.70 |     59.77 |         74.41 |\n|      Llama-2-13b-hf |     70.51 |     65.62 |     54.69 |      40.82 |       38.28 |     42.38 |         52.93 |\n| Llama-2-13b-chat-hf |     80.08 |     74.80 |     62.89 |      42.58 |       41.02 |     48.05 |         58.59 |\n|       Llama-2-7b-hf |     60.74 |     55.66 |     48.83 |      38.48 |       38.67 |     42.97 |         46.29 |\n|  Llama-2-7b-chat-hf |     75.00 |     66.21 |     57.03 |      40.04 |       38.09 |     42.97 |         52.93 |\n|          falcon-40b |     64.06 |     59.77 |     53.52 |      44.34 |       42.77 |     48.63 |         54.69 |\n| falcon-40b-instruct |     73.05 |     65.62 |     57.23 |      42.97 |       43.16 |     48.63 |         57.03 |\n|             mpt-30b |     76.37 |     73.24 |     60.74 |      47.85 |       45.31 |     50.78 |         58.98 |\n|    mpt-30b-instruct |     86.13 |     78.52 |     69.53 |      45.12 |       42.77 |     55.47 |         63.09 |\n|                 ul2 |     44.14 |     41.02 |     33.20 |      20.31 |       17.77 |     22.27 |         28.71 |\n|            flan-ul2 |     94.53 |     91.02 |     82.23 |      53.71 |       49.22 |     65.23 |         76.37 |\n|         flan-t5-xxl |     96.68 |     94.34 |     86.72 |      54.49 |       53.52 |     70.70 |         84.77 |\n\n\n**Scrambled QA on DREAM (relative accuracy gain)**\n|                     |  original | 20percent | 50percent | 100percent | substituted | keepfirst | keepfirstlast |\n|---------------------|-----------|----------:|----------:|-----------:|------------:|----------:|--------------:|\n|          gpt-4-0314 |       1.0 |      0.99 |      0.96 |       0.84 |         0.0 |      0.95 |          1.00 |\n|  gpt-3.5-turbo-0301 |       1.0 |      0.94 |      0.81 |       0.32 |         0.0 |      0.71 |          0.88 |\n|    text-davinci-003 |       1.0 |      0.95 |      0.76 |       0.20 |         0.0 |      0.71 |          0.90 |\n|      Llama-2-70b-hf |       1.0 |      0.95 |      0.78 |       0.16 |         0.0 |      0.56 |          0.82 |\n| Llama-2-70b-chat-hf |       1.0 |      0.89 |      0.69 |       0.06 |         0.0 |      0.31 |          0.63 |\n|      Llama-2-13b-hf |       1.0 |      0.85 |      0.51 |       0.08 |         0.0 |      0.13 |          0.45 |\n| Llama-2-13b-chat-hf |       1.0 |      0.86 |      0.56 |       0.04 |         0.0 |      0.18 |          0.45 |\n|       Llama-2-7b-hf |       1.0 |      0.77 |      0.46 |      -0.01 |         0.0 |      0.19 |          0.35 |\n|  Llama-2-7b-chat-hf |       1.0 |      0.76 |      0.51 |       0.05 |         0.0 |      0.13 |          0.40 |\n|          falcon-40b |       1.0 |      0.80 |      0.50 |       0.07 |         0.0 |      0.28 |          0.56 |\n| falcon-40b-instruct |       1.0 |      0.75 |      0.47 |      -0.01 |         0.0 |      0.18 |          0.46 |\n|             mpt-30b |       1.0 |      0.90 |      0.50 |       0.08 |         0.0 |      0.18 |          0.44 |\n|    mpt-30b-instruct |       1.0 |      0.82 |      0.62 |       0.05 |         0.0 |      0.29 |          0.47 |\n|                 ul2 |       1.0 |      0.88 |      0.59 |       0.10 |         0.0 |      0.17 |          0.41 |\n|            flan-ul2 |       1.0 |      0.92 |      0.73 |       0.10 |         0.0 |      0.35 |          0.60 |\n|         flan-t5-xxl |       1.0 |      0.95 |      0.77 |       0.02 |         0.0 |      0.40 |          0.72 |\n\nFor the further analysis, we intuitively thought that LLMs handle scrambled text by mapping the scrambled tokens to original ones. To investigate this, we analyze the hidden state of Llama-2 when inputting scrambled sentences. We calculate the cosine similarity of the hidden state (excluding embedding layers and last two layers, since they are strongly related to input and output, respectively) of each pair of scrambled sentences and original sentences. Then, we average the cosine similarity and normalize it from 0 (sentences with randomly substituted letters) to 1 (original sentences). The results are shown in the table below (it is difficult to show layer-wise comparison, since not figure allowed). For models of all three sizes, the similarity decreases through scrambling severity increases. And the similarity is not related to the performance on tasks. We also sort the samples by recovery rate and separate the samples into two groups based on the correctness on QA tasks and compare their similarities. However, no patterns can be found. So, we assume that LLMs handle scrambled sentences not by mapping them to original ones.\n\n|     | 10%  | 20%  | 30%  | 40%  | 50%  | 60%  | 70%  | 80%  | 90%  | 100% | kf   | kfl  |\n|-----|------|------|------|------|------|------|------|------|------|------|------|------|\n| 70b | 0.91 | 0.81 | 0.72 | 0.63 | 0.54 | 0.46 | 0.37 | 0.29 | 0.23 | 0.15 | 0.27 | 0.41 |\n| 13b | 0.92 | 0.81 |  0.7 | 0.59 | 0.49 | 0.39 |  0.3 | 0.22 | 0.15 | 0.09 | 0.18 | 0.32 |\n| 7b  | 0.97 | 0.92 | 0.86 |  0.8 | 0.73 | 0.67 |  0.6 | 0.53 | 0.46 | 0.39 | 0.49 |  0.6 |\n\n> Why GPT-4 performs better than text-davinci-003 and GPT-3,5-turbo?\n\nBecause it’s difficult to access the models and there is no detailed description of models, it is difficult to draw conclusions about the reason why GPT-4 performs better. Since GPT-4 can do much better on these tasks, we assume this might be an emergent ability related to parameter size and the way of pre-training (e.g., one of possible reasons: enough contents with typos existing in training data).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposes a novel task\n2. This paper proposes a benchmark for the corresponding task\n4. Experiments on three models seem interesting. Never image that tokenizer-based models are immune to the character permutation.", "reasons_to_reject": "1.  Did not test the performance of character-based/tokenizer-free models such as T5. \n2. Only test on models that are not public. It would be interesting to test the benchmark on public models such as LLaMa and see the distribution, tokenization results, and other statistics for further analysis.\n3. No further analysis.", "questions_to_authors": "Why GPT-4 performs better than text-davinci-003 and GPT-3,5-turbo?", "paper_topic_and_main_contributions": "This paper focuses on the ability of LLMs to handle character-level scrambled text. The researchers created a benchmark and conducted experiments to test this, finding that GPT-4 excels at understanding scrambled text, even when the letters of all words are scrambled. Other LLMs perform well too, but GPT-4 stands out for its remarkable performance in processing inputs with unnatural errors.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nk6Bj75WnX", "reviewer": "Reviewer_tGXe", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer tGXe:\n\nThank you very much for your review!\n\n> My main concern is that the test sets (179 samples and 155 samples for the scrambled sentence recovery task and the scrambled question answering task respectively) are too small to draw reliable conclusions. Similarly, the experiments on an additional dataset for the scrambled question answering task only concern 100 samples.\n\nWe extend the scrambled realtimeqa dataset (using data from 2023/03/17 to 2023/08/04). The data sizes are extended to 408 and 329 for the scrambled sentence recovery task and the scrambled question answering task, respectively. We would like to evaluate open-source models on this task, but the additional dataset AQUA itself is too challenging to some open-source models. Instead, we process DREAM dataset for a dialogue comprehension task as another additional dataset for question answering task (512 samples with categorical labels are selected). The setting of DREAM is similar to the scrambled RealtimeQA dataset (ask LLMs to answer questions based on scrambled dialogues). Furthermore, for fairer comparisons, we set the results of context with randomly substituted letters to replace that of no context as the lower bound. Additionally, we also test open-source models (Llama-2 series, T5 series, Falcon series and MPT series) on these tasks.\nIn the results, the results of GPT models are similar to that we reported in the paper. GPT-4 can outperform all other models significantly, especially on the extreme situations. For open-source models, several ones (e.g., Llama-2-70b) can handle scrambled text of mild level. No open-source models can well handle the 100% scrambled text.\n\n**Scrambled Sentence Recovery on RealtimeQA (edit distance)**\n|                     |  20percent | 50percent | 100percent | keepfirst | keepfirstlast |\n|---------------------|------------|----------:|-----------:|----------:|--------------:|\n|          gpt-4-0314 |       1.49 |      2.40 |       4.79 |      3.79 |          2.24 |\n|  gpt-3.5-turbo-0301 |       5.80 |     15.38 |      36.87 |     20.76 |          7.40 |\n|    text-davinci-003 |       5.38 |     18.89 |      84.39 |     22.21 |          8.36 |\n|      Llama-2-70b-hf |       7.85 |     37.30 |     169.85 |     56.25 |         18.99 |\n| Llama-2-70b-chat-hf |      11.11 |     54.49 |     196.44 |    106.31 |         43.18 |\n|      Llama-2-13b-hf |      14.18 |     59.13 |     192.67 |    103.50 |         42.67 |\n| Llama-2-13b-chat-hf |      20.75 |     73.04 |     201.50 |    119.52 |         56.03 |\n|       Llama-2-7b-hf |      20.90 |     79.83 |     194.48 |    135.50 |         68.18 |\n|  Llama-2-7b-chat-hf |      25.14 |     83.13 |     196.88 |    132.44 |         73.84 |\n|          falcon-40b |      17.27 |     69.86 |     185.26 |    119.00 |         66.03 |\n| falcon-40b-instruct |      18.79 |     74.26 |     192.86 |    130.82 |         62.39 |\n|             mpt-30b |      14.36 |     54.81 |     181.43 |     90.92 |         36.14 |\n|    mpt-30b-instruct |      19.36 |     66.05 |     170.88 |     99.38 |         50.44 |\n|                 ul2 |      18.12 |     68.79 |     153.53 |    109.47 |         71.70 |\n|            flan-ul2 |      20.51 |     62.91 |     137.15 |    105.67 |         74.35 |\n|         flan-t5-xxl |      16.60 |     59.34 |     136.41 |    105.44 |         73.97 |\n|            byt5-xxl |      79.27 |    104.28 |     148.44 |    123.97 |        102.50 |\n\n\n**Scrambled Sentence Recovery on RealtimeQA(recovery rate)**\n|                     |  20percent | 50percent | 100percent | keepfirst | keepfirstlast |\n|---------------------|------------|----------:|-----------:|----------:|--------------:|\n|          gpt-4-0314 |      94.51 |     96.46 |      96.45 |     96.37 |         97.04 |\n|  gpt-3.5-turbo-0301 |      78.57 |     77.29 |      72.63 |     80.14 |         90.22 |\n|    text-davinci-003 |      80.13 |     72.11 |      37.35 |     78.75 |         88.95 |\n|      Llama-2-70b-hf |       71.0 |     44.93 |         <0 |     46.19 |          74.9 |\n| Llama-2-70b-chat-hf |      58.97 |     19.56 |         <0 |        <0 |         42.93 |\n|      Llama-2-13b-hf |      47.62 |     12.71 |         <0 |      0.99 |         43.59 |\n| Llama-2-13b-chat-hf |      23.35 |        <0 |         <0 |        <0 |         25.93 |\n|       Llama-2-7b-hf |      22.77 |        <0 |         <0 |        <0 |          9.88 |\n|  Llama-2-7b-chat-hf |       7.11 |        <0 |         <0 |        <0 |          2.39 |\n|          falcon-40b |      36.18 |        <0 |         <0 |        <0 |         12.71 |\n| falcon-40b-instruct |      30.56 |        <0 |         <0 |        <0 |         17.53 |\n|             mpt-30b |      46.96 |     19.08 |         <0 |     13.02 |         52.23 |\n|    mpt-30b-instruct |      28.46 |      2.49 |         <0 |      4.92 |         33.33 |\n|                 ul2 |      33.03 |        <0 |         <0 |        <0 |          5.22 |\n|            flan-ul2 |      24.21 |      7.13 |         <0 |        <0 |          1.72 |\n|         flan-t5-xxl |      38.68 |      12.4 |         <0 |        <0 |          2.22 |\n|            byt5-xxl |         <0 |        <0 |         <0 |        <0 |            <0 |\n\n\n**Scrambled QA on RealtimeQA (accuracy)**\n|                     |  original | 20percent | 50percent | 100percent | substituted | keepfirst | keepfirstlast |\n|---------------------|-----------|----------:|----------:|-----------:|------------:|----------:|--------------:|\n|          gpt-4-0314 |     98.18 |     97.26 |     96.05 |      93.01 |       62.92 |     93.92 |         95.74 |\n|  gpt-3.5-turbo-0301 |     97.26 |     96.66 |     91.19 |      84.50 |       55.62 |     86.93 |         91.49 |\n|    text-davinci-003 |     94.22 |     92.10 |     87.84 |      78.72 |       55.62 |     82.98 |         91.49 |\n|      Llama-2-70b-hf |     88.75 |     86.32 |     82.98 |      65.65 |       59.88 |     78.42 |         82.67 |\n| Llama-2-70b-chat-hf |     96.05 |     92.71 |     85.71 |      64.74 |       55.62 |     75.68 |         84.19 |\n|      Llama-2-13b-hf |     66.87 |     62.92 |     57.75 |      46.81 |       46.20 |     55.02 |         55.62 |\n| Llama-2-13b-chat-hf |     85.41 |     75.99 |     66.87 |      48.02 |       45.90 |     55.32 |         61.40 |\n|       Llama-2-7b-hf |     46.81 |     44.38 |     41.64 |      40.73 |       38.30 |     40.12 |         44.07 |\n|  Llama-2-7b-chat-hf |     82.67 |     77.51 |     65.05 |      46.20 |       46.20 |     50.76 |         60.79 |\n|          falcon-40b |     49.85 |     47.42 |     41.03 |      29.79 |       28.88 |     31.00 |         34.65 |\n| falcon-40b-instruct |     57.14 |     56.23 |     48.94 |      35.56 |       33.13 |     39.21 |         41.03 |\n|             mpt-30b |     62.31 |     56.53 |     50.15 |      41.64 |       39.51 |     47.42 |         50.15 |\n|    mpt-30b-instruct |     64.13 |     63.83 |     58.97 |      48.33 |       42.86 |     56.23 |         62.92 |\n|                 ul2 |     87.23 |     84.19 |     75.68 |      48.94 |       38.60 |     62.92 |         70.52 |\n|         flan-t5-xxl |     97.26 |     94.53 |     88.15 |      50.76 |       38.30 |     65.35 |         79.64 |\n|            byt5-xxl |     72.04 |     56.84 |     46.81 |      27.36 |       24.62 |     34.35 |         42.55 |\n\n\n**Scrambled QA on RealtimeQA (relative accuracy gain)**\n|                     |  original | 20percent | 50percent | 100percent | substituted | keepfirst | keepfirstlast |\n|---------------------|-----------|----------:|----------:|-----------:|------------:|----------:|--------------:|\n|          gpt-4-0314 |       1.0 |      0.97 |      0.94 |       0.85 |         0.0 |      0.88 |          0.93 |\n|  gpt-3.5-turbo-0301 |       1.0 |      0.99 |      0.85 |       0.69 |         0.0 |      0.75 |          0.86 |\n|    text-davinci-003 |       1.0 |      0.94 |      0.83 |       0.60 |         0.0 |      0.71 |          0.93 |\n|      Llama-2-70b-hf |       1.0 |      0.92 |      0.80 |       0.20 |         0.0 |      0.64 |          0.79 |\n| Llama-2-70b-chat-hf |       1.0 |      0.92 |      0.74 |       0.23 |         0.0 |      0.50 |          0.71 |\n|      Llama-2-13b-hf |       1.0 |      0.81 |      0.56 |       0.03 |         0.0 |      0.43 |          0.46 |\n| Llama-2-13b-chat-hf |       1.0 |      0.76 |      0.53 |       0.05 |         0.0 |      0.24 |          0.39 |\n|       Llama-2-7b-hf |       1.0 |      0.71 |      0.39 |       0.29 |         0.0 |      0.21 |          0.68 |\n|  Llama-2-7b-chat-hf |       1.0 |      0.86 |      0.52 |       0.00 |         0.0 |      0.13 |          0.40 |\n|          falcon-40b |       1.0 |      0.88 |      0.58 |       0.04 |         0.0 |      0.10 |          0.28 |\n| falcon-40b-instruct |       1.0 |      0.96 |      0.66 |       0.10 |         0.0 |      0.25 |          0.33 |\n|             mpt-30b |       1.0 |      0.75 |      0.47 |       0.09 |         0.0 |      0.35 |          0.47 |\n|    mpt-30b-instruct |       1.0 |      0.99 |      0.76 |       0.26 |         0.0 |      0.63 |          0.94 |\n|                 ul2 |       1.0 |      0.94 |      0.76 |       0.21 |         0.0 |      0.50 |          0.66 |\n|         flan-t5-xxl |       1.0 |      0.95 |      0.85 |       0.21 |         0.0 |      0.46 |          0.70 |\n|            byt5-xxl |       1.0 |      0.68 |      0.47 |       0.06 |         0.0 |      0.21 |          0.38 |\n\n\n**Scrambled QA on DREAM (accuracy)**\n|                     |  original | 20percent | 50percent | 100percent | substituted | keepfirst | keepfirstlast |\n|---------------------|-----------|----------:|----------:|-----------:|------------:|----------:|--------------:|\n|          gpt-4-0314 |     96.68 |     96.29 |     94.73 |      88.48 |       45.70 |     93.95 |         96.48 |\n|  gpt-3.5-turbo-0301 |     94.92 |     90.23 |     81.05 |      44.73 |       21.29 |     73.44 |         85.74 |\n|    text-davinci-003 |     93.36 |     91.02 |     81.84 |      54.69 |       45.12 |     79.30 |         88.48 |\n|      Llama-2-70b-hf |     92.97 |     90.62 |     83.40 |      57.23 |       50.20 |     74.02 |         85.16 |\n| Llama-2-70b-chat-hf |     91.02 |     86.13 |     76.95 |      48.44 |       45.70 |     59.77 |         74.41 |\n|      Llama-2-13b-hf |     70.51 |     65.62 |     54.69 |      40.82 |       38.28 |     42.38 |         52.93 |\n| Llama-2-13b-chat-hf |     80.08 |     74.80 |     62.89 |      42.58 |       41.02 |     48.05 |         58.59 |\n|       Llama-2-7b-hf |     60.74 |     55.66 |     48.83 |      38.48 |       38.67 |     42.97 |         46.29 |\n|  Llama-2-7b-chat-hf |     75.00 |     66.21 |     57.03 |      40.04 |       38.09 |     42.97 |         52.93 |\n|          falcon-40b |     64.06 |     59.77 |     53.52 |      44.34 |       42.77 |     48.63 |         54.69 |\n| falcon-40b-instruct |     73.05 |     65.62 |     57.23 |      42.97 |       43.16 |     48.63 |         57.03 |\n|             mpt-30b |     76.37 |     73.24 |     60.74 |      47.85 |       45.31 |     50.78 |         58.98 |\n|    mpt-30b-instruct |     86.13 |     78.52 |     69.53 |      45.12 |       42.77 |     55.47 |         63.09 |\n|                 ul2 |     44.14 |     41.02 |     33.20 |      20.31 |       17.77 |     22.27 |         28.71 |\n|            flan-ul2 |     94.53 |     91.02 |     82.23 |      53.71 |       49.22 |     65.23 |         76.37 |\n|         flan-t5-xxl |     96.68 |     94.34 |     86.72 |      54.49 |       53.52 |     70.70 |         84.77 |\n\n\n**Scrambled QA on DREAM (relative accuracy gain)**\n|                     |  original | 20percent | 50percent | 100percent | substituted | keepfirst | keepfirstlast |\n|---------------------|-----------|----------:|----------:|-----------:|------------:|----------:|--------------:|\n|          gpt-4-0314 |       1.0 |      0.99 |      0.96 |       0.84 |         0.0 |      0.95 |          1.00 |\n|  gpt-3.5-turbo-0301 |       1.0 |      0.94 |      0.81 |       0.32 |         0.0 |      0.71 |          0.88 |\n|    text-davinci-003 |       1.0 |      0.95 |      0.76 |       0.20 |         0.0 |      0.71 |          0.90 |\n|      Llama-2-70b-hf |       1.0 |      0.95 |      0.78 |       0.16 |         0.0 |      0.56 |          0.82 |\n| Llama-2-70b-chat-hf |       1.0 |      0.89 |      0.69 |       0.06 |         0.0 |      0.31 |          0.63 |\n|      Llama-2-13b-hf |       1.0 |      0.85 |      0.51 |       0.08 |         0.0 |      0.13 |          0.45 |\n| Llama-2-13b-chat-hf |       1.0 |      0.86 |      0.56 |       0.04 |         0.0 |      0.18 |          0.45 |\n|       Llama-2-7b-hf |       1.0 |      0.77 |      0.46 |      -0.01 |         0.0 |      0.19 |          0.35 |\n|  Llama-2-7b-chat-hf |       1.0 |      0.76 |      0.51 |       0.05 |         0.0 |      0.13 |          0.40 |\n|          falcon-40b |       1.0 |      0.80 |      0.50 |       0.07 |         0.0 |      0.28 |          0.56 |\n| falcon-40b-instruct |       1.0 |      0.75 |      0.47 |      -0.01 |         0.0 |      0.18 |          0.46 |\n|             mpt-30b |       1.0 |      0.90 |      0.50 |       0.08 |         0.0 |      0.18 |          0.44 |\n|    mpt-30b-instruct |       1.0 |      0.82 |      0.62 |       0.05 |         0.0 |      0.29 |          0.47 |\n|                 ul2 |       1.0 |      0.88 |      0.59 |       0.10 |         0.0 |      0.17 |          0.41 |\n|            flan-ul2 |       1.0 |      0.92 |      0.73 |       0.10 |         0.0 |      0.35 |          0.60 |\n|         flan-t5-xxl |       1.0 |      0.95 |      0.77 |       0.02 |         0.0 |      0.40 |          0.72 |\n\n> Typos, Grammar, Style: The writing of the paper should considerably be improved and grammatical errors should be corrected.\n\nThank you so much for your careful inspection! We will definitely correct them in modified paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper addresses an interesting research question. Such an investigation could contribute to the probing of the abilities of recent large language models and to the comparison between different language models.\n- The research question is addressed from diverse perspectives, in particular considering two different tasks, experimenting on multiple corpora and using multiple evaluation metrics.", "reasons_to_reject": "My main concern is that the test sets (179 samples and 155 samples for the scrambled sentence recovery task and the scrambled question answering task respectively) are too small to draw reliable conclusions. Similarly, the experiments on an additional dataset for the scrambled question answering task only concern 100 samples.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work probes the robustness of recent language models, including GPT-3, GPT-3.5 and GPT-4 to scrambled text (with character-level permutations). This is done by addressing two different tasks: 1) scrambled sentence recovery and 2) scrambled question answering and by constructing test sets for each of the tasks. The results show an impressive performance for GPT-4 that succeeds almost perfectly on the test sets, also in case where the scrambling is unnatural i.e. the first and last letters of the word are also scrambled.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos, Grammar, Style:\nThe writing of the paper should considerably be improved and grammatical errors should be corrected.\n\n- lines 014-015: the human-like abilities -> human-like abilities\n- line 024: the 95% of original words -> 95% of the original words\n- line 027: tend to emergent -> tend to emerge\n- line 031: has been proposed -> have been proposed\n- line 034-035: are still under mysteries -> are still a mystery\n- line 042-043: on the level of unnatural for human intuition -> on an unnatural level for human intuition\n- line 044: it challenges -> they challenge\n- line 052: As leader may ware -> Note that\n- line 059: help to analyze -> help one to analyze\n- lines 054-055: humans often can easily recognize -> humans can often easily recognize\n- line 063: test suite -> a test suite\n- line 071: change -> changes\n- lines 099-100: LLMs do not understand what should do -> LLMa do not \"understand\" what they should do\n- line 101:  sentence -> the sentence\n- line 106: scrambled version -> a scrambled version\n- line 142: keep invariant -> are kept invariant\n- line 185: be appeared -> appear\n- lines 197- 198: of zero-shot setting -> on a zero-shot setting\n- line 200: type -> types\n- line 207: number -> numbers\n- line 210: decrease -> decreases\n- lines 210-211: getting become difficult -> become difficult (or get difficult)\n- line 212: keeps -> remains\n- line 213: regardless -> regardless of\n- line 216: A comma should be added between \"setting\" and \"compared\"\n- lines 217-218: including different evaluation metric -> also with a different evaluation metric\n- lines 220-221: severity -> severities\n- line 225: increase -> increases\n- line 227: do -> does\n- line 232: plot -> plots\n- line 235: its -> of its\n- line 255: reconstruct -> reconstructs\n- line 256: answer -> answers\n- lines 259 and 296:  the comma after \"that\" should be removed\n- line 263: and provide analysis -> and an analysis\n- line 271: change - > changes\n- line 297: and easier -> and it is easy\n- line 308: And We -> We\n- line 310: of the models ->  about the models\n- line 312: superior -> is superior\n- line 314: severer -> more severe\n- lines 315-316: belong to -> is due to\n\nPresentation:\n- In the abstract, two different fonts are used for GPT-4. I think the one in line 019 should be kept since it is also used in the rest of the paper.\n- Section 2:  I think it should be split into two separate sections: Related Work and then Scrambled Bench\n- lines 148-154: the motivation for the new measure and its interpretation should be provided.\n- lies 158-165:  Is RPG a new measure? The motivation for it and its interpretation should be provided.\n- lines 201-204: it would be better to write the location of the tasks on the figure in parenthesis *after* rather than *before* the name of the \n  task. For example: on (left) zero-shot scramble sentence recovery -> on zero-shot scramble sentence recovery (left).\n- line 217: It should be specified that Tables 3 and 4 are in the Appendix. Also, I don't see the edit distance metric in these tables.\n- line 240: Results 3: it should be made clear that you focus here on the scrambled QA task.\n- lines 298-299: This should be put in a footnote, preferably in the abstract so it will appear on the first page. On the final version, you could put the link to the code instead."}}, {"review_id": "XCtbu5XTGi", "reviewer": "Reviewer_Lbf2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Lbf2:\n\nThank you very much for your review!\n\n> Clearly introduce the weakness of existing approaches in the area of unnatural scrambled text, and emphasize the strength of the proposed method, which narrows the research gap.\n\nThank you for your suggestion. We will definitely add the diagram to clarify the difference from prior works, while we can not include figures during the rebuttal. Namely, we will add a figure comparing three types of robustness against (1) unnatural word-level permutation, (2) natural character-level permutation, and (3) unnatural character-level permutation (proposal). To best of all knowledge, there are no existing studies investigating LLMs' ability to handle the character-level permutation in extreme level that drastically change the tokenization. Here, we add a simple table to demonstrate the differences, we will definitely add the comparison in more details in the modified paper.\n\n|   |                                         Category | Example                                           |    Type   | Severity |\n|---|-------------------------------------------------:|---------------------------------------------------|:---------:|:--------:|\n|   |                        Original (No Permutation) | Does the order of letters in word matter for LLMs? |    None   |   None   |\n|   |     Unnatural Word-level Permutation (e.g., [1]) | Does letters the order for LLMs or word in matter? |    Word   |   High   |\n|   |  Natural Character-level Permutation (e.g., [2]) | Does the odrer of letters in wor matter for LLMs? | Character |    Low   |\n|   | Unnatural Character-level Permutation (proposal) | oeDs eth doerr fo lettres ni rwor ttemar rof LMLs? | Character |   Extreme   |\n\n[1] Sinha, Koustuv, et al. \"UnNatural Language Inference.\" Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.\n\n[2] Wang, Jindong, et al. \"On the robustness of chatgpt: An adversarial and out-of-distribution perspective.\" arXiv preprint arXiv:2302.12095 (2023).\n\n\n>  The validation of the method's effectiveness must be more convincing. Please include a comparison with many LLMs state-of-the-art solutions (e.g., Alpaca, FLAN-T5, Llama).\n\n\nFollowing the reviewers’ comments, we conducted experiments using various open-source LLMs (including llama-2, falcon, mpt, ul2, flan-ul2, flan-t5, and byt5). The model covers diverse model architectures (decoder-only and encoder-decoder), model sizes (from 7b to 70b), and training objectives (e.g., with or without further finetuning) and tokenizers (e.g., tokenizer-free: byt5-xxl). The results are shown below:\n\n**Scrambled Sentence Recovery on RealtimeQA (edit distance)**\n|                     |  20percent | 50percent | 100percent | keepfirst | keepfirstlast |\n|---------------------|------------|----------:|-----------:|----------:|--------------:|\n|          gpt-4-0314 |       1.49 |      2.40 |       4.79 |      3.79 |          2.24 |\n|  gpt-3.5-turbo-0301 |       5.80 |     15.38 |      36.87 |     20.76 |          7.40 |\n|    text-davinci-003 |       5.38 |     18.89 |      84.39 |     22.21 |          8.36 |\n|      Llama-2-70b-hf |       7.85 |     37.30 |     169.85 |     56.25 |         18.99 |\n| Llama-2-70b-chat-hf |      11.11 |     54.49 |     196.44 |    106.31 |         43.18 |\n|      Llama-2-13b-hf |      14.18 |     59.13 |     192.67 |    103.50 |         42.67 |\n| Llama-2-13b-chat-hf |      20.75 |     73.04 |     201.50 |    119.52 |         56.03 |\n|       Llama-2-7b-hf |      20.90 |     79.83 |     194.48 |    135.50 |         68.18 |\n|  Llama-2-7b-chat-hf |      25.14 |     83.13 |     196.88 |    132.44 |         73.84 |\n|          falcon-40b |      17.27 |     69.86 |     185.26 |    119.00 |         66.03 |\n| falcon-40b-instruct |      18.79 |     74.26 |     192.86 |    130.82 |         62.39 |\n|             mpt-30b |      14.36 |     54.81 |     181.43 |     90.92 |         36.14 |\n|    mpt-30b-instruct |      19.36 |     66.05 |     170.88 |     99.38 |         50.44 |\n|                 ul2 |      18.12 |     68.79 |     153.53 |    109.47 |         71.70 |\n|            flan-ul2 |      20.51 |     62.91 |     137.15 |    105.67 |         74.35 |\n|         flan-t5-xxl |      16.60 |     59.34 |     136.41 |    105.44 |         73.97 |\n|            byt5-xxl |      79.27 |    104.28 |     148.44 |    123.97 |        102.50 |\n\n\n**Scrambled Sentence Recovery on RealtimeQA(recovery rate)**\n|                     |  20percent | 50percent | 100percent | keepfirst | keepfirstlast |\n|---------------------|------------|----------:|-----------:|----------:|--------------:|\n|          gpt-4-0314 |      94.51 |     96.46 |      96.45 |     96.37 |         97.04 |\n|  gpt-3.5-turbo-0301 |      78.57 |     77.29 |      72.63 |     80.14 |         90.22 |\n|    text-davinci-003 |      80.13 |     72.11 |      37.35 |     78.75 |         88.95 |\n|      Llama-2-70b-hf |       71.0 |     44.93 |         <0 |     46.19 |          74.9 |\n| Llama-2-70b-chat-hf |      58.97 |     19.56 |         <0 |        <0 |         42.93 |\n|      Llama-2-13b-hf |      47.62 |     12.71 |         <0 |      0.99 |         43.59 |\n| Llama-2-13b-chat-hf |      23.35 |        <0 |         <0 |        <0 |         25.93 |\n|       Llama-2-7b-hf |      22.77 |        <0 |         <0 |        <0 |          9.88 |\n|  Llama-2-7b-chat-hf |       7.11 |        <0 |         <0 |        <0 |          2.39 |\n|          falcon-40b |      36.18 |        <0 |         <0 |        <0 |         12.71 |\n| falcon-40b-instruct |      30.56 |        <0 |         <0 |        <0 |         17.53 |\n|             mpt-30b |      46.96 |     19.08 |         <0 |     13.02 |         52.23 |\n|    mpt-30b-instruct |      28.46 |      2.49 |         <0 |      4.92 |         33.33 |\n|                 ul2 |      33.03 |        <0 |         <0 |        <0 |          5.22 |\n|            flan-ul2 |      24.21 |      7.13 |         <0 |        <0 |          1.72 |\n|         flan-t5-xxl |      38.68 |      12.4 |         <0 |        <0 |          2.22 |\n|            byt5-xxl |         <0 |        <0 |         <0 |        <0 |            <0 |\n\n\n**Scrambled QA on RealtimeQA (accuracy)**\n|                     |  original | 20percent | 50percent | 100percent | substituted | keepfirst | keepfirstlast |\n|---------------------|-----------|----------:|----------:|-----------:|------------:|----------:|--------------:|\n|          gpt-4-0314 |     98.18 |     97.26 |     96.05 |      93.01 |       62.92 |     93.92 |         95.74 |\n|  gpt-3.5-turbo-0301 |     97.26 |     96.66 |     91.19 |      84.50 |       55.62 |     86.93 |         91.49 |\n|    text-davinci-003 |     94.22 |     92.10 |     87.84 |      78.72 |       55.62 |     82.98 |         91.49 |\n|      Llama-2-70b-hf |     88.75 |     86.32 |     82.98 |      65.65 |       59.88 |     78.42 |         82.67 |\n| Llama-2-70b-chat-hf |     96.05 |     92.71 |     85.71 |      64.74 |       55.62 |     75.68 |         84.19 |\n|      Llama-2-13b-hf |     66.87 |     62.92 |     57.75 |      46.81 |       46.20 |     55.02 |         55.62 |\n| Llama-2-13b-chat-hf |     85.41 |     75.99 |     66.87 |      48.02 |       45.90 |     55.32 |         61.40 |\n|       Llama-2-7b-hf |     46.81 |     44.38 |     41.64 |      40.73 |       38.30 |     40.12 |         44.07 |\n|  Llama-2-7b-chat-hf |     82.67 |     77.51 |     65.05 |      46.20 |       46.20 |     50.76 |         60.79 |\n|          falcon-40b |     49.85 |     47.42 |     41.03 |      29.79 |       28.88 |     31.00 |         34.65 |\n| falcon-40b-instruct |     57.14 |     56.23 |     48.94 |      35.56 |       33.13 |     39.21 |         41.03 |\n|             mpt-30b |     62.31 |     56.53 |     50.15 |      41.64 |       39.51 |     47.42 |         50.15 |\n|    mpt-30b-instruct |     64.13 |     63.83 |     58.97 |      48.33 |       42.86 |     56.23 |         62.92 |\n|                 ul2 |     87.23 |     84.19 |     75.68 |      48.94 |       38.60 |     62.92 |         70.52 |\n|         flan-t5-xxl |     97.26 |     94.53 |     88.15 |      50.76 |       38.30 |     65.35 |         79.64 |\n|            byt5-xxl |     72.04 |     56.84 |     46.81 |      27.36 |       24.62 |     34.35 |         42.55 |\n\n\n**Scrambled QA on RealtimeQA (relative accuracy gain)**\n|                     |  original | 20percent | 50percent | 100percent | substituted | keepfirst | keepfirstlast |\n|---------------------|-----------|----------:|----------:|-----------:|------------:|----------:|--------------:|\n|          gpt-4-0314 |       1.0 |      0.97 |      0.94 |       0.85 |         0.0 |      0.88 |          0.93 |\n|  gpt-3.5-turbo-0301 |       1.0 |      0.99 |      0.85 |       0.69 |         0.0 |      0.75 |          0.86 |\n|    text-davinci-003 |       1.0 |      0.94 |      0.83 |       0.60 |         0.0 |      0.71 |          0.93 |\n|      Llama-2-70b-hf |       1.0 |      0.92 |      0.80 |       0.20 |         0.0 |      0.64 |          0.79 |\n| Llama-2-70b-chat-hf |       1.0 |      0.92 |      0.74 |       0.23 |         0.0 |      0.50 |          0.71 |\n|      Llama-2-13b-hf |       1.0 |      0.81 |      0.56 |       0.03 |         0.0 |      0.43 |          0.46 |\n| Llama-2-13b-chat-hf |       1.0 |      0.76 |      0.53 |       0.05 |         0.0 |      0.24 |          0.39 |\n|       Llama-2-7b-hf |       1.0 |      0.71 |      0.39 |       0.29 |         0.0 |      0.21 |          0.68 |\n|  Llama-2-7b-chat-hf |       1.0 |      0.86 |      0.52 |       0.00 |         0.0 |      0.13 |          0.40 |\n|          falcon-40b |       1.0 |      0.88 |      0.58 |       0.04 |         0.0 |      0.10 |          0.28 |\n| falcon-40b-instruct |       1.0 |      0.96 |      0.66 |       0.10 |         0.0 |      0.25 |          0.33 |\n|             mpt-30b |       1.0 |      0.75 |      0.47 |       0.09 |         0.0 |      0.35 |          0.47 |\n|    mpt-30b-instruct |       1.0 |      0.99 |      0.76 |       0.26 |         0.0 |      0.63 |          0.94 |\n|                 ul2 |       1.0 |      0.94 |      0.76 |       0.21 |         0.0 |      0.50 |          0.66 |\n|         flan-t5-xxl |       1.0 |      0.95 |      0.85 |       0.21 |         0.0 |      0.46 |          0.70 |\n|            byt5-xxl |       1.0 |      0.68 |      0.47 |       0.06 |         0.0 |      0.21 |          0.38 |\n\n\n**Scrambled QA on DREAM (accuracy)**\n|                     |  original | 20percent | 50percent | 100percent | substituted | keepfirst | keepfirstlast |\n|---------------------|-----------|----------:|----------:|-----------:|------------:|----------:|--------------:|\n|          gpt-4-0314 |     96.68 |     96.29 |     94.73 |      88.48 |       45.70 |     93.95 |         96.48 |\n|  gpt-3.5-turbo-0301 |     94.92 |     90.23 |     81.05 |      44.73 |       21.29 |     73.44 |         85.74 |\n|    text-davinci-003 |     93.36 |     91.02 |     81.84 |      54.69 |       45.12 |     79.30 |         88.48 |\n|      Llama-2-70b-hf |     92.97 |     90.62 |     83.40 |      57.23 |       50.20 |     74.02 |         85.16 |\n| Llama-2-70b-chat-hf |     91.02 |     86.13 |     76.95 |      48.44 |       45.70 |     59.77 |         74.41 |\n|      Llama-2-13b-hf |     70.51 |     65.62 |     54.69 |      40.82 |       38.28 |     42.38 |         52.93 |\n| Llama-2-13b-chat-hf |     80.08 |     74.80 |     62.89 |      42.58 |       41.02 |     48.05 |         58.59 |\n|       Llama-2-7b-hf |     60.74 |     55.66 |     48.83 |      38.48 |       38.67 |     42.97 |         46.29 |\n|  Llama-2-7b-chat-hf |     75.00 |     66.21 |     57.03 |      40.04 |       38.09 |     42.97 |         52.93 |\n|          falcon-40b |     64.06 |     59.77 |     53.52 |      44.34 |       42.77 |     48.63 |         54.69 |\n| falcon-40b-instruct |     73.05 |     65.62 |     57.23 |      42.97 |       43.16 |     48.63 |         57.03 |\n|             mpt-30b |     76.37 |     73.24 |     60.74 |      47.85 |       45.31 |     50.78 |         58.98 |\n|    mpt-30b-instruct |     86.13 |     78.52 |     69.53 |      45.12 |       42.77 |     55.47 |         63.09 |\n|                 ul2 |     44.14 |     41.02 |     33.20 |      20.31 |       17.77 |     22.27 |         28.71 |\n|            flan-ul2 |     94.53 |     91.02 |     82.23 |      53.71 |       49.22 |     65.23 |         76.37 |\n|         flan-t5-xxl |     96.68 |     94.34 |     86.72 |      54.49 |       53.52 |     70.70 |         84.77 |\n\n\n**Scrambled QA on DREAM (relative accuracy gain)**\n|                     |  original | 20percent | 50percent | 100percent | substituted | keepfirst | keepfirstlast |\n|---------------------|-----------|----------:|----------:|-----------:|------------:|----------:|--------------:|\n|          gpt-4-0314 |       1.0 |      0.99 |      0.96 |       0.84 |         0.0 |      0.95 |          1.00 |\n|  gpt-3.5-turbo-0301 |       1.0 |      0.94 |      0.81 |       0.32 |         0.0 |      0.71 |          0.88 |\n|    text-davinci-003 |       1.0 |      0.95 |      0.76 |       0.20 |         0.0 |      0.71 |          0.90 |\n|      Llama-2-70b-hf |       1.0 |      0.95 |      0.78 |       0.16 |         0.0 |      0.56 |          0.82 |\n| Llama-2-70b-chat-hf |       1.0 |      0.89 |      0.69 |       0.06 |         0.0 |      0.31 |          0.63 |\n|      Llama-2-13b-hf |       1.0 |      0.85 |      0.51 |       0.08 |         0.0 |      0.13 |          0.45 |\n| Llama-2-13b-chat-hf |       1.0 |      0.86 |      0.56 |       0.04 |         0.0 |      0.18 |          0.45 |\n|       Llama-2-7b-hf |       1.0 |      0.77 |      0.46 |      -0.01 |         0.0 |      0.19 |          0.35 |\n|  Llama-2-7b-chat-hf |       1.0 |      0.76 |      0.51 |       0.05 |         0.0 |      0.13 |          0.40 |\n|          falcon-40b |       1.0 |      0.80 |      0.50 |       0.07 |         0.0 |      0.28 |          0.56 |\n| falcon-40b-instruct |       1.0 |      0.75 |      0.47 |      -0.01 |         0.0 |      0.18 |          0.46 |\n|             mpt-30b |       1.0 |      0.90 |      0.50 |       0.08 |         0.0 |      0.18 |          0.44 |\n|    mpt-30b-instruct |       1.0 |      0.82 |      0.62 |       0.05 |         0.0 |      0.29 |          0.47 |\n|                 ul2 |       1.0 |      0.88 |      0.59 |       0.10 |         0.0 |      0.17 |          0.41 |\n|            flan-ul2 |       1.0 |      0.92 |      0.73 |       0.10 |         0.0 |      0.35 |          0.60 |\n|         flan-t5-xxl |       1.0 |      0.95 |      0.77 |       0.02 |         0.0 |      0.40 |          0.72 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strength:\n\nThe paper is well-grounded in current literature, citing recent approaches and advances in the field.\n\nThe methodology seems simple but well thought-out.", "reasons_to_reject": "The reviewer has the following comments to improve the quality of the paper:\n\n I suggest the author add the method diagram to the paper.\n\n Clearly introduce the weakness of existing approaches in the area of unnatural scrambled text, and emphasize the strength of the proposed method, which narrows the research gap.\n\n The validation of the method's effectiveness must be more convincing. Please include a comparison with many LLMs state-of-the-art solutions (e.g., Alpaca, FLAN-T5, Llama).", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the authors evaluated the performance of the GPT-4 method for two tasks: scrambled sentence recovery and scrambled question answering. Also, the authors propose two new datasets based on RealtimeQA and AQUA.\n\nThe experimental results show that the GPT-4 approach employed in this work achieves better results compared to other LLMs, namely Text-davinci-003 and GPT-3.5-turbo. \nThis research work is interesting and important.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Z4whoK7NKh", "reviewer": "Reviewer_EzCc", "summary": "", "questions": "1. Can you think of any methods to analysis the model behavior using OpenAI API?\n2. On application side, can you provide any usage of this scramble recovery ability other than typo correction?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer EzCc:\n\nThank you very much for your review!\n\n> 1. Can you think of any methods to analysis the model behavior using OpenAI API?\n\nIt is difficult to analyze the text generation API of OpenAI. Because we can only get generated contents but nothing about how the model works internally. So, we add some SOTA open-source models in our experiments and do some analysis using Llama-2. \n\nIntuitively, we also thought that LLMs handle scrambled text by mapping the scrambled tokens to original ones. To investigate this, we analyze the hidden state of Llama-2 when inputting scrambled sentences. We calculate the cosine similarity of the hidden state (excluding embedding layers and last two layers, since they are strongly related to input and output, respectively) of each pair of scrambled sentences and original sentences. Then, we average the cosine similarity and normalize it from 0 (sentences with randomly substituted letters) to 1 (original sentences). The results are shown in the table below. For models of all three sizes, the similarity decreases through scrambling severity increases. And the similarity is not related to the performance on tasks. We also sort the samples by recovery rate and separate the samples into two groups based on the correctness on QA tasks and compare their similarities. However, no patterns can be found. So, we assume that LLMs handle scrambled sentences not by mapping them to original ones.\n\n|     | 10%  | 20%  | 30%  | 40%  | 50%  | 60%  | 70%  | 80%  | 90%  | 100% | kf   | kfl  |\n|-----|------|------|------|------|------|------|------|------|------|------|------|------|\n| 70b | 0.91 | 0.81 | 0.72 | 0.63 | 0.54 | 0.46 | 0.37 | 0.29 | 0.23 | 0.15 | 0.27 | 0.41 |\n| 13b | 0.92 | 0.81 |  0.7 | 0.59 | 0.49 | 0.39 |  0.3 | 0.22 | 0.15 | 0.09 | 0.18 | 0.32 |\n| 7b  | 0.97 | 0.92 | 0.86 |  0.8 | 0.73 | 0.67 |  0.6 | 0.53 | 0.46 | 0.39 | 0.49 |  0.6 |\n\n> 2. On application side, can you provide any usage of this scramble recovery ability other than typo correction?\n\nOne can require LLMs to do malicious tasks by scrambled text, then the text can pass filters but be still understandable for LLMs (some users already successfully did it and share their trials on some social media). This finding may be helpful to set more effective content filters to prevent malicious use.\n\nThis finding shows that the most powerful LLMs can understand and handle scrambled text. Investigating the reasons behind this phenomenon may be challenging, but it could shed light on how LLMs learn and aid in improving their robustness.\n\n\n> The authors should also investigate some open-source LLMs like Llama or T5. \n\nWe extend the dataset and investigate the open-source LLMs. The results are shown below:\n\n**Scrambled Sentence Recovery on RealtimeQA (edit distance)**\n|                     |  20percent | 50percent | 100percent | keepfirst | keepfirstlast |\n|---------------------|------------|----------:|-----------:|----------:|--------------:|\n|          gpt-4-0314 |       1.49 |      2.40 |       4.79 |      3.79 |          2.24 |\n|  gpt-3.5-turbo-0301 |       5.80 |     15.38 |      36.87 |     20.76 |          7.40 |\n|    text-davinci-003 |       5.38 |     18.89 |      84.39 |     22.21 |          8.36 |\n|      Llama-2-70b-hf |       7.85 |     37.30 |     169.85 |     56.25 |         18.99 |\n| Llama-2-70b-chat-hf |      11.11 |     54.49 |     196.44 |    106.31 |         43.18 |\n|      Llama-2-13b-hf |      14.18 |     59.13 |     192.67 |    103.50 |         42.67 |\n| Llama-2-13b-chat-hf |      20.75 |     73.04 |     201.50 |    119.52 |         56.03 |\n|       Llama-2-7b-hf |      20.90 |     79.83 |     194.48 |    135.50 |         68.18 |\n|  Llama-2-7b-chat-hf |      25.14 |     83.13 |     196.88 |    132.44 |         73.84 |\n|          falcon-40b |      17.27 |     69.86 |     185.26 |    119.00 |         66.03 |\n| falcon-40b-instruct |      18.79 |     74.26 |     192.86 |    130.82 |         62.39 |\n|             mpt-30b |      14.36 |     54.81 |     181.43 |     90.92 |         36.14 |\n|    mpt-30b-instruct |      19.36 |     66.05 |     170.88 |     99.38 |         50.44 |\n|                 ul2 |      18.12 |     68.79 |     153.53 |    109.47 |         71.70 |\n|            flan-ul2 |      20.51 |     62.91 |     137.15 |    105.67 |         74.35 |\n|         flan-t5-xxl |      16.60 |     59.34 |     136.41 |    105.44 |         73.97 |\n|            byt5-xxl |      79.27 |    104.28 |     148.44 |    123.97 |        102.50 |\n\n\n**Scrambled Sentence Recovery on RealtimeQA(recovery rate)**\n|                     |  20percent | 50percent | 100percent | keepfirst | keepfirstlast |\n|---------------------|------------|----------:|-----------:|----------:|--------------:|\n|          gpt-4-0314 |      94.51 |     96.46 |      96.45 |     96.37 |         97.04 |\n|  gpt-3.5-turbo-0301 |      78.57 |     77.29 |      72.63 |     80.14 |         90.22 |\n|    text-davinci-003 |      80.13 |     72.11 |      37.35 |     78.75 |         88.95 |\n|      Llama-2-70b-hf |       71.0 |     44.93 |         <0 |     46.19 |          74.9 |\n| Llama-2-70b-chat-hf |      58.97 |     19.56 |         <0 |        <0 |         42.93 |\n|      Llama-2-13b-hf |      47.62 |     12.71 |         <0 |      0.99 |         43.59 |\n| Llama-2-13b-chat-hf |      23.35 |        <0 |         <0 |        <0 |         25.93 |\n|       Llama-2-7b-hf |      22.77 |        <0 |         <0 |        <0 |          9.88 |\n|  Llama-2-7b-chat-hf |       7.11 |        <0 |         <0 |        <0 |          2.39 |\n|          falcon-40b |      36.18 |        <0 |         <0 |        <0 |         12.71 |\n| falcon-40b-instruct |      30.56 |        <0 |         <0 |        <0 |         17.53 |\n|             mpt-30b |      46.96 |     19.08 |         <0 |     13.02 |         52.23 |\n|    mpt-30b-instruct |      28.46 |      2.49 |         <0 |      4.92 |         33.33 |\n|                 ul2 |      33.03 |        <0 |         <0 |        <0 |          5.22 |\n|            flan-ul2 |      24.21 |      7.13 |         <0 |        <0 |          1.72 |\n|         flan-t5-xxl |      38.68 |      12.4 |         <0 |        <0 |          2.22 |\n|            byt5-xxl |         <0 |        <0 |         <0 |        <0 |            <0 |\n\n\n**Scrambled QA on RealtimeQA (accuracy)**\n|                     |  original | 20percent | 50percent | 100percent | substituted | keepfirst | keepfirstlast |\n|---------------------|-----------|----------:|----------:|-----------:|------------:|----------:|--------------:|\n|          gpt-4-0314 |     98.18 |     97.26 |     96.05 |      93.01 |       62.92 |     93.92 |         95.74 |\n|  gpt-3.5-turbo-0301 |     97.26 |     96.66 |     91.19 |      84.50 |       55.62 |     86.93 |         91.49 |\n|    text-davinci-003 |     94.22 |     92.10 |     87.84 |      78.72 |       55.62 |     82.98 |         91.49 |\n|      Llama-2-70b-hf |     88.75 |     86.32 |     82.98 |      65.65 |       59.88 |     78.42 |         82.67 |\n| Llama-2-70b-chat-hf |     96.05 |     92.71 |     85.71 |      64.74 |       55.62 |     75.68 |         84.19 |\n|      Llama-2-13b-hf |     66.87 |     62.92 |     57.75 |      46.81 |       46.20 |     55.02 |         55.62 |\n| Llama-2-13b-chat-hf |     85.41 |     75.99 |     66.87 |      48.02 |       45.90 |     55.32 |         61.40 |\n|       Llama-2-7b-hf |     46.81 |     44.38 |     41.64 |      40.73 |       38.30 |     40.12 |         44.07 |\n|  Llama-2-7b-chat-hf |     82.67 |     77.51 |     65.05 |      46.20 |       46.20 |     50.76 |         60.79 |\n|          falcon-40b |     49.85 |     47.42 |     41.03 |      29.79 |       28.88 |     31.00 |         34.65 |\n| falcon-40b-instruct |     57.14 |     56.23 |     48.94 |      35.56 |       33.13 |     39.21 |         41.03 |\n|             mpt-30b |     62.31 |     56.53 |     50.15 |      41.64 |       39.51 |     47.42 |         50.15 |\n|    mpt-30b-instruct |     64.13 |     63.83 |     58.97 |      48.33 |       42.86 |     56.23 |         62.92 |\n|                 ul2 |     87.23 |     84.19 |     75.68 |      48.94 |       38.60 |     62.92 |         70.52 |\n|         flan-t5-xxl |     97.26 |     94.53 |     88.15 |      50.76 |       38.30 |     65.35 |         79.64 |\n|            byt5-xxl |     72.04 |     56.84 |     46.81 |      27.36 |       24.62 |     34.35 |         42.55 |\n\n\n**Scrambled QA on RealtimeQA (relative accuracy gain)**\n|                     |  original | 20percent | 50percent | 100percent | substituted | keepfirst | keepfirstlast |\n|---------------------|-----------|----------:|----------:|-----------:|------------:|----------:|--------------:|\n|          gpt-4-0314 |       1.0 |      0.97 |      0.94 |       0.85 |         0.0 |      0.88 |          0.93 |\n|  gpt-3.5-turbo-0301 |       1.0 |      0.99 |      0.85 |       0.69 |         0.0 |      0.75 |          0.86 |\n|    text-davinci-003 |       1.0 |      0.94 |      0.83 |       0.60 |         0.0 |      0.71 |          0.93 |\n|      Llama-2-70b-hf |       1.0 |      0.92 |      0.80 |       0.20 |         0.0 |      0.64 |          0.79 |\n| Llama-2-70b-chat-hf |       1.0 |      0.92 |      0.74 |       0.23 |         0.0 |      0.50 |          0.71 |\n|      Llama-2-13b-hf |       1.0 |      0.81 |      0.56 |       0.03 |         0.0 |      0.43 |          0.46 |\n| Llama-2-13b-chat-hf |       1.0 |      0.76 |      0.53 |       0.05 |         0.0 |      0.24 |          0.39 |\n|       Llama-2-7b-hf |       1.0 |      0.71 |      0.39 |       0.29 |         0.0 |      0.21 |          0.68 |\n|  Llama-2-7b-chat-hf |       1.0 |      0.86 |      0.52 |       0.00 |         0.0 |      0.13 |          0.40 |\n|          falcon-40b |       1.0 |      0.88 |      0.58 |       0.04 |         0.0 |      0.10 |          0.28 |\n| falcon-40b-instruct |       1.0 |      0.96 |      0.66 |       0.10 |         0.0 |      0.25 |          0.33 |\n|             mpt-30b |       1.0 |      0.75 |      0.47 |       0.09 |         0.0 |      0.35 |          0.47 |\n|    mpt-30b-instruct |       1.0 |      0.99 |      0.76 |       0.26 |         0.0 |      0.63 |          0.94 |\n|                 ul2 |       1.0 |      0.94 |      0.76 |       0.21 |         0.0 |      0.50 |          0.66 |\n|         flan-t5-xxl |       1.0 |      0.95 |      0.85 |       0.21 |         0.0 |      0.46 |          0.70 |\n|            byt5-xxl |       1.0 |      0.68 |      0.47 |       0.06 |         0.0 |      0.21 |          0.38 |\n\n\n**Scrambled QA on DREAM (accuracy)**\n|                     |  original | 20percent | 50percent | 100percent | substituted | keepfirst | keepfirstlast |\n|---------------------|-----------|----------:|----------:|-----------:|------------:|----------:|--------------:|\n|          gpt-4-0314 |     96.68 |     96.29 |     94.73 |      88.48 |       45.70 |     93.95 |         96.48 |\n|  gpt-3.5-turbo-0301 |     94.92 |     90.23 |     81.05 |      44.73 |       21.29 |     73.44 |         85.74 |\n|    text-davinci-003 |     93.36 |     91.02 |     81.84 |      54.69 |       45.12 |     79.30 |         88.48 |\n|      Llama-2-70b-hf |     92.97 |     90.62 |     83.40 |      57.23 |       50.20 |     74.02 |         85.16 |\n| Llama-2-70b-chat-hf |     91.02 |     86.13 |     76.95 |      48.44 |       45.70 |     59.77 |         74.41 |\n|      Llama-2-13b-hf |     70.51 |     65.62 |     54.69 |      40.82 |       38.28 |     42.38 |         52.93 |\n| Llama-2-13b-chat-hf |     80.08 |     74.80 |     62.89 |      42.58 |       41.02 |     48.05 |         58.59 |\n|       Llama-2-7b-hf |     60.74 |     55.66 |     48.83 |      38.48 |       38.67 |     42.97 |         46.29 |\n|  Llama-2-7b-chat-hf |     75.00 |     66.21 |     57.03 |      40.04 |       38.09 |     42.97 |         52.93 |\n|          falcon-40b |     64.06 |     59.77 |     53.52 |      44.34 |       42.77 |     48.63 |         54.69 |\n| falcon-40b-instruct |     73.05 |     65.62 |     57.23 |      42.97 |       43.16 |     48.63 |         57.03 |\n|             mpt-30b |     76.37 |     73.24 |     60.74 |      47.85 |       45.31 |     50.78 |         58.98 |\n|    mpt-30b-instruct |     86.13 |     78.52 |     69.53 |      45.12 |       42.77 |     55.47 |         63.09 |\n|                 ul2 |     44.14 |     41.02 |     33.20 |      20.31 |       17.77 |     22.27 |         28.71 |\n|            flan-ul2 |     94.53 |     91.02 |     82.23 |      53.71 |       49.22 |     65.23 |         76.37 |\n|         flan-t5-xxl |     96.68 |     94.34 |     86.72 |      54.49 |       53.52 |     70.70 |         84.77 |\n\n\n**Scrambled QA on DREAM (relative accuracy gain)**\n|                     |  original | 20percent | 50percent | 100percent | substituted | keepfirst | keepfirstlast |\n|---------------------|-----------|----------:|----------:|-----------:|------------:|----------:|--------------:|\n|          gpt-4-0314 |       1.0 |      0.99 |      0.96 |       0.84 |         0.0 |      0.95 |          1.00 |\n|  gpt-3.5-turbo-0301 |       1.0 |      0.94 |      0.81 |       0.32 |         0.0 |      0.71 |          0.88 |\n|    text-davinci-003 |       1.0 |      0.95 |      0.76 |       0.20 |         0.0 |      0.71 |          0.90 |\n|      Llama-2-70b-hf |       1.0 |      0.95 |      0.78 |       0.16 |         0.0 |      0.56 |          0.82 |\n| Llama-2-70b-chat-hf |       1.0 |      0.89 |      0.69 |       0.06 |         0.0 |      0.31 |          0.63 |\n|      Llama-2-13b-hf |       1.0 |      0.85 |      0.51 |       0.08 |         0.0 |      0.13 |          0.45 |\n| Llama-2-13b-chat-hf |       1.0 |      0.86 |      0.56 |       0.04 |         0.0 |      0.18 |          0.45 |\n|       Llama-2-7b-hf |       1.0 |      0.77 |      0.46 |      -0.01 |         0.0 |      0.19 |          0.35 |\n|  Llama-2-7b-chat-hf |       1.0 |      0.76 |      0.51 |       0.05 |         0.0 |      0.13 |          0.40 |\n|          falcon-40b |       1.0 |      0.80 |      0.50 |       0.07 |         0.0 |      0.28 |          0.56 |\n| falcon-40b-instruct |       1.0 |      0.75 |      0.47 |      -0.01 |         0.0 |      0.18 |          0.46 |\n|             mpt-30b |       1.0 |      0.90 |      0.50 |       0.08 |         0.0 |      0.18 |          0.44 |\n|    mpt-30b-instruct |       1.0 |      0.82 |      0.62 |       0.05 |         0.0 |      0.29 |          0.47 |\n|                 ul2 |       1.0 |      0.88 |      0.59 |       0.10 |         0.0 |      0.17 |          0.41 |\n|            flan-ul2 |       1.0 |      0.92 |      0.73 |       0.10 |         0.0 |      0.35 |          0.60 |\n|         flan-t5-xxl |       1.0 |      0.95 |      0.77 |       0.02 |         0.0 |      0.40 |          0.72 |\n\n\n>  \"leader\" in row 52 should be \"reader\"\n\nThank you so much for your careful inspection! We will definitely correct them in modified paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is an interesting finding on LLMs' capability.", "reasons_to_reject": "This paper lacks of further analysis and explanation on the LLMs' behavior on scramble texts. E.g. the authors show the differences on tokenization of the regular sentence and scrambled one in Fig 1, but they did not analysis how LLMs encode these different tokens and how LLMs map the scrambled tokens to original ones. \nAuthors may argue that OpenAI API does not fully support this kind of analysis, which lead to my second point: the authors should also investigate some open-source LLMs like Llama or T5. Although they are not as powerful as OpenAI GPT models, it is easier to conduct more analysis on these models.", "questions_to_authors": "1. Can you think of any methods to analysis the model behavior using OpenAI API?\n2. On application side, can you provide any usage of this scramble recovery ability other than typo correction?", "paper_topic_and_main_contributions": "This paper investigates how well LLMs handle the scrambled text. The authors use two evaluation methods: 1. scrambled sentence recovery and 2. scrambled question answering on three OpenAI products: GPT-3.5, ChatGPT and GPT4. They create a dataset which applied different scramble strategies and evaluate the models on this dataset. They show that LLMs can handle the scrambled text.", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "\"leader\" in row 52 should be \"reader\""}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores the ability of large language models (LLMs) to handle character-level scrambled text. The authors created a benchmark and conducted experiments to test this. They found that GPT-4 excels at understanding scrambled text, even when the letters of words are scrambled. Other LLMs also perform well, but GPT-4 stands out for its remarkable performance in processing inputs with unnatural errors. The authors evaluated the performance of the GPT-4 method for two tasks: scrambled sentence recovery and scrambled question answering. They also propose two new datasets.  This work is interesting and well conducted.

Pros:
This paper addresses an interesting research question and proposes a benchmark for the corresponding task. 
The paper is well-grounded in current literature, citing recent approaches and advances in the field. 
The methodology seems simple but well thought-out. The  investigation could contribute to the probing of the abilities of LLMs. 
The authors extended  the evaluation models and datasets during the rebuttal.
Most of the cons covered in the rebuttal.
Typos, proofreading is highly recommended.
Decision: Accept-Main