StringLLM: Understanding the String Processing Capability of Large Language Models
Authors: Xilong Wang
Hao Fu
Jindong Wang
Neil Zhenqiang Gong
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
String Processing
Benchmarks
Datasets
Primary area: datasets and benchmarks
TL;DR: The first comprehensive study to understand the string processing capability of Large Language Models.

Abstract: String processing, which mainly involves the analysis and manipulation of strings, is a fundamental component of modern computing. Despite the significant advancements of large language models (LLMs) in various natural language processing (NLP) tasks, their capability in string processing remains underexplored and underdeveloped. To bridge this gap, we present a comprehensive study of LLMs' string processing capability. In particular, we first propose StringLLM, a method to construct datasets for benchmarking string processing capability of LLMs. We use StringLLM to build a series of datasets, referred to as StringBench. It encompasses a wide range of string processing tasks, allowing us to systematically evaluate LLMs' performance in this area. Our evaluations indicate that LLMs struggle with accurately processing strings compared to humans. To uncover the underlying reasons for this limitation, we conduct an in-depth analysis and subsequently propose an effective approach that significantly enhances LLMs' string processing capability via fine-tuning. This work provides a foundation for future research to understand LLMs' string processing capability. Our code and data are available at https://github.com/wxl-lxw/StringLLM.

Published as a conference paper at ICLR 2025
STRINGLLM: UNDERSTANDING THE STRING PRO- CESSING CAPABILITY OF LARGE LANGUAGE MODELS
Xilong Wang" 12, Hao Jindong Wang', Neil Zhenqiang Gong' Duke University, "Li Auto, "William Mary [xilong. wang, neil. feil.sd gong (oong))duue.ccom duuee. edu, fuhao8@lixiang,co jwang80@wm. @du
ABSTRACT
String processing, which mainly involves the analysis and manipulation of strings, is a fundamental component of modern computing. Despite the significant ad- vancements of large language models (LLMs) in various natural language process- ing (NLP) tasks, their capability in string processing remains underexplored and underdeveloped. To bridge this gap, we present comprehensive study of LLMMs' string processing capability. In particular, we first propose StringI.LMM, method construct datasets for benchmarking string processing capability LLMs. We use StringLLM to build series of datasets, referred to as StringBench. It en- compasses wide range of string processing tasks, allowing us to systematically evaluate LLMs' performance in this area. Our evaluations indicate that LLMs struggle with accurately processing strings compared to humans. To uncover the underlying reasons for this limitation, we conduct an in-depth analysis and sub- sequently propose an effective approach that significantly enhances LLMs' string processing capability via tuning This work provides a foundation for future research to understand LLMs' string processing capability. Our code and data are available https: //gittttbb/t com/ wxl- 1xw/ St ietrrngg LigLIMM
INTRODUCTION
String processing one of the most essential tasks in modern computing. is mainly involved with string analysis and manipulation such as accessing characters at specific index (string indexing) or locating substring within a string (substring searching). For instance, search engines match user queries to relevant documents using substring searching, with Google processing over 3.5 billion searches per day. String processing is also crucial for searching through large databases, such as Amazon's 's over 400 million product listings. It is also vital for cleaning and normalizing messy text data in machine learning. Additionally, string processing plays a critical role in I.LM reasoning. For instance, the chain-of-thouge examples in the OpenAl ol documentation (OpenAI, 2024b), such as cipher and crossword problems, frequently involve string processing. Programming languages like Python offer powerful built-in functions for string manipulation, making easier for humans to solve complex string-related problems.
With the advance of LLMs natural language processing (NLP) tasks (OpenAI, 2024b;a; 2023a), it is intuitive to assume that LLMs should also achieve great performance on string-related problems since strings are texts. However, the reality that LLMs often struggle with these seemingly simple challenges. A popular example (Goodside, 2024) shows that GPT-4o (OpenAI, 2024a) frequently fails in counting the number of "r"s in the word "strawberry". Additionally, the use of LLMS for string processing remains largely unstudied. Existing works (Shin & Kancko, 2024; Tan et al., 2024; Yehudai et al., 2024; Zhou et al., 2023) are mostly limited to case studies with only a few string processing tasks, lacking comprehensive evaluation of LLMs' capabilities and limitations in this domain. Consequently, it remains unclear how well LLMs can handle such tasks, why they fail, and what might improve their performance.
Work done when Xilong Wang an intern Li Auto mentored by Hao Fu.

---

Published as a conference paper at ICLR 2025
StringLLM: In this paper, we conduct the first comprehensive study toward understanding the string processing capability of LLMs. One core challenge is how to create large representative datasets with diverse set of string processing tasks, different types of strings, and guaranteed ground truth answers. To address the challenge, we propose StringI.IM, the first method to construct datasets for benchmarking string processing capability of L.MMs. StringI.LMN LMM begins by manually collecting set of fundamental string processing tasks called atomic tasks. Then, StringLLM combines these atomic tasks to create more intricate ones, referred to as composite tasks. Given a task, St generates question templates for it, which are then used to derive question inputs. Finally. StringLLM obtains the ground truth answer for a question input by executing the Python code of the corresponding task, generating question-answe ansser pairs for string processing.
We use StringLLM to construct a series of datasets, referred to as String- Bench. These datasets include 1,511 string processing tasks spanning a broad range of applications, and three common types of strings: hash strings, multilingual natural language strings, and random strings. StringBench allows us to thoroughly assess the ability of LLMs to process strings across various scenarios.
Crafting new datasets:
Benchmarking string processing capabilities of LLMs: Building on StringBench, we present the first systematic evaluation of LLMs on string processing tasks, using three prompting strategies: raw instructions, Chain of Thought (CoT) (Wei et al., 2022), and Program of Thought (PoT) (Chen et al., 2022). Our comprehensive experimental results demonstrate that: 1) LLMs struggle with string pro- cessing compared to human capability. In particular, they achieve a maximum of 48. 89.8% accuracy using raw instructions; 2) LLMs' performance varies across datasets, revealing significant dispar- ities in their ability to process different types of strings. Specifically, random strings are the most challenging, with accuracy peaking at 43.9 43.94% using raw instructions; and 3) Prompt engineering significantly improves performance. Some LLMs show over a 20% improvement when using PoT compared to raw instructions, offering valuable insights for designing better solutions.
Understanding why LLMs struggle with string processing: We conduct the first in-depth analysis to investigate why LLMs struggle with string processing. Our analysis reveals that: 1) Tokenization fails to split strings into individual characters, resulting in lack of character-level understanding in LLMs; and 2) Token embedding lacks character information such as token length informa- tion. further highlighting LLMs' limited character-level comprehension of strings. Our analysis and Yehudai et al. (2024) show that Transformers have limited ability in solving string processing tasks. To address this, we propose an effective solution to enhance L.LMs' performance in string process- ing, without altering the architecture of Transformers. Utilizing our well-constructed StringBench, we conduct supervised fine-tuning on three different open-source LLMs. Our fine-tuned models improve average test accuracy of our datasets by at least 38.80% %, compared to the pest-performing prompt engineering technique, PoT. We then evaluate the foundational capabilities of our fine-tuned models on three general-purpose benchmarks. The results show that the string processing capa- bilities of our fine-tuned models are significantly enhanced without substantially degrading their foundational capabilities. Specifically, the three fine-tuned L.MSs sacrifice most 1.35% aver- age across the three pencral- benchmarks.
2 RELATED WORK
LLMs' string processing capability is underexplored: LLMs have advanced significantly in re- cent years, demonstrating impressive capabilities across diverse NLP tasks, such as reasoning (Wei et al., 2022: Chen et al., 2022), coding (Roziere al., 2023; Zhu et al., 2024), and instruction follow- ing (Ouyang et al., 2022). However, their string processing capabilities remain understudied. Shin & Kaneko (2024) explored L.MS' ability to handle character-level tasks as opposed to token-level tasks. Their study revealed that LLMS struggle with simple character-level operations (e.g.. char- acter insertion, deletion, reordering), which humans can perform with ease. In contrast, LLMs' performance on token-level tasks (e.g., sentence retrieval, word insertion) is relatively stronger, highlighting a discrepancy between token and character-level understanding. Zhou et al. (2023) proposed a benchmark to evaluate the instruction- followiinn capabilities of LLMs, including some string processing tasks. However, the tasks are few and lack comprehensiveness. limiting their use to case studies. Tan et al. (2024) further noted that LLMS often struggle with basic counting tasks,
2

---

Published as a conference paper at ICLR 2025
Manually
Siep I: Censrrating
Step 2: Cunsiracing
Aromic Tacks
Composte laoks
Code
mafs]
Varialiis
LLM
Gonzzring Quadiuneeeeee
Step S: Generating Question Templates
Ducomentatioss
LIM
pow = = emnooribes)
Teesplate
wils
Fillr
Veribles
Trmplano
Doos revens of (M] end with [x]?
6 2284638.
"836684""
4B44 TEWA37
Samplng
2252252291
Diffinant
Figure 1: Overview of how StringLLM builds the benchmark datasets. Code represents Python code expression for each string processing task, Template denotes the generated question template for each task, Variables are the placeholders within both Code and Template, and Answer refers to the groundtruth for each sample.
such as generating a paragraph with a specific word count and then correctly identifying that num- ber. Correspondingly, Ychudai et al. (2024) discussed strategies to address this counting problem in Transformers. However, these works are limited to case studies focused on single or few string processing tasks, lacking a comprehensive evaluation. Furthermore, they do provide in-depth analysis of why LLMs struggle with accurately processing strings, nor do they offer solid experi- mental evidence support their conclusions. Consequently, they fail to propose effective solutions to enhance string processing capability of LLMs.
3 CONSTRUCTING DATASETS VIA STRINGLLM
3.1 OVERVIEW
An overview of StringLLM is shown in Figure/ We begin by creating tasks. Specifically, we first create tasks that are fundamental and cannot be broken down further, referred to as atomic tasks. These are either manually designed by us or sourced from public programming language documentation. Once we have these atomic tasks, we combine multiple of them into more complex tasks, referred to as composite tasks. Using both atomic and composite tasks, we generate natural language question-answer pairs for LLMs, resulting in a series of datasets called StringBench. It includes three distinct datasets, each designed to query LLMs in different ways:
Multilingual: Queries LLMs to process multilingual natural language strings.
Hash: Queries LLMs to process strings encoded by different hash functions.
Random Strings: Queries LLMs to process random strings composed of printable characters.
The three datasets cover diverse usage of LLMs, including natural language tasks and code-related tasks. The Multilingual dataset focuses on natural language tasks with special attention to multi- ple languages. Hash strings represent typical type of strings in code-related tasks. To cover wider range of characters, we also consider random strings of all printable characters. Manually constructing datasets of this nature extremely time-consuming and impractical. We leverage GPT- 4o (OpenAI, 2024a) to assist in the data construction process. We avoid direct use of GPT-4o for string processing, as we will show in Section that it struggles with string-related tasks. Instead, we make use of its coding abilities, including code generation, understanding, and summarization. We take code as a proxy to produce accurate datasets. Previous study (OpenAI, 2024a) has shown that GPT-4o performs exceptionally well in these areas, achieving score of 90.2% the Hu- manEval benchmark (Chen et al., 2021). This highlights its remarkable capabilities in code-related applications, enabling us it for data construction.
3

---

Published as a conference paper at ICLR 2025
3.2 CONSTRUCTING STRING PROCESSING TASKS
Atomic string processing tasks: We begin by thoroughly examining the built-in function documen- tation of Python to identify atomic string processing tasks. Specifically, we focus on distinguishing unique functions while filtering out those that are redundant or exhibit overlapping functionalities. For example, the splitlines () function, which is designed for splitting strings line bound- aries, is obviously a specialized case of the more general split tt () function. Therefore, we inte- grate splitlinee
() into the broader split it () task and exclude it from separate consideration. Following this, set of atomic string processing tasks is obtained. However, this initial set proves to insufficient for our comprehensive needs. Many common tasks are not encapsulated within the Python built-in functions. To address this gap, manually develop additional set of atomic tasks. These include fundamental tasks such as indexing, slicing, and concatenation. They are fre- quently used but not explicitly provided as built-in functions. Once we have determined the complete set of atomic tasks, we proceed to formally express each task in Python code. We man- ually create Python scripts that perform these tasks using placeholder variables. These scripts then serve as templates for constructing composite tasks (see Table 12 in Appendix for examples).
By manually reviewing the built-in function documentation and crafting an additional set of atomic tasks, we ensure that these atomic tasks are representative and can compose many diverse composite tasks. Moreover, by filtering out tasks that are redundant or exhibit overlapping functionalities, we avoid the selected atomic tasks having conflicted meanings that could form bad composite tasks.
Composite string processing tasks: Building on atomic tasks, we iteratively generate more com- plex string processing tasks. each iteration, we randomly shuffle and then input all atomic tasks into GPT-4o, combining them into composite tasks. Specifically, we prompt GPT-4o to generate Python code for composite tasks based on the code atomic tasks, This approach fully leverages GPT-4o's Program of Thought (PoT) capability, instead of using it for direct string processing, As a result, we generate a total 1,462 composite tasks, covering a wide range of complicated string processing scenarios commonly used in practice.
3.3 GENERATING QUESTION-ANSWER PAIRS FOR EACH TASK
Constructing question templates: After obtaining the atomic and composite tasks, we once again use GPT-4o to generate question templates for these tasks. As illustrated in Figure 1, question template describes the task with placeholders. Given the Python code, GPT-4o is prompted to create a template that asks how the task described by the code would be accomplished. This ensures that the question templates are aligned with the tasks defined by the Python code. Thus, such templates can effectively guide LLMs to perform the required tasks. This procedure leverages GPT-40's capability of code summarization,
Table 1: Dataset statisties.
Dataset
Atomic Tasks
Composite Tasks
Question-answer Pairs
Multilingual
49
1.462
22.161
without requiring it to process strings directly. As with previous steps, this approach skillfully exploits GPT-4's ' strengths while minimizing its limitations. To promote diversity, we generate three paraphrased question templates for each task. Specifically, to ensure the quality of the templates, we first instruct GPT-4o to generate 10 templates. Then, we have GPT-4o itself rank these generated templates and select the top three.
Hash
49
1,462
18,441
Random String
49
1,462
21,159
Generating question-answer pairs based on the template: Once the question templates are gen- crated, we replace the placeholders in the templates with specific strings to be processed. This procedure transforms the templates into question inputs. To promote the diversity of our dataset, we create three types of strings to each question template. This results in StringBench, containing three distinct datasets:
methos
4

---

Published as a conference paper at ICLR 2025
Multilingual: We randomly sample strings from the Flores-200 dataset (Costa-jussä al., 2022), which is a machine translation dataset including data in 200 languages.
We first randomly sample strings from the Flores-200 dataset. For each string, we ran- domly select hash function from a set of 10 different functions to encode it. The 10 used cryptographic hash functions are listed in Table 11 in Appendix.
Hash:
Random String: We randomly sample strings using all printable characters in Python, including letters, numbers, punctuation, escape characters, and other special symbols.
Finally, we execute the Python code to generate the ground truth answers for all datasets. This en- sures a reliable and accurate reference for both training and evaluation purposes. The guaranteed ground truth data is significant strength of our approach. Unlike other datasets that may suffer from inconsistencies or ambiguities in labeling, our ground truth provides a definitive reference, improv- ing the reliability of evaluations. This rigorous standard allows with high precision and offers a robust foundation for training LLMs to achieve better performance. By leveraging this guaranteed ground truth, we ensure that our datasets serve as a valuable bench- mark for both research and practical applications. Statistics of our datasets are shown in Table 1, and randomly selected question-anse answer pairs from each dataset are presented in Table 2.
measure LLMs' performance
Table 2: Randomly selected question-answer pairs from our datasets.
Dataset
Question
Answer
Retrieve the 29 characters from string "Felici aila, rakamboita dutu rechi kamu chechi pa Sirero y Furri cane, yakanata kusvika kuderera kwatropical ichizopra neChipiri".
"Felicia, rakamboita duzu rech"
Multilingual
Get the index of the first instance of "de" in string "c11cc8a5954766ddettte0 dce2acabal
Hash
Take every 4th character from the start of string "N/ Sgee i - AdveN/am' JGFN /a-n+ 885: 3BA?d JnaF44, (WG-9:1Vw] " up to the 50th character.
"Ne8d/U/ 41 ?a&G"
Random String
3.4 POST-PROCESSING
We filter out duplicate and low-quality data after the above steps. Duplicate code and question templates removed in Steps 2 and 3 in Figure 1. All Python code is validated through execution, and any code that results in execution errors is also removed.
4 BENCHMARKING LLMS' STRING PROCESSING CAPABILITIES
4.1 EXPERIMENTAL SETUP
LLMs: We use the following LLMs for our evaluation: GPT-4-Turbo (OpenAI, 2023b), GPT-4o (OpenAI, 2024a), GPT-3.5 (OpenAl, 2022), DeepSeek-Chat (DeepSeek-Al, 2024), Gemma-2-9h (Google, 2024), L1ama-3,1-8B (Meta, 2024), Mistral-7B-vO.3 (Mistral-AL, 2024c), Mathstral-7B- v0.1 (Mistral-AI, 2024b), Codestral-22BB 220-v0..2 (Mistral-AI, 2024a) and DeepSeck- -odder (Zhu et al., 2024). Table 10 in Appendix shows more details of the models.
Training-test split of our datasets: For the test sets, we randomly split 20% of the data from each of the three datasets-Muulillnguuu Multilingual, Hash, and Random String. We ensure that the test sets cover the full range of 1,511 string processing tasks in our datasets. This approach guarantees that the diversity and complexity of these tasks are well-represented in the test sets, allowing comprehensive evaluation of the model's performance. The remaining 80% of our datasets used as the training sets for our experiments fine-tuning LLMs in Section 6.
Prompt engineering: In our experiment, we apply three prompt engineering techniques to evaluate the performance of LLMs. 1) Raw instructions: The questions are input directly to the LLMs without any additional guidance, allowing LLMs to choose their own methods for processing the strings. 2) Chain of Thought (CoT) (Wei et al., 2022): Prompts LLMs to generate step-by-step solutions for string processing tasks. By breaking down problems into smaller steps, this technique helps L.MMs handle complex reasoning tasks more systematically. 3) Program of Thought (PoT)
5

---

Published as a conference paper at ICLR 2025
Table 3: Accuracy for string processing capability of humans and different LLMs.
LLM
Method
Multilingual
Hash
Random String
AVG
Raw Inst.
43.09
48.89
43.94
45.31
GPT-4o
CoT
50.05
52.01
45.63
49.23
PoT
69.19
68.00
49.06
62.08
Raw Inst.
35.49
41.61
33.87
36.99
GPT-4-Turbo
CoT
43,98
48.10
43.09
45.06
PoT
66.50
71.01
48.74
62.08
Raw Inst.
8.36
15.93
13.60
12.63
GPT-3.5
CoTT
30.81
29.94
24.77
28.51
PoT
42.63
42.62
25.26
36.84
Raw Inst.
17.79
21.34
13.97
17.70
DeepSeek-Codee
CoT
22.87
29.72
22.87
25.15
PoT
54.85
57.46
29.42
47.24
Raw Inst.
4.26
3.50
1.90
3.22
DeepSeek-
CoT
7.89
9.99
7.30
8.39
PoT
7.12
7.76
2.39
5,76
Raw Inst.
21.49
21.01
14.72
19.07
Gemma-2-9b
CoT
23.81
24.67
18.84
22.44
PoT
55.34
52.71
14.49
40.85
Raw Inst.
4.54
3.83
2.67
3.68
Mistral-7B-v0.3
CoT
10.56
8.77
6.02
8.45
PoT
32.54
31.57
14.07
26.06
Raw Inst.
4.85
5.81
2.08
4.25
Mathstral-7B-v0.
CoT
13.25
15.72
10.07
13.01
PoT
21.40
25.95
12.31
19.89
Raw Inst.
19.03
15.99
12.52
15.85
CoT
16.08
24.46
10.56
17.03
PoT
23.92
14.39
15.75
18.02
Raw Inst.
11.87
16.61
10,30
12.93
Llama-3.1 3-BB
CoTT
19.85
22.04
17.13
19.67
PoT
39.35
42.62
21.79
34.59
Manual Python
42.27
95.74
93.55
77.19
Human
98.12
97.99
98.61
98.24
(Chen al., 2022): Prompts LLMs to generate responses mainly in programming language. This technique structures the reasoning process LLMs in a programmatic manner. Thus, it enables LLMs perform intricate string processing tasks with greater precision.
Evaluation metries: evaluate the performance of LLMs on string processing tasks, we use Accurucy (Acc). It measures the correctness of LLMs' responses input questions. As mentioned Section 3, we execute the Python code of each task to obtain the ground truth answer. Hence, due to the deterministic nature of these answers, correctness is determined by exact matches between the model's ' final output and the ground truth answer.
We conducted a human study to evaluate the string processing capabilities of LLMs in comparison with humans. We human annotators to perform string processing tasks in the test set, either manually or by writing Python code.
Human study:
4.2 EXPERIMENTAL RESULTS
LLMs struggle to process strings compared humans: The experimental results in Table reveal that all LLMs struggle with string processing tasks across all datasets and prompt engineering techniques. For example, when using raw instruction prompts, even the best-performing LLMs, GPT-4o and GPT-4-Turbo, achieve only 36.99% and 45.31% Acc average, respectively. Other LLMs perform significantly worse. For instance, GPT-3.5 reaches an average Ace of just 12.63%, which is 32.68% lower than its updated version GPT-40. L.MMs with much fewer parameters exhibit

---

Published as a conference paper at ICLR 2025
even more significant performance drops, with average Acc below 20%. Specifically, Mistral-7B- v0.3 and DeepSeek- -hhat exhibit particularly poor performance. They achieve average Ace of only 3.68% and 3.22%, respectively. In contrast, humans demonstrate near-perfect performance across all datasets. They do struggle to process strings manually, as they cannot identify and segment Multilingual characters. For instance, in the case of the Arabic language, non-native speakers may find it challenging to process such strings due to the difficulty of identifying individual characters. However, when using Python code to process strings, humans achieve an average Acc of 98.24 18.24%. This highlights the limitations of LLMs compared to human capability.
Random strings are harder to process: As shown in Table 3, LLMs exhibit significant perfor- mance variation across our datasets. In general, LLMs perform better on the Hash dataset compared to the Random String and Multilingual datasets. For instance, with raw instruction prompts, GPT-4o achieves 48.8 8..9% Acc on the Hash dataset. This is 5.8% higher than its performance on the Mul- tilingual dataset and 4. 4.95% higher than on the Random String dataset. Overall, 6 out of 10 LLMs perform best on the Hash dataset, indicating that LLMs excel in processing hash strings. Addition- ally, 8 out of 10 LLMs show their worst performance on the Random String dataset when using raw instruction prompts. This finding suggests that LLMs lack expertise in processing random strings. One possible reason is that LLMs are not trained on data containing random strings, making it an out-of-distribution problem.
Prompt engineering improves performance: Table 3 shows that LLMs perform largely better with PoT and CoT than with raw instructions, where all 10 LLMs exhibit their lowest average Acc. In contrast, when prompted with PoT, LLMs consistently deliver the best results. Specifically, 9 out of 10 LLMs show their highest performance. For example, the DeepSeek- Coder achieves an average Acc of 47.24 24% across the three datasets when using POT. It is 22.09% and 29.54% higher than with CoT and raw instructions, respectively. This is likely because, compared to CoT, PoT structures the reasoning process more hierarchically and incorporates programming to solve problems. As a result, it is better suited for precise character- manipulation required in string processing. Examples of GPT-40 solving string processing task using different prompt engineering techniques shown in Figure 2 - in Appendix. These examples further highlight PoT's advantage over CoT and raw instructions in string processing.
UNDERSTANDING WHY LLMS STRUGGLE WITH STRING PROCESSING
5.1 TOKENIZATION CANNOT SPLIT STRINGS INTO CHARACTERS
Tokenization is the process of breaking down texts into smaller units called tokens, which can be words, subwords, or characters. This process is fundamental to how LLMs process and understand text. However, L.Ms lose character-level details of input, when it is tokenized. When an input text is tokenized into subwords words, LLMs may lose the granular character-level details of the input. This can lead to issues for string processing tasks, which require precise character-level manipulation or analysis. For instance, checking if a string is a palindrome (reads the same backward as forward) can be problematic due to tokenization. Taking a simple example, the phrase "A man plan a canal Panama" " should be recognized as a palindrome when ignoring spaces and case differences. However, the tokenizer might split it into [ ["A", "man" "a", "p an", "a", "canal 1 ", "Panama J. As result, L.MS may fail to perform the check correctly, since they may treat each word as a separate token instead of considering the entire character sequence.
Table 4 shows ratios of actual length to the number of tokens for sampled strings in our datasets, when tokenized by different LLMs. From Table 4, we can draw the following conclusion:
For the Multilingual dataset, as shown in Table 4, all three LLMs show the highest ratio. This is because in multilingual natural language strings, words and subwords remain indivisible during tokenization. Consequently, tokenization does not break down the strings into finer fragments. This could contribute to lower Ace in such scenarios.
For the Hash dataset, tokenization breaks the string into finer fragments. As shown in Table 4, two out of three LLMs display the lowest ratio. Furthermore, hash strings typically consist of alphanumeric characters without spaces or other natural language patterns. They also have a fixed length, making them relatively structured. Additionally, they generally do not contain punetuation
7

---

Published as a conference paper at ICLR 2025
escape characters, further simplifying their structure. In this context, hash strings easier for LLMs to process and amalyze, leading to higher Acc.
For the Random String dataset, tokenization segments the strings into smaller units as well. How- ever, these strings often contain escape characters, punctuation, and other special characters. These elements add complexity to the structure of random strings, making it challenging for LLMs to analyze and understand them accurately.
shown in Table 3, LLMs perform best on the Hash dataset due the simpler, more structured nature of the strings. For the Multilingual dataset, performance is lower because tokenization preserves the integrity of words and sub- words. Consequently, it could be more dif- ficult for LLMs to establish character-level understanding of such strings, For the Ran- dom String dataset, the added complexity from special characters further challenges LLMs, resulting in the lowest Acc.
Correspondingly,
Table 4: Ratios of actual length the number of kens for sampled strings in our datasets, when tok- enized by different LLMs.
L.MM
Multilingual
Hash
Random String
Gemma-22 2-9h
2.53
1.12
1.30
Mistral-7B-vv
1.73
1.06
1.15
Llama-3.1-88
2.13
1.67
1.32
5.2 TOKENIZATION MAKES LLMs LACK CHARACTER- UNDERSTANDING
As stated in the previous section, tokenization typically splits strings into word-level or subword- level tokens instead of individual characters. However, existing studies do not provide sufficient evaluation to confirm that these tokens lack character-level information. In this section, we aim further demonstrate that current tokenization, which cannot perform such fine-grained splitting, makes LLMs indeed lack character-level information. Table 5 shows the string processing capa- bility of different LLMs, where white spaces are inserted between all characters of strings. This insertion of white spaces aims segment every character in strings, avoiding the use of word-level or subword-level tokens, thereby introducing potential character-level information to LLMs.
Table 5: Aceuracy for string processing capability of different LLMs, where white spaces are serted between all characters of strings.
LLM
Method
Multilingual
Hash
Random String
AVG
Raw Inst.
20.48
21.23
14.31
18.67
Gemma-2-9h
CoT
24.22
25.16
20.07
23.15
PoT
55.69
52.73
18.98
42.47
Raw Inst.
4.97
3,99
2.79
3.92
Mistral-7B-v0.3
CoT
12.45
10.53
7.34
10.11
PoT
32.28
32.67
15.32
26.76
Raw Inst.
12.83
16.56
9.76
13.05
Llamm-3.1180 1--8B
CoT 20.64
23.21
19.20
21.02
PoT
42.01
43.30
24.74
36.68
Compared Table the performance of LLMS is improved, indicating that the word-level or subword-level tokens make LLMs lack character-level understanding. In contrast, augmenting the tokenizer with finer-grained character-level segmentation could be promising direction for improv- ing LLM performance.
5.3 LLMs' UNDERSTANDING OF STRINGS IS LIMITED
Our experimental results have demonstrated that tokenization cannot split strings into individual characters, and word-level subword-level tokens do not include sufficient character-level infor- mation. Considering the architecture of Transformers, tokenization the starting point, and its lim- itations can lead to further errors. Consequently, the attention mechanism and entire Transformer architecture of LLMs cannot effectively analyze character-level information, leading to a lack of fundamental understanding of strings.
8

---

Published as a conference paper at ICLR 2025
Table 6: Acc for string processing capability of different LLMs before and after fine-tuning, when prompted with PoT.
LLM
Multilingual
Hash
Random String
Avg. Change
Before
55.34 82.49
52.71
14.49
Gemma-2-99
+ 38.80
After
87.98
68.46
Before
32.54
31.57
14.07
Mistral-7B-v0.3
+ 56.51
After
83.24
90.28
74.18
Before After
39.35 79.83
42.62 85.64
21.79
IJama-3. 1-8B
+ 422.87
66.91
Table Acc/Acc-Norm for foundational capabilities of different I.L.Ss before and after fine-tuning.
LLM
MMLU
Hellaswag
ARC
Avg. Change
Before
71.88
80.08
64.76
Gemma-2-9b
-0.87
After
70.92
80.39
62.80
Before
59.74
82.90
58.62
Mistral-7B-v0.3
-..5
After
57.56
82.14
57.51
Before
67. .4
79.33
55.03
Llama-3. 3.BB 1-88B
+ 0.51
After
66.24
79.48
58.11
IMPROVING STRING PROCESSING CAPABILITY VIA FINE-TUNING
In this section, we further explore the efficacy of fine-tuning in improving the performance LLMs. We use the training sets described in Section 4.1 to fine-tune three LLMs: Alama-3.1-88 (Meta, 2024), Gemma-2-9b (Google, 2024), and Mistral-7B-v0.3 (Mistral-AI, 2024c), with the help of the LlamaFactory framework (Zheng et al., 2024) and LoRA (Hu al., 2022). As shown in Table 3, PoT performs the best across our three datasets compared to raw instructions and COT. Therefore, structure our questions guide the LLMs in generating PoT solutions, using Python code from our datasets as expected output. Additionally, we include general-purpose datasets (Alpaca-GPT-4 (Peng et al., 2023) and D51ly-15k (Conover et al., 2023)) and programming datasets (Code Alpaca (Chaudhary, 2023) and OpenCoder (Huang et al., 2024)) to improve the robustness of fine-tuned models. The inclusion of general-purpose datasets fine-tuning inspired by state-of-the-ar cod- ing LLMs (Roziere et al., 2023; Zhu et al., 2024), which use seneral-purposss datasets during their training phases. The inclusion of such datasets is a common practice to ensure the models have certain level of foundational capabilities. Hence, excluding general datasets would compromise the foundational capabilities the models, which shown in Table We continue to use the Acc as the primary evaluation metric for our datasets. Additionally, to evaluate the foundational capabil- ities of the LLMs, we utilize three datasets: MMLU (Hendrycks et al., 2021), Hellaswag (Zellers al., 2019), and ARC (AI2 Reasoning Challenge) (Clark al., 2018). We evaluate LLMMs these benchmarks using the Evvallaaionn conducted zero-shot setting. We use Ace or Ace-Norm stated the framework an evaluation metric. Specifically, for Hellaswag and ARC, we use Acc-Norm; for MMLU, we use Acc.
framework (Gao et al., 2024), and all evaluations are
Table shows the string processing capability of LLMs when prompted with PoT, while Table shows the foundational capabilities of LLMs and Table 8 shows their performance on code genera- tion benchmarks, both before and after fine-tuning. From these results, we can draw the following conclusions: 1) Our fine-tuning is effective. The results in Table 6 clearly show that the perfor- mance of all three I.MMs significantly improves after fine-tuning. For instance, the Mistral-7B-vO.. (Mistral-AI, 2024c) model's Acc on the Random String dataset improves by 60.11 This finding demonstrates that our fine-tuning effectively enhances the LLMs' capability in string processing. 2) The fine-tuned LLMS maintain their foundational capabilities. Table 7 demonstrates that our fine-tuning has minimal impact on the foundational capabilities of LLMs. The results show that all evaluated LLMs exhibit average performance degradation of less than 1.35%. In performance even improves, likely due to the inclusion of two general- purpose datasets during fine- tuning. This finding highlights that fine-tuning only enhances L.LMs* string processing
cases,

---

Published as a conference paper at ICLR 2025
Table 8: Performance of various LLMs on HumanEval, HvmanEval+, MBPP, and MBPP+ bench- marks before and after fine-tuning.
LLM
HumanEval
HumanEval+
MBPP
MBPP+
Avg. Change
Before
67.7
59.1
73.3
63.0
Gemma-2-9b
-0.78
After
66.5
59.1
72.8
62.2
Before
36.0
31.1
50.3
42.1
Mistral-7B-v0.3
-0.68
After
34.8
30.5
49.2
42.3
Before
64.5
55.5
68.0
55.6
Jlama-3.1-88
-1.05
After
63.4
54.3
67.5
54.2
Table 9: Acc/Ace-Norm for foundational capabilities of different LLMs before and after fine-tuning, where LLMs are finetuned without general-purpose datasets.
LLM
MMLU
Hellaswag
ARC
Avg. Change
Before
71.88
80.08
64.76
Gemma-2-9b
-..49
After
71.41
79.76
61.09
Before
59.74
82.90
58.62
Mistral-7B-vd.
1.60
Afier
57.72
82.52
56.23
Before 67.94
79.33
55.03
Llama-3.1-83
0.46
After
65.65
80.60
54.68
ability but also preserves their foundational capabilities. 3) The fine-tuned LLMs maintain their coding capabilities. Results in Table indicate that the code generation performance of LLMs shows slight decline (no more than 1.05% average) before and after fine-tuning. However, our fine-tuning method is quite naive, since it is not our primary focus. Al companies with ad- vanced model training pipelines and substantial computational resources can better integrate our dataset with general and code-specific data, enabling them to maintain both the general and coding capabilities of LLMs.
Comparing finetuning with alternative approaches operating at the tokenizer level is another inter- esting direction. As a brief investigation, Table 13 Appendix shows LLMs' performance where the characters in the string are encoded separately. In this setup, each character is encoded by its token ID the vocabulary, rather than being processed as part of a complete word or subword. Surprisingly, the performance of LLMs degrades compared Table 3 in the paper. One possible explanation that this essentially employs a different tokenizer, as it encodes the same input text in different way. The performance drops since LLMs are trained on the original tokenizer. Re- training LLMs with new finer finer-graine grained tokenizers would be another alternative approach. Exploring such approaches would be exciting avenue for future LLM developers, but it goes beyond the scope of our work. Additionally, retraining an LLM with a new tokenizer would require significant computational resources, which are unavailable
7 CONCLUSION
In this paper, we presented the first comprehensive study on LLMs' capability in string process- ing. we proposed StringLLM, method construct datasets for benchmarking string processing capability of LLMs. Following this, we used StringLLM to develop a series of datasets covering a broad range of string processing tasks and different types of strings. Extensive experiments con- ducted on these datasets indicated that current LLMs have limited capability in processing strings compared to humans.
address this limitation, we provided a detailed analysis to uncover the underlying reasons for LLMs struggle with string processing tasks. Building on these insights, proposed an effective solution that enhanced LLMs' performance via fine-tuning. Our fine-tuned models increased average test accuracy of three datasets by least 38.80%. Interesting future work includes 1) training more capable LLMs for string processing; 2) refining our datasets; and 3) further analyzing the underlying limitations of L.LMs in string processing.
10

---

Published as a conference paper at ICLR 2025
8 ACKNOWLEDGMENTS
We thank the anonymous reviewers for their very constructive comments. This work was supported by the Microsoft Accelerating Foundation Models Research program, the Commonwealth Cyber Initiative (CCT), and NSF under grants No. 2112562, 1937787, and 2125977. We also acknowledge the credits provided by Microsoft Azure.
REFERENCES
Sahil Chaudhary. Code alpaca: An instruction-follloiing llama model for code generation. https:
//github. com/ a sahil wwaii2800 wwtni1aooli. / aodaa//aaaa 2023. 9
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint 0rXiv:2211.12588, 2022. 2,
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv: /803.05457v 2018. 9
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi. Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open
instruction-tumed Ilm, 2023. URL viablllyyyyyleeeeee instruction tuned 11m. 9
http::://wwwwatttttte com com/blog http//llog///// 2023/0224 /4/ 12/dolly first open coweerccialllyyeoooo.o...cccitie
Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Hefferman, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. 2022. 5
DeepSeek-AL. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.044: 2024. 5
Lco Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL https: httpp::///eeooo..oo org records/ 12608602. 9
com Riley Goodside. Riley's s blog post, 2024. URL httt:s//ss.ccm/ppodssdd////assa
Google. Gemma 2 is now available to researchers and developers, 2024. URL https:
google/technol logy/ htwplllygy/eeeeeeeeeeeee developeeesesseeeeeeee haappperr//ggleeeeeeeeeerreeeeee wwwgg/oooolglgooooooe gemma 2/. 5,
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation large language models. In International Con- ference Learning Representattons, 2022. URL https:/ opwnreview. net a forum?
idd=nee/keeFYf. 9
Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J Yang, JH Liu, Chenchen Zhang, Linzheng Chai, et al. Opencoder: The open cookbook for top-tier code large language models. arXiv preprint arXiv:241 1.04005 2024. 9

---

Published as a conference paper at ICLR 2025
Meta. Introducing llama 3.1: Our most capable models to date, 2024. URL htthhttp://// /ettt.
hon//boo//metee 11ama- 3-1/. 5,
Mistral-AI. Codestral: Hello, world!, 2024a.
URL http:://misttrll.all/ess
codestral/. 5
Mistral-AI. Mathstral, 2024b. URL htt.httpssmmmttaaaaaaaass/s/s/as
Mistral-AL.
Mistral 7b, 2024c.
URL
https://mistraalaal//ewww
antouncing- 7b/. 5,9
OpenAl. Introducing ChatGPT, 2022. URL www.httt////eet
OpenAl. GPT-4 technical report. arXiv preprint arXI::230330874,222333
OpenAI.
New
models
and
developer
announced https://openai.com /indxx///
products
at devday, 2023b. new new-models- and developers oro.d
URL
OpenAl. Hello GPT-4o, 2024a. URL www.::ttps///e UtW.arrttps//pppp......c///////e 3,5
1,
OpenAL. Learning to reason with Ilms, 2024b. URL Beamiing-to--rasoo--tttbb----
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35:
27730-27744 2022. 2
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308 2tw.:22008..2900, 2023. 2,
Andrew Shin and Kunitake Kaneko. Large language models lack understanding of character com- position of words. arXiv preprint arXiv:2405. 1357, 2024. 1,2
Zhiquan Tan, Lai Wei, Jindong Wang, Xing Xie, and Weiran Huang. Can i understand what i create? kolfl-nnwweedge evaluation of large language models. arXiv preprint arXiv:2406.061440 2024. 1, 2
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of thouhht prompting elicits reasoning in large language models. Advances in neural information processing systems, 08844844248377 2022. 2,
Gilad Yehudai, Haim Kaplan, Asma Ghandeharioun, Mor Geva, and Amir Globerson. When can transformers count to n? arXiv preprint arXiv:2407.15160, 2024. 2,
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint arXiv: /9050078300 2019. 9
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Anmual Meeting of the Association for Computational Linguisties (Volume 3: System Demonstrations, Bangkok, Thailand, 2024. Association for Computational Linguis- tics. 9
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-followii evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. 1.2
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao. Shirong Ma, et al. DeepSeek-Coder. Coddeee--Code-V2 Breaking the barrier of clored-source models in code intelligence. arXiv preprint arXiv:2406.1 1931, 2024. 5,9
12

---

Published as a conference paper at ICLR 2025
A
APPENDIX
LLMs
#Parameters
Model Provider
GPT-4-Turbo
Unknown
OpenAl
GPT-4o
Unknown
OpenAI
GPT-3.5
Unknown
OpenAl
DeepScek-Coder
16B
DeepSeek
DeepSeek-Chin
16B
DeepSeek
Gemma-2-9ba
9B
Google
Mistral-7B-v0.3
7B
Mistral Al
Mathstral-7BB
7B
Mistral Al
Codestral-22B-vil
22B
Mistral AI
1 1-BB -88B-Instruct
8B
Meta
Table 10: Number of parameters and model providers of L.MSs used in our experiments.
Hash Function
Length
MD5
32
SHA-1
40
SHA-256
64
BLAKE2b
128
SHA3-224
56
SHAKE-12S
32
BLAKE2s
64
SHA3-512
128
SHAKE-256
64
SHA-384
96
Table 11: Hash functions used in our data construction and their output lengths (number of alphanu- meric characters).
Code
Variables
Templates
ans +
a, b
Concat string {a} and {b).
ans
a,
Repeat string {a} for {b} times.
ans a [:y]]
a Y
Retrieve the first {y} characters from string (a).
ans a [[:-1]
Get the reverse of the string (a).
ans len (a)
Determine the number of characters in the string (a).
ans in
x,
Does string {y} contain substring (x}?
cocnttt.cc (x)
a,
Check the occurrence of (x) within string (a).
ans a.strip (x)
a, x
Remove characters {x) from both ends string (a).
ans
a.startswith (x)
a, x
Identify string {a} starts with {x}.
ans a. a.endswith wtwwith.r (x)
a, X
Determine if string {a} ends with substring (x).
Table 12: Examples of atomic tasks.
13

---

Published as a conference paper at ICLR 2025
Table 13: Accuracy for string processing capability of different LLMs, where each character of strings is assigned single token ID.
LLM
Method
Multilingual
Hash
Random String
AVG
Raw Inst.
18.46
19.69
12.34
16.83
Gemma-2-99
CoT
20.55
23.59
16.23
20.12
PoT
50.86
54.35
16.95
40.72
Raw Inst.
4.37
4.68
2.82
3.96
Mistral-7B-vDl
CoT
10.19
10,98
5.22
8.80
PoT
29.18
30.63
13.31
24.37
Raw Inst.
9.62
17.09
8.11
11.61
Llama-3.1-80
CoT
16.76
22.32
18.14
19.07
PoT
38.20
39.06
17.74
31.67
14

---

Published as a conference paper at ICLR 2025
Prompt
Use Chain-of- Thought reasoning, solve the question below step-by-step:
[Question] Retrieve string "B2r8DbCcr fAO o10
2#281779550A799 99 with its characters in reverse order.
Model Output
To
solve the problem of reversing the string
B278DbCcFrA01 www.01eccc6899988 C31 7C8D2 65233cc04651119920555556 7F9553877777797777778 using
Chain-of- Thought reasoning, let's break down the steps:
Understand the Problem: We need to reverse the string, which means that the first character of the original string should be the last character in the reversed string, the second character should be the second last, and on.
Input String:
DECE FA BBBECCCEFFA.66E00010000066111113
3. Reversing the String: To reverse a string, we take the last character and move towards the first character, appending each character in reverse order.
Let's break down the reversal step-by-step:
Last character:
Second character:
Third last character: 7
Continue process until reach first character: B.
4. Construct the Reversed String: By appending each character from the last to the first, we
get:
5. Verify the Reversed String: We can check the reversed string by reading it in reverse and verifying it matches the original string.
Final Answer:
The string in reverse order is:
X
Final Output
(a) CoT
Figure 2: Examples of GPT-4o solving a string processing task using different prompt engineering techniques.
15

---

Published as a conference paper at ICLR 2025
Prompt
Retrieve 12
17595A0A799 with its characters in order.
Model Output
The reversed order of the string prg.ae/ccctaaaeecaaaaa...1111110
51 E11779000082881199555 71w.002002889990077787
Final Output
Raw Instruction
Prompt
Write Python code to solve the question below:
[Question] Retrieve "B2F8DbCC B 211.1111111111100222000000111110 #2817195207799 with its characters reverse order.
Model Output
To solve the problem reversing string Python, you can use slicing technique. Here's that will reverse the given string:
Original string
original oag.
Reversed string
seversed string original string[::-1]
string
reversed string
Final Output
(b) PoT
Figure 3: Examples of GPT-4o solving string processing task using different prompt engineering techniques.
16


Reviews:
[{"review_id": "yH2PjHZjY2", "reviewer": "Reviewer_b4nW", "summary": "First comprehensive study of string processing capabilities of LLMs. Aside from just evaluating, authors also proposed specific solutions (via fine-tuning) to improve their capabilities. Critically, while this fine-tuning does improve their string processing capabilities significantly, it does not majorly harm their general-purpose aibilities.", "questions": "Please rethink Figure 2. Same data, same basic idea, but better presentation. For example, do we really need 2 digits for the decimals on b)? Do we even need numbers there at all, given the colors?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This work handles a very specific problem, and it does so well: well-principled, reasonable evaluation, and useful solution. The analysis of the errors is also interesting.\n\nString processing capabilities have an indirect effect on all LLMs tasks, and a direct effect on some of them, so this work is clearly relevant.\n\nAdditionally, given the fact that authors are willing to publish data and code, this works seems reproducible.", "weaknesses": "The main source of LLMs struggle with string processing is tokenization (this is not news). Given that, I'd have liked to see more studies on tokenization. Perhaps pretraining small LMs on different tokenization strategies. \n\nThe presentation of this paper is good enough, but it doesn't have the excellence on this aspect one would like to see on an ICLR paper. For example, Figure 2 is very relevant to the message of the paper, yet it's badly formatted/presented.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cbgGsYqOYW", "reviewer": "Reviewer_iEx9", "summary": "1. The authors presented a methodology to construct a large scale string manipulation dataset, consisting of sets of (question, code, answer) tuples. This is done by first hand crafting a set of code templates, then generate natural language instruction templates describing the code template using GPT, and finally getting the final answers by filling in variables with different types of strings and executing those code.\n2. They presented an analysis attempting to explain why LLMs struggle with string processing tasks.\n3. They demonstrated that fine-tuning improves LLM performance on this task.", "questions": "Overall, I think the paper made some good contributions, but there are some missing experiments. I am happy to revise my evaluation based on author response.\n\n\nBelow is just a small nitpick, though to me it doesn't really undermine the contribution of the paper, since they mainly used PoT to get around the problem.\n\nIn section 5.2. authors claimed that since there is no observable diagonal pattern in the embedding similarity matrix in figure 2b, it suggests that token embedding lacks information on character-level token length. I fail to see how this is the case.\n\nConsider a toy example with just 4 tokens. Let's design the embedding s.t. it is 2d and the first dimension always encode the character length. Let's say these four vectors are (grouped by character length) E1 = [[1,1],[1,-1]] and E2=[[2,2],[2,-2]]. By definition we would get Sim(E1,E1) = 0, Sim(E2,E2)=0, but Sim(E1,E2)=0.5, where there is NO diagnoal pattern, but by construction it captures character level token length.\n\nMaybe the authors can consider proposing an alternative analysis to strengthen their claim that tokens lack character level information.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. I appreciate the effort to address this problem. This seems like a useful benchmark for improving LLMs code generation ability for string manipulation\n2. They presented a clever way of leveraging closed source LLMs for constructing a dataset, and made a good and thorough effort at demonstrating that it poses a challenge to current LLMs.\n3. They showed PoT fine-tuning helps without hurting performance on other NLU tasks.", "weaknesses": "The authors only reported performance on MMLU/ARC/HellaSwag and show that it only degrades a little. I think since this is finetuned on code gen specifically for string manipulation, this should also be benchmarked on code generation benchmarks such as HumanEval, MBPP, CoNaLa, or the newer BigCodeBench/EvalPlus to make sure it doesn't hinder performance in those areas.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v8TxXIxis9", "reviewer": "Reviewer_BYDo", "summary": "This paper is concerned with the string processing abilities of large language models, and the paper introduces a novel benchmark for quantifying string processing abilities. The benchmark is used to compare a number of existing models, and it is shown that finetuning can improve the performance of models on the benchmark.", "questions": "How much of the current weaknesses of models to handle string processing tasks is related to relatively simple facts and practices related to tokenization? For example, how would if affect the results if all strings in the test data were replaced with string that featured white space between all characters?\n\nRegarding multilinguality, even if the data contains strings from different languages, all questions are still in English. Would it not make sense to also include multilingual instructions in the data if the goal is to measure multilingual capacity?\n\nThe experiments in section 6 includes the use of some general-purpose datasets. What would happen if these were not included? The conclusion in this section states that \"our finetuning not only enhances LLM's string processing ability but also preserves their foundational capabilities\". But one suspects that this is only due to having included the general-purpose datasets in the finetuning?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper present a new benchmark for quantifying string processing abilities of large language models. This is a novel contribution that may be helpful to rank and select models for certain types of practical applications.", "weaknesses": "The claim regarding the multilinguality of the dataset is not convincing, since it only pertains to the strings and not the questions in the dataset.\n\nThere are some redundant material in the paper, e.g. the section on how LLMs analyze input text via Transformers can be removed since it is common knowledge in the community, and so is the cosine similarity.\n\nI do not agree with the intuition that the token embeddings should contain information about token lengths, and as such the results in section 5.2 are not very surprising. Even so, I do not agree with the conclusion that this explains why models fail to perform string processing tasks, since my intuition is that this is more or less completely dependent on tokenization. \n\nThe paper shows that results can be improved by finetuning, but there are no comparison with alternative approaches that operate on the tokenizer level rather than updating the whole model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oe81wdta3b", "reviewer": "Reviewer_rXuV", "summary": "This paper proposes a set of new tasks (and a method to generate the datasets) to evaluate LLMs on string operations. String operations have been a challenge since the beginning of LLMs. In this paper, a method based on GPT-4o is proposed in order to generate large sets of evaluation data for string operations. Instead of asking the model directly, they leverage the high programming capabilities of GPT-4o to generate a program that performs the string operations, and in that way, generates the corresponding input-output pairs correctly. Regarding the evaluation, they evaluate a battery of LLMs ranging from GPT-4o to open-source alternatives like Mistral and Llama. The result suggests, as previously expected, that the LLMs struggle with character-level operations, particularly for random strings and hashes.", "questions": "* **Q1**: Why is the human baseline so low when it comes to string operations in the Multilingual setting?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The paper presents a clear experimental setup with problem definition, automatic dataset generation, and model evaluation. \n* The paper is well-written and easy to follow.", "weaknesses": "* The major weakness of this paper comes from the motivation: it is well known that LLMs struggle with character-level operations, because, as they mention, the tokenization process removes the information about the characters. One may ask why you would ask an LLM to perform such operations, and I think it is not properly reflected in the introduction of the paper. From another perspective, they made use of an LLM to generate the evaluation dataset, which means that an LLM is capable of performing string operations, even though not directly (executing the code the LLM themselves generate). So, would not be a solution for the problem you present to provide an LLM the ability to use a Python interpreter or a predefined set of functions? I think the paper will be more strong if these problems are addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)