CUTE: Measuring LLMs' Understanding of Their Tokens
Authors: Lukas Edman, Helmut Schmid, Alexander Fraser


Reviewer paper summaries: he paper explores the extent to which LLMs can learn orthographic information by proposing a benchmark called CUTE (Character level understanding of Token Evaluation).  
The research questions consist of the following:

1. Do LLMs know which characters make up their tokens?
2. Can they understand difference between semantic and orthographic similarity?
3. Can they change text at character level?

CUTE has the following task types:

1. Composition
2. Similarity
3. Manipulation
4. Insertion
5. Deletion
6. Substitution
7. Swapping
Summary of strengths: The paper is well written, it does not promise any more or less than what it delivers. The paper, for its size, is comprehensive in the number of phenomena it explores.
Summary of weaknesses: The paper fails to make a comprehensive case on why character-level understanding of the tokens is an important property of an LLM. Furthermore, the discussion is not perfectly logical at some points (lines 265 to 276; performance gap could be artifacts of more training and not always scaling).
Comments, suggestions and typos: The authors may benefit from removing some aspects of the task descriptions and adding what these tests signify for LLM development.
Overall assessment scores: 3.5