Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors of this paper propose various subtokenization strategies affecting input string length in a way so as to improve the efficiency of LLMs when trained on source code and when applied to downstream tasks such as code generation, code summarization and code clone detection.
Authors propose a strategy the restricts vocabulary size as well as compresses lengths of inputs without affecting performance via several combination strategies in the UnigramLM and BPE tokenization schemas.
The main contribution of this paper is to study the impact of tokenization for source code based applications.
strength_and_weaknesses: Strengths
- this paper studies an interesting problem.
- authors consider code from multiple programming languages.

Weaknesses
- this paper is a little hard to follow at times, writing could be clear and so could the organization of sections in the paper.
- the paper primarily compares UnigramLM and BPE, with the authors claiming in the motivations that such tokenizers have demonstrated improvements in NL tasks. However, the authors fail to provide qualitative examples explaining why similar strategies achieve improvements on source code.
- while authors use the Sentencepiece vocabulary, they do not compare against Sentencepiece. I am not sure why the comparisons were restricted to UnigramLM and BPE alone.
clarity,_quality,_novelty_and_reproducibility: The paper is hard to read and follow in some instances.
While it is known that vocabulary size and combination steps in BPE style algorithms impact the performance of a LM, it is hard to fully understand why it is the case for source code and exactly what gap are these general NL techniques addressing on source code.

Experiments on their own seem easy to reproduce.
summary_of_the_review: While the problem is interesting, the presented analysis fails to provide insight into how proposed subtokenization strategies exactly help with source code.
Furthermore, I find it a little puzzling why comparisons against Wordpiece were omitted. Similarly while the point was to demonstrate the strengths of subword tokenizers, commentary on how they compare against other tokenization schema, i.e whitespace etc would be nice to see given that the vocabulary of source code is fairly limited.
On the whole while I see the experiments as thorough I am still unsure as to why these results need to be considered as novel.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Subtokenization is one of the unsung heroes of application of deep learning to code. The paper fills the gap by systematically investigating the results of several subtokenization approaches: BPE, UnigramLM, punctuation combination, native, and systematically reports statistics.
strength_and_weaknesses: (+) Studies an important under-studied problem.

(+) Experimental results are convincing

(-) It is not clear WordPiece is not included as several code-related model use it.

(-) While the numbers are interesting, the recommendation (using punctuation combination tokenization) appears somewhat obvious with hindsight – given the punctuation-heavy syntax of programming languages.

clarity,_quality,_novelty_and_reproducibility: The presentation is clear though the presentation can be more insightful and inspiring. The paper has some empirical value.
summary_of_the_review: Subtokenization is an important aspect of models for code. The paper systematically explores the area and presents numbers which mostly confirm the the implicit assumptions of current models. The recommendation of punctuation combination approach is helpful information. Not including wordpiece is limiting.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper implements and evaluates several tokenization schemes for pretraining large language models for coding tasks. The paper considers BPE vs UnigramLM algorithms, different vocabulary sizes, and different levels of composition allowed in tokens. Then they pre-trained PLBART-like models with these different tokenizations and compare results on several downstream tasks including code translation, code summary and code generation.  

strength_and_weaknesses: With the recent increase in pretrained large language models for code, investigating the effect of the different tokenization schemes is an important problem. This paper takes a step in that direction by thoroughly experimenting with different combinations of the design space and drawing useful insights. 

Although the results have a huge variance and does not really show one choice of tokenization is fully superior over another, I commend the authors for the effort it took to do this thorough evaluation and believe, the results will be useful for the people in this community working on similar areas. 

One concern I have is that all the evaluation was done with a single model, PLBART. And so, it is not clear if these insights drawn in this paper would apply to other kinds of models/sizes and other coding tasks. 


clarity,_quality,_novelty_and_reproducibility: The paper is mostly well-written, but the presentation of the results could be a bit more precise in several places. For example, the abstract says that the proposed approach reduces average length by 17-40% without downstream performance drop. But 40% reduction is achieved with level 4, for which there is a significant performance drop. There are several such imprecise statements through out the paper, which makes the paper hard to read. 

In figure 2, please use the same scale in the x-axis for the all the plots for easy comparison.

As an additional result, it would be useful to look at how does training and inference times change with the different tokenization schemes. 


summary_of_the_review: The experiment setup and evaluation is thorough and the paper presents previously understudied insights for the community, although it is not totally clear how the insights extend to other kinds of model architectures.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work studies the impact of various sub-tokenization methods for neural language models trained on source code. Across a range of tasks and datasets, it demonstrates a number of actionable insights, such as that UnigramLM trained models perform better than BPE ones, and that certain types of tokenization (especially ones that fuse tokens across whitespace and newlines) tend to yield poorer downstream task performance in many settings, while others can be safely used to achieve both roughly comparable results while using fewer tokens to represent the same sized input.
strength_and_weaknesses: I reviewed a previous version of this paper that was similar to the current submission. As in my review then, I maintain that this is an interesting and novel paper. While it is perhaps more empirical in nature than most work published in ML venues, it provides many actionable insights and highlights the ways in which modeling source code is distinct from natural languages.

Given that the authors addressed my comments on the prior version, I have no significant weaknesses to note, other than, perhaps, that the model (PLBART) that all the results are based on is relatively small by current standards (two 6-layer Transformers). Given that this already required nearly 10K GPU hours due to the breadth of tasks and settings the model was trained for, I do not fault the authors.
clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to follow. The results are portrayed quite clearly. For the most part, the confidence intervals are rather wide and the results somewhat variable from one task to another, so that some of the findings do not readily permit any interpretation with statistical significance (e.g., Fig. 5). It may be worth emphasizing this in the discussion.
summary_of_the_review: This work presents actionable insights on a timely topic. It is rather empirical in nature, but comprehensive and well written. It would be good to publish it.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work explores how different tokenization strategies affect transformer-based models of source code. It shows that the tokenization method has an impact on model performance for a variety of code-related tasks. 

## Strengths
- Exposes an interesting problem in source code-related language models that has been overlooked since these methods were originally developed for natural language
- Shows that some tokenization methods can effectively reduce sequence size, allowing for generating longer sequences or improved computational speed.
- Provides actionable insights to anyone interested in training code-related LLMs and other transformer-based models.

## Weaknesses
- It is unclear how these observations are affected through model scale. For example, it's unclear if models with 100s of billions of parameters would have similar performance characteristics with respect to subtokenization. (Yet, sequence length will still be reduced)
-  A larger variety of subtokenization methods could have been explored.
- The presentation can be improved at some places (see reviewer comments)

## Other
A minor comment:
I would encourage the authors to include the SentencePiece training args in the appendix for the various settings they study.
justification_for_why_not_higher_score: While the ideas discussed in this paper are broadly applicable, it relates only to a small subset of the community. In additional, due to (presumed) constraints, the authors do not compare how tokenization affects really large language models, which could have made this work interesting to a wider audience.
justification_for_why_not_lower_score: This work is technically sound and exposes an under-explored issue that affects most transformer-based models of code. If this were to be rejected, the visibility of this work would be diminished. This would be unfortunate as transformer-based models of code (e.g. LLMs) are of interest to many in the community.
Decision: Accept: poster