CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Code
Authors: Nadezhda Chirkova
Sergey Troshin
Venue: ICLR 2023 poster (2023)
Keywords: source code processing
tokenization
byte-pair encoding

Abstract: Recent works have widely adopted large language model pretraining for source code, suggested source code-specific pretraining objectives and investigated the applicability of various Transformer-based language model architectures for source code. This work investigates another important aspect of such models, the effect of different subtokenization options, and aims at identifying most effective and length-efficient subtokenizations, taking into account source code specifics. We propose subtokenziation that reduces average length by 17--40% without downstream performance drop, and show that a carefully chosen subtokenization may improve  quality by 0.5-2%, possibly with some length increase.

Published as a conference paper at ICLR 2023
CODEBPE: INVESTIGATING SUBTOKENIZATION OPTIONS FOR LARGE LANGUAGE MODEL PRETRAINING ON SOURCE CODE
Nadezhda Chirkova
Sergey Troshin
Naver Labs Europe
University of Amsterdam s. Atroshin@uva..
andia.chirkovaCe @ @awveraaaa.cooo
ABSTRACT
Recent works have widely adopted large language model pretraining for source code, suggested source code-specific pretraining objectives and investigated the applicability of various Transformer-based language model architectures for source code. This work investigates another important aspect of such models, namely the effect of different subtokenization options, and aims at identifying most effective and length-efficient subtokenizations, taking into account code specifics. We propose subtokenziation that reduces average length by 17% without downstream performance drop, and show that a carefully chosen subtokenization may improve quality by 0.5-2%, possibly with some length increase.
INTRODUCTION
With the inspiration from the success of large language model (LM) pretraining in natural language processing (NLP), BERT-like models have been widely adopted for source code processing (Feng al. 2020 Kanade al., 2020), as code has similar discrete sequential structure to natural text. Being trained on huge source code corpora in a self-supervised manner, large LMs often substan- tially outperform domain-specific models developed purposely for applied tasks, especially in the tasks with limited parallel / labelled data (Ahmad et al. 2021a). These tasks include fixing code bugs, generating text from code and vice versa, or translating code between programming languages.
Recent works advanced large LM pretraining on source code in two main directions. First, various model kinds were utilized for source code: CodeBERT (Feng al.
(2000) and CuBERT (Kanade et al. 2020) rely on the classic encoder-only RoBERTa (Liu et al.) (201)), CodeGPT (Lu et al.
2021 uses decoder-only GPT (Radford Narasimhan 2018). PLBART (Ahmad et al.. 202 Ta) is based on the denoising sequence-to-sequenee BART (Lewis et al., 2020) model, and To (Wang) cct al., (222b) utilizes multitask sequence-to-sequeeee T5 (Raffel et al. 2020 Second, a range of code-specific supervupere pretraining tasks were proposed to enrich the classic masked language modeling (MLM) objective, e. g. GraphCodeBERT (Guo et al. 2021) predicts data flow connections during pretraining (one variable is computed from another variable), and CodeT5 (Wang et al.) 2021b) and DOBF (Roziere el al. (021) use a variable naming objective.
This work is devoted to investigating usually not paid much attention when pretraining large LMs on source code. Modern LMS usually preprocess sequences using open-vocabulary models such Byte-pair encoding (BPF, [Sennrich et al. 2016) which split long tokens into smaller subtokens. Though this process is often referred to as tokenization, we call subtokenization, became a standard part of all widely-used LMs pretrained on natural text or code, because it ensures the relatively high frequency of all subtokens (compared to the whitespace-separateed tokenization, tokens), at the same time producing sequences
more important component, subtokenization, which is
underline its smaller granularity. Subtokenization
which results in large portion of out-of- of reasonable length (compared to character-level tokenization). Though subtokenization was ini- tially introduced for NLP, it is especially relevant for code, programming languages usually permit identifiers unrestricted complexity, e. g. variable or function names (Chirkova & Troshin 2021).
Work done while heing HSE University.

---

Published as a conference paper at ICLR 2023
BPE-50K
Commoniy used
Grouping punctuation
17% langth reduction without quality drop
Lists [[D. 0.] for range
Freq Lists o for range
$
Unigrami.M- 10K
+0.5.2% quality
Grouping frequent combinations
40% length reduction
19% ength increase)
(sometimes quality drop)
Freq List 0 for range 2
Freq List [0,0] fon_i_in_roage (vo
Figure I: Example subtokenizations (all numbers compared to the commonly used BPE-50K).
Though subtokenization is often chosen with only superficial dcliberation, is of the essential model components which may affect both quality and prediction speed. First, an inaccurately chosen subtokenization procedure may substantially increase sequence lengths and consequently slow down prediction. As a simple example, the work on CodeT5 (Wang et al. 2021b) notices that using BPE trained specifically trained on natural text. Second, a line of recent research points at the positive effect of the carefully chosen subtokenization procedure Durrett (2020) show that using a UnigramLM (Kudo, 2018) subtokenization algorithm instead of BPE improves the quality of BERT-based question answering or textual entailment in English by 1%, and Ding et al. (2019) show that adjusting BPE vocabulary size in translation may produce +4 BLEU. the same time, for large LMs, the particular subtokenization procedure chosen the pretraining stage becomes inseparable part of the model and must later be used in applied tasks. This underlines the need for careful choice of subtokenization options when pretraining large LMs.
source code corpora makes sequences 30 45% shorter than using BPE
the model's performance in NLP. For example, [Bostrom &
In this work, we conduct deep study of subtokenization options for large LM pretraining source code, using PLBART as a testing ground. In addition to investigating general aspects, e. g. the subtokenization algorithm and the vocabulary size, study the ways of adapting subtokenization to the specific properties of code, such as a large amount of punctuation marks and frequently- used token combinations, a variety of complex identifiers, languages. We aim at choosing optimal subtokenization options that (a) lead to the best performance or (b) minimize sequence lengths (and thus speed up the model) without downstream performance drop. Our contributions are as follows we show that for large LMs pretrained on source code:
relative similarity of programming
Grouping punctuation chars in single tokens reduces the average length by 17% without downstream performance drop (we call this approach CodeBPE or CodeUnigramLMM) and permitting more complex composite tokens reduces lengths by 40%, sometimes with qual- ity drop (Section])
UnigramI.M is generally preferable over BPE (Section
Smaller vocabularies may improve quality with 3-19% length increase (Section 5):
Subtokenizers are well transferable between programming languages (Section 6):
Our length-efficient subtokenization procedure (see examples in Figure compresses sequences by 17% without quality drop and our most effective subtokenization improves performance by 0.5-2% significantly in three out of eight tasks and by one standard deviation in two other tasks.
2 METHODOLOGY AND EXPERIMENTAL SETUP
The existing works on large LMs for source code usually choose a particular subtokenization li- brary, for example the same as in the base LM the work uses, and train the subtokenizer with the vocabulary size of 30-50K on source code corpora used for pretraining. Often code is pre- processed before subtokenization, on white-spaces and punctuation marks so that these tokens are further split into subtokens, g. for in range (vocSize) will be split into ['for', 'i', 'in', 'range', '(, 'vocSize', '] even if for i in is generally a frequent combination. The latter principle appears to be intuitively reasonable, since it ensures that subtokenization preserves syntactically meaningful boundaries of tokens (Kanade et al.] 2020). We refer to this principle as prohibiting composite tokens. More details subtokenization in different LMs for code given in Section 7
g. by replacing \n with NEW_LINE and split into tokens
2

---

Published as a conference paper at ICLR 2023
We treat the described commonly approach a baseline, and conduct series of experiments, each modifying the baseline subtokenization procedure in one dimension and pretraining PLBART with the new subtokenization. The dimensions vary are follows: the allowed complexity of composite tokens, the subtokenization algorithm, the vocabulary size, the set of languages the subtokenizer is trained on, and the of stochastic subtokenization. These dimensions inspired either by the specifics of source code or by the recent works on subtokenization in NLP.
Experimental setup. As our base model, we use PLBART (Ahmad et al. 202]a). since it comes with the released pretraining code and data preprocessing routine under the MIT license. We use the same model size, the pretraining dataset size and other hyperparameter settings, including finetuning hyperparameters, as in PLBART In particular, we use an encoder-decoder Transformer architecture with 6 layers in cach part, with the model dimension of 768 and 12 heads (140M parameters). The pretraining data consists of 230M Python functions, 470M Java functions (crawled through BigQuery and 47M natural language (NL) descriptions (crawled from StackOverflov referred to as sequences below. The BigQuery dataset consists of repositories with clear open-source license. We pretrain all PLBART models for 100k updates, as in the original paper.
As applied tasks, we consider three tasks from the PLBART paper: code generation (generating a Java function based on an NL description; CONCODE (Iyer al. 2018) dataset, CodeBLEU (Ren iet al. 2020) metric), code summarization (generating an NL description for a Python or Java func- tion; CodeSearchNet (Husain al. 2020) dataset, BLEU metric), code clone detection (classifying whether two Java functions implement the same functionality; BigCloneBench dataset (Svajlenko) & Roy 2013); Fl metric), and one additional task of code translation (translating code from Python to Java and vice versa; AVATAR dataset (Ahmad et al. 2021b)). Here we consider original data with the CodeBLEU metric (Code Translation-1] and the smaller version of data with tests and the Computational Accuracy metric which portion of generated functions passed all tests (Code Translation-2). We chose tasks so that we have both code generative and discriminative tasks and that datasets are either in Python Java.
We clip all sequences by 510 subtokens, except summarization where we clip by 250 subtokens following /Ahmad et al. (2021a). Such clipping remains the majority of sequences unclipped in all subtokenizations: 96-99. 1% in the pretraining data, 93-99% in translation, 88-100% in generation, 76-93% in summarization, and 37-80% in clone detection. In the main text we report average lengths computed on the randomly chosen subset of pretraining data before clipping, Appendix A reports length statistics for downstream data with similar trends observed for the pretraining data. We only clip sequences passed to neural networks and use unclipped target sequences when computing metrics.
Baseline subtokenization. Following Ahmad et al. (2021a), we use a SentencePiece (Kudo & Richardson 2018) library, which is one of the most widely used solutions for subtokenization. We train subtokenizers on 10M functions and NI. descriptions randomly selected from the pretraining data (different from the random subset on which
measure average lengths). Though/Ahmaat et al.) (2021a) use BPE subtokenization algorithm, our baseline subtokenization uses another algorithm, UnigramLM, because it was shown to be quantitatively and qualitatively more suitable for pretrain- ing in NLP than BPE (Bostrom & Durrett 2020). We also perform their comparison for code in Section 4 We set the vocabulary size to 50K (the commonly used size for large LMs of code) and character coverage to 99.99% (enough to cover English chars and punctuation).
We also use PLBART's preprocessing which includes removing comments and docstrings, replacing \n, indents and dedents in Python with NEW_LINE, INDENT and DEDENT tokens as they are part of the language syntax, and removing formatting Java as it does not affect the language syntax. Our baseline subtokenizer follows the commonly used strategy of prohibiting composite tokens described above. The only exception we make that we allow underscores inside tokens, because they do not represent syntax unit, other punctuation chars do.
github-repan
3

---

Published as a conference paper at ICLR 2023
Table 1: Different levels of allowed composite tokens complexity considered in the paper. Green emphasizes tokens which could not be obtained in the previous level, and gray emphasises the re- maining tokens that could not be obtained Level Levels list allowed merges, but what particular merges to perform is chosen by the tokenizer.
Lev.
Description
Example
0
Whitespaces in the middle of tokens prohibited and each punctuation char treated separate token (except "_) Similar to Level O, but tokens consisting of several punctuation chars are allowed
['for', 11,, 'in", 'range', 11, fat", 'shape', ""," "t', "NEN_ LINE", "INDENT', 'print', "." 12", "y, "NEW LINE', 'print', (aey" "colunns", "1"
I'foc', 'I, 'in'. 'zange'. '(, 'df", ".'. "shape', 'C, 'I, '1) 'NEW LINE INDENT", "print', "17, NEW LINE,,," 'print', 'ds",""" 'colunna", "11,1
2
Similar to Level 1, but dots are allowed in tokens
I'for', '4', 'in', "range', "t" 'dE", "shupe", "P 11 "1 INDENT", 'prine', C.
NEWL,NE,
'print', C. ".""",, 'columan'', "T", '1", 1)
3
Whitespaces and single punctuation chars
[Mfor in range', '(dF, S, shape [T,, 1) LINE INDENT', 'print'. '(, NEW'LINE", "peint", '( dr, column', [1'
allowed in tokens, except [XW_I.INE
171
4
Composite tokens of arbitrary complexity are allowed
range', "( dr, shape', 'E T.I, "P', LINE,,,', INDENT print', " L NNEWLLING print', ' df, columns', '[i])]
transi.1 (Py. CodeBLEU) Code transl--2 (Py, Comp. Acc.)
summ. (Py. BLEU)
Code gen. (ja, CodeBLEU)
length
Level
Leve
100
Levn
55
19.5
20.0
(Ja. CodeBLEU)
(ja, Comp. Acc.)
detection (la,
50
Level
Level
Leve
47
48
49
45
50
55
18.5
19.0
19.5
97.5
98.0
98.5
01234
Figure 2: Results on various subtokenization granularity, averaged over 4 finetuning runs (mean = standard deviation). Level () baseline subtokenization. Numerical data for all plots is given in Appendix.
SUBTOKENIZATION GRANULARITY
In natural text, portion of punctuation chars is small and thus their separation in subtokenization does not affect lengths much. In contrast, in source code, punctuation constitutes 12.8% of chars and often forms frequent combinations joining which into composite tokens may substantially reduce lengths. Further, the presence of a large amount of commonly used patterns is another specific feature of source code, g. for (int i 0; in Java def _intt_ (self) f) in Python, and these patterns again may form composite tokens. This section investigates the effects of the use of composite tokens on performance and length-efficiency.
We consider several levels of allowed complexity of composite tokens listed in Table [ and empir- ically compare them in Figure 2 The extreme cases are composite tokens (Level 0, equal to the baseline subtokenization) and unrestricted composite tokens complexity (Level 4, composite tokens constitute 48.6% of the vocabulary). The average sequence length Level is 40% less than that in Level 0). At the same time, the effect on performance depends on the task: in code-generative tasks (translation and generation). Level performs significantly worse than Level O, and in code understanding tasks, Level 4 is either similar marginally worse than Level 0 (code summarization) or even significantly better (clone detection). Because of quality loss encountered in several tasks, consider intermediate levels.
Level makes one step further from Level 0 and allows punctuation char merges, e. g. ")) or "]):'. Though such punctuation composite tokens constitute only 3.4% of the vocabulary, their use reduces average length by 17%: from 97 to 80.7, and since this level does not mix punctuation with other chars, it presumably should not complicate code generation much. Level 2 makes more
4

---

Published as a conference paper at ICLR 2023
step further and allows merging dots with textual tokens. This reduces the average length by 23% compared to Level 0. The motivation for Level 2 is that a lot of API name tokens almost always go with the dot, e. g. join or split in Python. Figure 2 shows that Level I model performs similar or better than Level 0 model in all tasks, and Level 2 performs similar or better than Level 0 in six tasks, marginally worse in Python code summarization and significantly worse in Java code generation.
Level 3 makes a step back from Level and restricts the complexity of composite tokens in such manner that each composed token may represent either simple one-line code pattern or a punctu- ation combination. but could not combine them. Quantitatively, Level performs generally better than Level 4, but (marginally or significantly) worse than the previous Level 2 in six tasks and similarly in two tasks (generation and clone detection).
To sum up, punctuation combinations (Level 1) result in sequence lengths reduction by 17% without performance drop in all tasks. We verify this result for BPE in Appendix B and call this approach CodeBPE Code Undeenigr. Length reduction could be increased up to 24% in most tasks by allowing dots attached to tokens (Level 2) and up to 40% in most code understanding tasks by al- lowing arbitrary subtoken combinations (Level 4). we investigate the transferability of subtokenizers between programming languages in Section
One of the potential issues with using composite tokens in code- onde-generative tasks is that an inaccurate generation of a "long" token may change the entire following generated code. For example, in Java- Python code translation, cycle which traverses all unique element pairs in array, converts to
for in range 0 / arr_ size - -1 ) for r in range ( 1 + arr_
While the Level 0 model generates exactly the specified cycle and the Level I model only modi- fies the first eycle: range ( arr_size ), making even more concise, Level model generates
for in range 0 arr arr_size for r in range ( arr size )
which results in traversing some elements twice. Here the first cycle begun with tokens 'for 1 in' and 'range ( and the second cycle begun with tokens 'for r in' and 'range ( where the latter repeats the previously used token and starts an incorrect line. However, according to our manual prediction analyses, such inaccurate generation, if it happens, rarely results in wrong code and often does not affect code semantics. For example, the Level 3 model may generate ['range 0 ,, 'n )'] instead of cquivalent range (n.). Another example is that this model may generate [ o
for i in range { ) ] instead of two nested cycles by beginning with tokens [ [' ('and" '0 ] *", resulting in even more concise code.
As for composite tokens in Level 1, they contain only punctuation and are "simpler" than in Level 3. Besides, Level composite tokens serve more often for statement closing (e. g. ') " at the end of the cycle specification) than for a harder starting of new statements: 46.3% of Level composite tokens contain only closing brackets, 12.8% contain only opening brackets and 26.7% contain both. We also check that using punctuation composite tokens does not deteriorate syntactic correctness: in Java-Python code translation-I. I, Level 0 and Level I models generate a similar number of syntac- tically correct test code snippets: 1226 and 1239 correspondingly. At the same time, for the Level model, this quantity only equals 1163.
In Appendix C we also analyse how much do input and output subtoken sequences intersect in different Levels and find that generally the higher granularity leads to the lower intersection rate. This may be another explanation for the superiority of the lower granularity subtokenizations, as intuitively it should be casier for the model to predict correct subtokens if they present in the input sequence.
As aurogregressive decoding is a slowest part of the encoder-decoder pipeline (Berard et al. 202 it is important to check that the length statistics of sequences generated by the models comprising composite tokens are close to those of the data. We check it for Java-Python translation- 1: while groundtruth sequences at Levels 1 and 3 are 13.5% and 50% shorter than at Level O, the generated sequences these levels are 15% and 40% shorter than sequences generated at Level O.
5

---

Published as a conference paper at ICLR 2023
Code transi-2 CodeBLEV)
Code transl Comp, Acc.
Code
Code CodeBLLE!!
Av. length
BPE 50K
130
UnILM
45 Code transi-1 (j. CodaBLEU)
66
19.0
10.5
20.0
37
38
30
100
Code transi. 22 (a. Comp, Acc.)
Code summ. (ja, BLEU
Clone detection (a. F1)
BPF Uei N SOK
UnitM 2x
19.5
50000 1802
Figure 3: Comparison of BPE and UnigramL.M subtokenizers and of several vocabulary sizes. Uni-
gramLM 50K - baseline subtokenization.
Table 2: Example subtokenization of identifiers by UnigramLM and BPE subtokenizers
Original token
UnigramLM subtok-
BPE subtokenization
Native subtokenization
enization
(Camel- snake_case)
tvomhottedString
['trom's 'string'l
'notted'
['rom', 'not', 'ted', 'string']
['rrom,, 'tocted', "string']
isInstantiated
['aa 'Istaantiate' 'a'l
C'sstn', 'stanti", 'ated'] ['GridRag', 'converter"]
Pis', "nettattiaaaed"
G GRagooooeeerrr
['srid', Converter"]
'Bag'.
['Erid', 'Bag', 'Converter']
isSameSize Hori corizontally ly
I'isSame".
"size',
('isSame', "size', 'H'. 'orizontally]]
l'ss'. 'Same'. "Size'. "Horizontally"
"Horizontally]
PA PALRiorarchy_ID
"itiarcchhr (LDD"
[*eA', "_,, "ierarchy', '_D']
['FA', "2. "Hierorchy', L. 'TD']
We note that though we use sequence clipping which clips lower granularity subtokenizations stronger, it does not devalue our results, as generally lower granularity subtokenizations perform better in our experiments, and also such clipping is a widely used practical scenario; we provide more comments in Appendix C At the same time, in rare cases when lower granularity subtok- enizations perform worse than higher granularity ones, this may be indeed due to the max-length clipping. For example, we find that the more "fair" cropping eliminates the superiority of Level compared to Level ( in code summarization and they start performing similarly (see Appendix C..
4 SUBTOKENIZATION ALGORITHM
Bostrom & Durreti 2020 compare two most popular subtokenization approaches, BPE and Uni- gramLM (Kudo) 2018). , for pretraining of large LMs on natural text data. While BPE constructs the vocabulary in the bottom-up fashion, starting from characters and gradually joining them, the Un- igramLM algorithm works in the top-down fashion, staring from large vocabulary and gradually filtering it. The paper finds that UnigramL.M outperforms BPE in range of downstream tasks and suggests several reasons for the superiority of UnigramLM, including better alignment with mor- phology and the more efficient vocabulary allocation. Since most existing pretrained LMs on source code use BPE, we decided to compare the two algorithms for source code.
Figure compares BPE and UnigramLM for PLBART. In three tasks, UnigramLM outperforms BPE by one standard deviation, and in remaining tasks their performance is very close. Since the average length of two tokenizations is similar, we recommend using UnigramLM for source code.
Bostrom & Durrett (220)) argue that one of the potential reasons for the superiority of UnigramLM subtokenization that it better aligned with natural text morphology and thus simplifies the com- position of words by parts. We find that similar effect appears for identifiers in source code: although 80% of identifiers are subtokenized identically by UnigramLM and BPE, for some of the remaining 20%, UnigramLM provides more "reasonable" splits into subtokens, see examples in Ta- ble 2 More formally, we observe that UnigramLM subtokenization better resembles splitting into subtokens based Came CCase or snake. case, which call native subtokenization. To estimate this effect quantitatively, we consider the Python corpus and randomly select a set of 150K identifiers with different UnigramL M and BPE subtokenizations consisting of > 2 native subto-

---

Published as a conference paper at ICLR 2023
kens, and measure the average Jaccard similarity J(A, |/n B| between the set of native subtokens and the set of subtokens produced by each subtokenizer. The resulting score for Uni 26.6%, is much higher than for BPE, 15.2%. As could be observed from the sometimes subtokenizers join two native subtokens into one
third and the fourth rows in Table 2 (isSame, Gri Sriddag) g). If we split each subtoken produced by tokenizer based on Came 1 Ca or snake case to eliminate this effect and again measure average Jaccard similarities, UnigramLM's score, 55.2%, is still much higher than BPE's, 47. 77, again indicating that UnigramLMMss s tokeniza- tion is better aligned with the native one. In Appendix C we measure intersections between inputs and outputs in the intersection rate than BPE, which may be connected to the better alignment with native subtok- enization and serve as a a possible explanation of UnigramL.M slight performance superiority.
tasks and find that UnigramLMM leads a slightly higher
A relatively frequent pattern is that BPE tends to detach the first uppercase letter from native subto- kens (H ori brizontall y row 4, iera erarchy in row 5). Among 150K identifiers considered in the previous paragraph, 14.6% of BPE tokenizations contain at least one single uppercase letter X and 4.4% - at least one subtoken of kind _X, while for UnigramLM these scores are substantially lower and equal to 1 11.8% and 1.4% correspondingly. At the same time, BPE merges two native subtokens more frequently (GridEs iBag in row 3): 45.8% BPE tokenizations contain at least one to- ken which could be split into two more based on Came I Case, while for UnigramL.M this score only equals to 39.2%.
5 VOCABULARY SIZE
This section studies the effect of vocabulary size, one of the main subtokenizer's hyperparameters, on the downstream quality of PLBART. Though the existing pretrained LMs for code use relatively large vocabularies of 30-50K tokens, we are interested, whether using smaller and less length- efficient vocabularies could result in better performance, and if yes, how large is the length increase.
Figure presents the comparison of PLBARTs trained with vocabulary sizes 50K (large), 10K (medium) and 2K (small). We find that in code translation, all vocabularies lead to similar per- formance. In code summarization, small and medium vocabularies outperform the large one by one standard deviation. In code generation, the medium vocabulary significantly outperforms the large one. Finally, in clone detection, decreasing the vocabulary size deteriorates quality. At the same time, with the large vocabulary, sequences are shorter than with the smaller vocabulary by 9.5% (10K) and 33% (2K), but the model size is larger (139M for 50K, 108M for 10K, and 102M for 2K). We conclude that vocabulary size reduction may lead a slight performance improvement bur with sequences elongation, thus it may be helpful in applications with high cost of errors and weak restrictions on sequences lengths. We verify the highlighted result for BPE Appendix. We note that compared to BPE 50K which is used in most existing large LMS of code, UnigramI-M 10K improves performance significantly in three tasks and by one standard deviation in two other tasks.
Reducing vocabulary size increases the granularity of identifiers subtokenization, e. e.g. reachable subtokenized as reachable with the 50K vocabulary, reach able - with 10K and re ach able with 2K. In other words, vocabulary size reduction may be seen stronger prohibition of complex tokens than Level 0 in Section [ Our results on the effectiveness of smaller granularity agree with the machine translation results of Ding et al. (2019). Programs in code generation and summarization data are more identifier-centered, e. g. the model often needs to choose correct API based on the natural language description which seems be easier by compos- ing from smaller subtokens. On the contrary, in code translation, data is more algorithmic-centered with mostly short identifiers encoded in 1-2 subtokens with all vocabulary sizes. The length increase of 10K vocabulary compared to 50K one is 6-19% in the former two tasks (6% in generation, 19% in summarization) and only 3.5% in the latter one scode-translation-1
even
TRANSFERABILITY BETWEEN PROGRAMMING LANGUAGES
Due to the high computational cost of large LM pretraining and relative programming languages similarity, e. g. compared to how dissimilar natural languages could be, pretrained LMs on source code often used for programming languages that were not considered during pretraining. In this
7

---

Published as a conference paper at ICLR 2023
Code trans.12 (Py, CodeBLEU)
Code transi-2 (Py, Comp, Acc.)
Code summ. (Py, BLEU)
Code Us. CodeBLEU)
Average length
45.5
46.0
46.5
65
19.50
19.75
38.0
38.5
013
Code transl.1 CodeBLEU)
Code transl-2 2 Us. Comp,
Code summ. BLEU
Clone detection (a, Fi)
45
49
56
58
18.5
19.0
97.5
98.0
by
(Orly (Py-lst (dirly
Figure 5: Results of transferability between programming languages. Py+Ja subtokenizer is trained on all data (baseline), Only Py- on Python and natural language data only.
section, we investigate the effect of using subtokenizer trained on one programming language for another programming language.
Figure g visualizes the number of tokens having particu- lar frequencies in Python and Java languages, and black rectangles denote language-specifie areas. We find that the baseline Level 0 granularity vocabulary seems to lunguage-universal the majority of subtokens have large frequencies both languages, and only small number of subtokens, 12.6%, are frequent in one language and rare in another. Interestingly, for Level 4 vocabulary, this quantity is not much higher, 20.1 %. though it should in- clude all language-specific composite tokens. As com- posite tokens occupy almost half of the Level vocab- ulary, the remaining 30% composite tokens are common for two languages.
Frequency in Pythen
Level granularity granularity
7550
3000
2550
Lang-
Frequency in Java Frequency Java
lohams
Figure 4: Number of tokens of differ- ent frequency in two languages, Uni- gramLM 50K vocabularies.
Analysing sequence lengths (Figure 5). we observe that training the subtokenizer without Java (Only Py) shortens Python sequences marginally and increases Java sequences by 6.5% compared to the baseline subto- kenizer trained on data (Py+Ja). The latter happens because some widely used Java identifiers not merged into single tokens they not used in Python; still, the length increase is not so large. For the Level 4 granularity subtokenizer, Only Py's length increase on Java is larger, 13%, since it contains more language-specific composite tokens. However, due to common composite tokens, the resulting Level 4 Only Py's Java average length is still smaller than Level Only Py's Java sequences: 79 83.
As for performance, using the Only Py subtokenizer instead of Py+Ja changes quality up to one stan- dard deviation and could both increase and decrease it Java data (quality increase may be caused by the increased subtokenization granularity). Note that we only change subtokenizer configuration PLBARRT is still pretrained on languages, this may happen in practice LM's developers use the subtokenizer from another project, e. g. for comparison purposes. Summing up. we conclude that the baseline subtokenizer is universal and, if needed, could be used for other programming languages it was not trained on, with small length increase and slight quality change. We note that Python and Java have quite different syntax and are usually used different applications.
7 RELATED WORK
Subtokenization studies for NLP. Subtokenization has become an essential component mod- em NLP pipelines and thus - subject of a line of empirical NLP studies. While word-based models suffer from the oot-of-vocabulary problem, subtoken-based (open-vocabulary) as well as char-based approaches cover arbitrary novel words. Among various open-vocabulary approaches, BPE (Sennrich et al.) 2016), WordPiece (Wu et al. (2066) and UnigramLM (Bostrom & Durrett, 2020 became most widely used, and UnigramLM was shown outperform BPE for LM pre- training (Bostrom & Durrett 2020). line of studies investigate the optimal granularity word subtokenization: Ding et al. (2019) find that in Transformer based neural machine translation, small vocabularies of 0 0-4K subtokens outperform large ones by up to 4 BLEU, and VOLT (Radford et al.) 2018) automates the search of a proper subtoken vocabulary with proper size by formulating
8

---

Published as a conference paper at ICLR 2023
optimal transport problem. The smallest char-based granularity is often avoided because of substantial sequences elongation, but has particular strengths, c. g. much less number of hyperpa- rameters and better robustness, and thus appears to be a promising research direction (Gupta al. 2019 Clark et al., 2021 Tay et al. 2021). Provilkov et al. (2020); Kudo (2018) propose stochastic subtokenization way improve new words composition and (Wang et al. 2021a) adapt it to pretrained LMs. Finally, an actively studied challenge is that various natural languages need differ- ent subtokenization decisions and are hard to subtokenize with one common model (Chung et al.) 2020 Rust et al., (021). Our work investigates most of the specified directions for source code. For a more detailed review on subtokenization, see (Mielke al., 202T).
Subtokenization practices in neural source code processing. Subtokenization was first tested for source code in (Karampatsis et al., 2020) and later used in the majority of Transformer-base models. Almost all LMs pretrained on source code use BPE-like subtokenization with large vocabu- lary: CodeBERT uses the WordPiece (Wu et al. 2016) algorithm (a modified BPE, 50K), CuBERT an algorithm from the Tensor2Tensor project Vaswani et al. 2018) (50K), PLBART and CodeGPT
BPE (50K), CodeT5 - byte-level BPE (32K), DOBF uses subtokenization procedure of either CodeBERT Rooziree et al. (2020) (BPE 64K) for fair comparison, AlphaCode Li et al. (2022) SentencePiece (8K, algorithm not specified), InCoder Fried et al. (2022) BPE (50K). To the best of our knowledge, existing works do not provide an in-depth experimental analysis of various subto- kenization options for code and, particularly, do not investigate various levels of composite tokens complexity. Though the concurrent work of FFied al. (2022) uses unrestricted composite tokens (our Level 4), they do not compare them to any other subtokenizations. Level 4 composite tokens are conceptually similar to code idioms used in ([yer et al. 2019; Shin et al. 2019) for code gener- ation, but the mentioned works develop specific procedures for mining idioms, which need separate implementation, while we rely on the commonly-use subtokenization procedure.
8 CONCLUSION
In this work, we conducted an empirical study of varying subtokenization options for large LMs pretraining on source code. We believe that main the value of our work is not in improved numer- ical criteria, but importantly in providing reference experiments for the community showing which impact (both substantial or small) subtokenization choices may have in pretraining LMs for code. This underlines which directions to look more carefully at in practice (the use of composite tokens) and which are less important to experiment with (vocabulary size, subtokenization algorithm), and whether the later directions can bring at least a slight improvement or not (yes, they can). Currently most works experiment with subtokenizations options only superficially or do not experiment all, and we hope that our work will provide motivation to do that.
Our recommendations. As for particular direct recommendations from our results, first, we rec- ommend to use the proposed punctuation combination approach, which we call CodeBPE or Code- UnigramLMM depending on the used subtokenization algorithm, that shortens sequences by 17% without quality drop. We suppose that with larger pretrained LMs higher levels of composite tokens may also achieve comparable performance; we were not able to experiment with then as they require very extensive computational resources. Second, if changing the subtokenization algorithm is easy, e.g. when using the SentencePiece library, we recommend using the UnigramI.M MM, since it performs slightly better than commonly used BPE with similar lengths. Third, recommend considering releasing models with smaller vocabularies, as they may perform slightly better than larger vocab- ularies. In our experiments the UnigramLM-100 subtokenizer was 0.5-2% more effective than the commonly-used BPE 50K in 5/8 experiments, but with 3.5-19% length increase.
Limitations The main work's limitation that we consider only the PLBART model, due to the limited computational resources. However, we believe that the provided recommendations will motivate and simplify the process of the subtokenizer's tuning for future works, as described above. Another limitation is that we focus on finding optimal subtokenization options only for source code, though some downstream tasks also include the processing of natural language. Investigating the ways of choosing optimal subtokenization for both code and natural language may be interesting direction for future research. Finally, we only compare BPE and UnigramLMM, while it could be interesting to investigate the performance of other algorithms, e. g. WordPiece.

---

Published as a conference paper at ICLR 2023
BROADER IMPACT
We do not anticipate any direct negative social impact of our work. However, our results may potentially be used for developing new pretrained LMs for source code, and detailed discussion on their broader impact is provided in Chen et al. (202I) (Section 7), e. over-reliance on generated code or producing vulnerable code. Unfortunately, our work may cause negative environmental impact because of computation (~5K Tesla A-100 GPU hours and ~4K Tesla V-100 GPU hours at the internal cluster).
ACKNOWLEDGMENTS
The results were supported by the Russian Science Foundation grant N: 1919-71-000 The research was supported in part through the computational resources of HPC facilities at NRU HSE.
REFERENCES
Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai- Chang. Unified pre-training for program understanding and generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computaaionnll Linguistics: Human Language Tech- nologies, pp. 2668-2668, Online, June 2021a. Association for Computational Linguisties. URL https: / www. www.aacle..
ac www.:www....bbbbbb...ee........ ooge 2021 nwa.cacccc..co....cc....a.. -aaac-mmain 211
Wasi Uddin Ahmad, Md Golam Rahman Tushar, Saikat Chakraborty, and Kai- Reii Chang. Avatar: A parallel corpus for java-python program translation, 2021b.
Alexandre Berard, Dain Lee, Stephane Clinchant, Kweonwoo Jung, and Vassilina Nikoulina. Ef- ficient inference for multilingual neural machine translation. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pp. 8563-8583, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguis- tics. doi: 10.18653/v1/002 wwtp:/221...mmpmmmmm....... /tt httpt//s///aaaaaooooooo.ooo///// emn emm.pp ma maain.. 674
Kaj Bostrom and Greg Durrett. Byte pair encoding is suboptimal for language model pretraining. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4617-4624, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.18653/w120000
findings-emnlp.414. URL https: https://aclaatthooooy.ooo wttp:ll......./220....00 findingindnggg emnlp.
414
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings. Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss,, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, 1lya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code, 2021.
Nadezhda Chirkova and Sergey Troshin. A simple approach for handling vuo-bl-vocabulary identi- fiers in deep learning for source code. In Proceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 278-288, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1 2021.naacl-main.25 URL :tttps:/i / a ant howooolo.y......... www.ogg..o...ggoooo 2021 naacl m..in.2
Hyung Won Chung, Dan Garrette, Kiat Chuan Tan, and Jason Riesa. Improving multilingual models with language-clusters vocabularies. In Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pp.
Online, November
10

---

Published as a conference paper at ICLR 2023
2020. Association for Computational Linguisties. doi: 10.186533w122000emmmll. main URL htt.:y/oorrr2222...oaaaamaaaaaaa emnlp-main. 367]
Jonathan Clark, Dan Garrette, Iulia Ture, and John Wieting. Canine: Pre-training an efficient tokenization-free encoder for language representation, 2021.
Shuoyang Ding, Adithya Renduchintala, and Kevin Duh. A call for prudent choice of subword merge operations in neural machine translation. In Proceedings of Machine Translation Summit XVII: Research Track, pp. 204-213 213, Dublin, Ireland, August 2019. European Association for Machine Translation.
Zhangyin Feng, Daya Guo, Duyu Tang. Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin liang, and Ming Zhou. CodeBERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1536-1547, Online, November 2020, Association for Computational Linguistics. doi: 10. 2t00://..022...n..gg-ooooooo weniiiinnnnnngnnl.... 2020.
URL https: www.wwwww......w...wwwwwwwwwwwww wtw../wwww..../tttooooooo//////
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis, 2022. https::::://axiiiii
org/ abs/ 05999
Daya Guo. Shuo Ren, Shuai Lu. Zhangyin Feng. Duyu Tang, Shujie LIU, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. Graphcode (bert]: Pre-training code representations with data flow, In International Conference Learming Representations, 2021. URL
n httpe//ooummmdd-oooeede
Rohit Gupta, Laurent Besacier, Mare Dymetman, and Matthias Gallé, Character-based nmt with transformer. ArXiv, 1.149991..9997
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Mare Brockschmidt. Code- searchnet challenge: Evaluating the state of semantic code search, 2020.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Mapping language to code in programmatic context. In Proceedings of the 2018 Conference Empirical Meth- ods in Natural Language Processing, pp. 1643-1652, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/D18-11 1111D18--19.... 7 //acllnnthloogy.orggg w1wpgy/.ggg///--1111111
h
Srinivasan lyer, Alvin Cheung, and Luke Zettlemoyer. Learning programmatic idioms for scalable semantic parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5426-5435, Hong Kong, China, November 2019. Association for Com- putational Linguistics. doi: 10.18653/v1/D199 1545. URL http::::///llaaaholloy.....
D19-1545
Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Learning and evaluating contextual embedding of source code. In Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 12-18 July 2020, Proceedings of Machine Learning Research. PMLR, 2020.
Rachel-Michael Karampatsis, Hlib Babii, Romain Robbes, Charles Sutton, and Andrea Janes. Big code != big vocabulary: Open-vocabulary models for source code. In Proceedings of the ACM/IEE 42nd International Conference Software Engineering, ICSE '20, pp. 1073-1085, New York, NY, USA, 2020. Association for Computing Machinery, ISBN 978 457781450372166.. doi:
00555/33778113338044 /ttp:::s:///di org. 1ttp:///..1111337111113333
Taku Kudo. Subword regularization: Improving neural network translation models with multi- ple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 66-75, Melbourne, Australia. July 2018. Association for Computational Linguistics. doi: 10. 10.18653/v11/P 1007. URL https: www.llantoooooy..rooo.
11

---

Published as a conference paper at ICLR 2023
Taku Kudo and John Richardson. SentencePiece: A simple and language independent sub- word tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing: System Demonstrations, pp- 66-71, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 2.18653/v1/D18-2011 URt.htpss:/ antho wwwphhhlllyyoor//1-------
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-seqqeence pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, On- mc1nmann7333
line. July 2020. Association for Computational Linguistics. doi: URL
www.://gg..00../. acl-main 703
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang. Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Push- meet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode, 2022. URL pitetps://ariiee
abs /203. 07814
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11199 2019.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. Codexglue: A machine learning benchmark dataset for code understanding and generation. CoRR, abs/2 144.0644,,4 2021.
Sabrina J. Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y. Lee, Benoît Sagot, and Samson Tan. Between words and characters: A brief history of open-vocabulary modeling and tokenization nlp, 2021.
Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. BPE-dropout: Simple and effective sub- word regularization. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics. pp. 1882-1892, Online, July 2020. Association for Computational Linguis- URL
tics. doi: 10. 15.11866//1222 actp:/2000....mmamama acl -am-mminn.. 170
Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre- training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Lan- guage models are unsupervised multitask Iearners. 2018. URL /tLtttsss//dmaeeeeeeeee
models language Langgageee m.dees..ddedd...
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Research, 21(140): 1-7,, 2020. URL http: 7/jmlr wttp:/pptppppoooootottttt v2tps/v11/22 - html
Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, M. Zhou, Ambrosio Blanco, and Shuai Ma. Codebleu: a method for automatic evaluation of code synthesis. ArXiv, abs/2009.10297, 2020.
Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. Unsupervised translation of programming languages. In H. Larochelle, M. Ranzato, R. Hadsell, M. Bal- can, and H. Lin (eds.). Advances in Neural Information Processing Systems, volume 33, pp. 20601-20611. Curran Associates, Inc., 2020. URL /tL/stttpp///ppoceedgnn..gu new.eedddddnnn...nsss...........
12

---

Published as a conference paper at ICLR 2023
Baptiste Roziere, Marie-Anne Lachaux, Mare Szafraniec, and Guillaume Lample. Dobf: A deob- fuscation pre-training objective for programming languages. In Advances in Neural Information Processing Systems (NeurIPS 2021), Online, 2021.
Phillip Rust, Jonas Pfeiffer, Ivan Vulié, Sebastian Ruder, and Iryna Gurevych. How good is your to- kenizer? on the monolingual performance of multilingual language models. Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 3118 18-3135, On- line, August 2021. Association for Computational Linguistics. doi: 10.1 111111/16/022222/ lacl-oog.. 243, URL //g.y//o////////////a/ooooooooon
wtw.l/oo.orrgg/222. 2wc.o2o0......ooooo 243
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Com- putational Linguistics (Volume Long Papers), pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 10.18653/v1/1166 1/6. URL https: //aclanntolooyy.oorr//i---
Richard Shin, Miltiadis Allamanis, Mare Brockschmidt, and Oleksandr Polozov. Program Synthesis and Semantic Parsing with Learned Code Idioms. Curran Associates Inc., Red Hook, NY, USA, 2019.
Jeffrey Svajlenko and Chanchal K. Roy. Evaluating clone detection tools with bigelonebench. In 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 131- 140, 2015. doi: 10.1 109//CS
Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. Charformer: Fast character transformers via gradient-based subword tokenization, 2021.
Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan Gomez, Stephan Gouws, Llion Jones, łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. Tensor2Tensor for neural machine translation. Proceedings of the 13th Con- ference of the Association for Machine Translation in the Americas (Volume I: Research Track), pp. 193-199, Boston, MA, March 2018. Association for Machine Translation in the Americas. org/ W18 1819
URL https:
Xinyi Wang, Sebastian Ruder, and Graham Neubig. Multi-view subword regularization. In Pro- ceedings the 2021 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, pp. 473-482 Online, June 2021a. As- sociation for Computational Linguistics. doi: 10.18653/v1/200 naacl-main.40 URL https:
//aclanthol www.ology.rrg......... org/2021. naa.l m..in.00
Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8696-8708, Online and Punta Cana, Dominican Republic, November 2021b. Association for Computational Linguistics. URL httpp::/aaat www.annoooo.oooooooooo
emnleennlemmle main. 685
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin John- son, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.082 44, 2016. URL http:://aaaat org/ abaaabssb10 00016.811400
13

---

Published as a conference paper at ICLR 2023
A AVERAGE LENGTHS ON THE FINETUNING DATA
In the main text, we report average lengths computed over a randomly chosen subset of the pre- training data (different from the random subsets subtokenizers were pretrained on). Table VableBrrep reports average lengths computed over finetuning data, we also include average lengths computed over held-out pretraining data. For programming languages, pretraining data: Level compresses sequences by 11.6-17.7% (the lower compression in the code summarization task, 11.6-12.89 12,8%, is explained by the higher percent of identifiers in this data than in other tasks); Level 4 compresses sequences by 32.4-50%; 10K and 2K vocabularies increase lengths by 3.5-1 and 12.7-35.99% respectively; and BPE and UnigramI.M average lengths are similar.
observe similar trends as on the general
As for natural text, its average lengths are not affected by the Level 1 subtokenization because it only affects punctuation char sequences, rarely present the natural language data. At the same time, higher level subtokenizers may produce natural language subtoken sequences of higher lengths, because their vocabularies are occupied by programming-focuse natural language data, while frequent words may be absent in these vocabularies. Small vocabulary subtokenizers also have programming language- focused vocabularies, resulting in higher lengths increase for natural text than for programming languages.
composite tokens rarely present in
In Table 4 we report time needed to generate predictions for the test set in two tasks. The measure- single Tesla V100 GPU, in the same session for all runs. The speed-up is
ment was conducted stronger than the length decrease, because of quadratic Transformer complexity.
B ADDITIONAL EXPERIMENTS WITH BPE
Figure 6 presents the comparison of BPE Level 0 and Level 1 subtokenizations and of BPE 50K and 10K vocabularies. The results are similar to those of UnigramLM reported in the main text: the performance of Level 0 and Level subtokenizations is again close, the model with 10K vocabulary again significantly outperforms the model with 50K vocabulary in code generation and outperforms by one standard deviation in Java code summarization, and in other tasks the performance of 10K and 50K models similar.
Code transi (py. CodeBLEU) Code transl.22 (Py, Comp.
Code (Py, BLEU)
Code gen. (ja, Code
longth
Level
BPE Level
BPE
100
68 Code transi. 1 (ja, CodeBLEU) Code transi.22 (ja Comp. Acc.)
45
46
66
19.0
18.5 Code summ. (ja, BLEU)
37
38
39
Clone detection (j., F1)
BPE Level
50
BPE Level
BPE 10k
18.5
98.0
L0L110K
Figure 6: Comparison of BPE Level 0, BPE Level (both with 50K vocabulary as in the main text) and BPE 10K.
C ADDITIONAL ANALYSIS
Additional experiments with "fair" subtoken sequence cropping. In our main experiments, we clip all subtoken sequences with the same maximum length threshold, as this scenario is most close to how done in practice. should be noted that for the vast majority of examples, subtoken sequences fit to the maximum length limit and are not affected by this clipping (see statistics in the main text, Section 2). However, for the remaining long examples, subtoken sequences produced by subtokenizers of higher granularity may be detokenized into longer character sequences, than that of the smaller granularity subtokenizer's ss For example, if one uses the maximum length of (only used for illustration), then the code snippet
sum numbers ) may be tokenized by the Level 4 subtokenizer into I'x =', "sum (', 'numbers', into ['x", "=', 'sum", "(, 'numbers', '], which will be clipped into ['x", *=', 'sum", "(" with
(fits into the limit) and by 2K subtokenizer
14

---

Published as a conference paper at ICLR 2023
Table 3: Average lengths computed finetuning data in different tasks. All subtokenizations expect the last one use the UnigramLM algorithm; Base ans Level 1-4 subtokenizations use vocab- ulary of 50K; 10K and 2K subtokenizations based on Level preprocessing.
Subtok.
Base
Level
Level 2
Level 3 Level 4
10K
2K
BPE
Code translation Python
Av. len 150.6
126.9
125.1
91.5
87.1
156.0
169.7
150.5
vs. Base
0.0%
-15.7%
-16.9%
-39.2%
-42.2%
+3.6%
+12.7%
-0.1%
Code translation
Java
Av. len 226.8
193.9
173.6
127.3
113.5 -50.0%
234.7
263.0
226.8
vs. Base 0.0%
-14.5%
-23.5%
-43.9%
+3.5%
+16.0%
0.0%
Code summarization - Python
Av. len 151.4
133.9
125.7
107.0 -29.3%
102.3
166.3
202.1
151.3
vs. Base 0.0%
-11.6%
-17.0%
-32.4%
+9.8%
+33.5%
-0.1%
Code summarization Java
Av. len vs. Base 0.0%
132.7
115.7
109.4
95.0
84.9
146.3
180.4
132.5
-12.8%
-1.66
-28.4%
-36.0%
+10.2%
+35.9%
-0.2%
Code summarization natural text
Av. len Base
12.5
12.5
13.3
17.8
11.2
14.5
19.9
12.5
0.0%
0.0%
+6.4%
+42.4%
-10.4%
+16.0%
+59.2%
0.0%
Code generation Java
Av. len vs. Base 0.0%
30.6
25.4
23.8
20.4
18.8
32.7
39.9
30.6
-17.0%
-22.2%
-33.3%
-38.6%
+6.9%
+30.4%
0.0%
Code generation natural text
Av. len 166.5
166.4
166.2
203.5
163.9
205.5
282.3
166.7
vs. Base 0.0%
-0.1%
-0.2%
+22.2%
-1.6%
+23.4%
+69.5%
0.1%
Clone detection
Java
Av. len 349.3
287.6
264.0
220.1
196.9
377.1
449.6
349.6
vs. Base 0.0%
-17.7%
-24.4%
-37.0%
-43.6%
+8.0%
+28.7%
+0.1%
Test pretraining data Python
Av. len vs. Base 0.0%
121.2
100.6
91.8
80.3
75.0
131.4
159.44
121.2
-17.0%
-24.3%
-33.7%
-38.1%
8.4%
31.5%
0.0%
Test pretraining data Java
Av. len 85.3
71.2
66.6
56.6
51.7
93.8
115.3
85.4
vs. Base
0.0%
-16.5%
-21.9%
-33.6%
-39.4%
10.0%
35.2%
0.1%
Table 4:
Time required to generate predictions for the whole test set, in seconds. Numbers in brackets indicate relative speed-up versus Level 0. UnigramLM tokenization, 50k vocabulary.
Level
Level 1
Level
Code translation (Python, 1693 examples) Code generation (Java, 2000 examples)
502
387 (77%)
270 (54%)
346
279 (80%)
211 (60%)
the loss of information. Note, however, that such inequality between subtokenizers does not devalue the results we report. First, the described clipping procedure corresponds to the conventionally used practical setting with the use of maximum available input information, and second, results show
15

---

Published as a conference paper at ICLR 2023
that longer (less compact) subtokenizations, most affected by clipping, generally perform better than shorter (more compact) subtokenizations which allow less information loss. Using the more "fair" setting may make the performance of more compact subtokenizations worse (and not affect less compact subtokenizations), amplifying performance differences further.
In this section, conduct such "fair" experiment for three representative downstream tasks and crop subtoken sequences produced by all subtokenizers so that they are all detokenized into a similar character sequence. In the example above, the 2K subtokenization will stay unchanged while the Level 4 subtokenization will be cropped into X'x =', 'sum (') to achieve equality. The results are shown in Figure 7 Comparing the observations the hypothesis given above, we observe that the performance of smaller vocabulary subtokenizations, less affected by cropping, stays similar (as ex- pected) and that the performance of higher granularity subtokenizations, most affected by cropping, often reduces (Levels 0 and 1 in translation, Level in summarization) or stays similar eLevel O in summarization, Level in clone detection), again as we expected. Surprisingly, in some cases (Level 4 in all tasks and Level 0 in clone detection), the performance of higher granularity subtokenizations improve slightly after cropping. We hypothesize that the reason may be that the last parts of se- quences may be not important for correct prediction, g. if name
main iin_nn"
statements in Python translation examples may be auxiliary and not used during translation.
As for the conclusions emphasized in the main text, they hold for the cropped setting well: (a) the marginal performance superiority of smaller vocabularies compared to the Level 0 50K vocabulary, is amplified (translation) or similar (summarization) as in the main text; (b) the relative performance of Level 0 and Level 4 is similar as in the main text; (c) the observation that the Level subtokeniza- tion performs not worse than the Level 0 subtokenization stays same as in the main text.
Interestingly, in Java code summarization, Level I subtokenization performs slightly better than Level 0 subtokenization in the main text, while after cropping their performance is similar. We thus attribute the superiority of Level I in the main text to the less strong clipping than of Level 0 subtokenizations.
Code transl.-1 (Py. CodeBLEU)
Code summ. (j. BLEU)
Clone detection (ja. F1)
2K
10K
Level 0
Lovel
Level 4
19.50
97.5
98.0
98.5
Figure 7: Additional experiments with subtoken sequences being cropped so that for each datapoint, the subtoken sequences produced by all subtokenizers are detokenized into a similar character se- quence. Effectively this cropping means that the 2K subtokenizer's sequences stay unchanged while all others may be cropped, and the Level 4 subtokenizer's s sequences are cropped the most. Blurred bars visualize original performance reported in the main paper, with "unfair" clipping used in prac- tice. Both 2K and 10K subtokenizations use Level 0 preprocessing and all Level X subtokenizations use the 50K vocabulary, all experiments with UnigramI.M.
How much do input and output sequences intersect for various subtokenizations? In attempt better understand the effect of subtokenizations on downstream performance, we measure the Jaccard similarity J(A, B) [A n BI/ U B between the sets of input and output subtokens in sequence-tor sequence-to-sequence sesseueecce tasks. The intuition correct subtokens if they are present in the input sequence. We only include textual subtokens in the considered sets (subtokens which do not include any programming language punctuation), since "copying" subtokens which include punctuation seems to be unrealistic (in text-to-code code- to-text tasks, the text part does not include programming language punctuation at all, and in the translation task, punctuation of Python and Java are very dissimilar). We consider the setting with "fair" cropping described in the previous paragraph, however for the conventional setting the results similar.
that it should be easier for the model to predict
The results are presented in Table 5 The general trend that the higher the granularity, the lower the intersection rate, appears be reasonable. For example, we observe the explainable monotonic
16

---

Published as a conference paper at ICLR 2023
decrease in intersection for 10KKKKK vocabularies: the smaller the vocabulary, the smaller the granularity of identifiers/words subtokenization, the more chance that different parts of words will repeat. This correlates with our empirical observation that generally smaller vocabulary sutokeniza- tions perform slightly better.
Interestingly, Level subtokenization leads slightly higher intersection rate than Level O. We explain it that punctuation combinations occupy a portion of vocabulary (3.4%) and thus reduce the effective vocabulary allocated for textual tokens, which will be slightly more often split into parts. This effect shows that if one has some "length budget", , it is better to be spent on splitting identifiers into subtokens rather than considering punctuation chars as separate tokens (they can be grouped).
For further levels, two forces start competing: the first one that composite tokens occupy a part of the vocabulary and thus out-of-vocabular identifiers are split into smaller pieces that are repeated rela- tively frequently, and the second one that composite tokens themselves are longer and are repeated rarely between input and output. The second force is absent for Level 1 considered above because Level 1 composite tokens are punctuation-only and do not include letters digits. We observe that for Level 2, the first force outweighs the second one in translation and summarization (intersection rate is higher than for Levels 0 and 1) and underweights in generation (intersection rate lower than in Levels O and 1). Interestingly. this correlates with performance of Level 2 compared to Levels and 1: it is par with them translation and summarization and lower in generation. Further on, at Levels 3 and 4 the intersection rate drops, with small increase Level 4 compared to Level 3, explainable by the specifics of Level 3 construction focused programming statements.
Finally, we compare the intersection rate of UnigramLM and BPE (both Level O, 50K) and find that in summarization and generation it is slightly higher for UnigramLMM than for BPE and in translation they are equal. This observation complements our findings that UnigramLM subtokenizations are better aligned with native subtokenizations than BPE subtokenizations and provides some intuition how it matters (similarity to native subtokenization presumably increases the number of repeated subtokens which should be easier correctly predict).
Table 5: Average Jaccard similarities between the sets of input and output textual subtokens, for different subtokenizations. All subtokenizations expect the last one use the UnigramL.M algorithm; Base ans Level 1 1-4 subtokenizations use vocabulary of 50K; 10K and 2K subtokenizations are based on Level preprocessing.
2K
10K
Level 0
Level
Level 2
Level
Level 4
BPE
Code trans. (Py)
14.61
13.91
13.68
13.81
14.38
6.65
7.79
13.67
Code gen. (Ja)
11.65
10.51
9.57
9.61
8.94
4.70
6.51
8.88
Code sum. (Ja)
10.25
7.26
5.99
6.03
6.33
2.84
5.80
5.67
Subtoken frequences visualization. In Figure we visualize subtoken frequencies computed over 1/8 of the pretraining corpora for four subtokenizers: UnigramI.M Level 0 50K, BPE Level O 50K, UnigramLM Level 1 50K and UnigramLM Level 4 50K. Comparing UnigramLM and BPE, we find that UnigramLM has slightly heavier tail which results in greater number of subto- kens having higher frequncies and thus better embeddings, the similar observation was also noticed close. Level 4 vocabulary exhibits less contrast in frequencies than Levels 0 and I: in the left range, Level frequencies are lower, while in the right range, Level 4 frequencies are higher, which hypothetically could provide some advantage to Level 4 (but it does not, according to the reported performance results). We hypothesize that having high frequeneies in the left range is important for high per- formance (more reliable "basic" subtokens) but high granularity subtokens reduce frequencies of smaller subtokens which negatively affects their embeddings.
in inBostromm & Durrett (2020) The frequency profiles of Level 0 and Level subtokenizers
D NUMERICAL. RESULTS
Table presents the numerical results for figures in the main text and Appendix
17

---

Published as a conference paper at ICLR 2023
109
Subtoken frequency
10a
10?
10c
10a
104
103
UniLM Lev.
UniLM Lev. 1
BPE Lev.
UniLM Lev. 4
102
10000
20000 30000
40000
50000
Subtoken frequency rank
Figure Subtoken frequencies over pretraining data.
Table Numerical data for figures in the main text. CT1: Code Translation-1 (CodeBLEU), CT2:
Code Translation 2 (Computational Accuracy), CS: Code Summarization (BLEU), CG: Code Gen-
eration (CodeBLEU), CD: Clone Detection (F1). Py Python, Ja Java.
Subtokenizer
CT1 (Py)
CT1
CT2
CT2
CS
CS
CG
CD
(Ja)
(Py)
(Ja)
(Py)
(Ja)
(Ja)
(Ja)
UnigramI.M 50K Level 0
46.1
48.2
65.3
57.1
19.7
18.9
38.2
97.8
UnigramLM 50K Level 1
45.9
48.4
67.3
57.8
19.7
19.4
38.2
98.3
UnigramLM 50K Level 2
45.9
48.0
67.0
56.8
19.5
19.3
37.3
98.2
UnigramLM 50K Level 3
45.0
47.7
56.7
45.5
19.5
19.1
37.5
98.5
UnigramLM 50K Level 4
44.2
46.7
54.3
43.7
19.5
18.9
36.7
98.3
UnigramLM 10K Level 0
45.8
48.6
65.7
57.35
19.9
19.2
39.1
97.7
UnigramLM 2K Level
46.2
48.0
66.1
56.2
19.8
19.2
39.1
97.5
UnigramL.M 50K Level
46.1
47.5
68.3
58.6
19.8
38.6
98.0
(Only Py)
BPE 50K Level
45.5
47.7
66.5
57.4
19.3
18.8
37.7
98.0
BPE 50K Level 1
45.9
48.0
65.5
56.9
19.6
19.0
37.4
98.3
BPE 10K Level
45.5
48.1
66.5
57.2
19.5
19.2
38.9
97.9
18


Reviews:
[{"review_id": "hiD2Gj0pdjw", "reviewer": "Reviewer_SjVD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper implements and evaluates several tokenization schemes for pretraining large language models for coding tasks. The paper considers BPE vs UnigramLM algorithms, different vocabulary sizes, and different levels of composition allowed in tokens. Then they pre-trained PLBART-like models with these different tokenizations and compare results on several downstream tasks including code translation, code summary and code generation.  \n", "summary_of_the_review": "The experiment setup and evaluation is thorough and the paper presents previously understudied insights for the community, although it is not totally clear how the insights extend to other kinds of model architectures. \n", "strength_and_weaknesses": "With the recent increase in pretrained large language models for code, investigating the effect of the different tokenization schemes is an important problem. This paper takes a step in that direction by thoroughly experimenting with different combinations of the design space and drawing useful insights. \n\nAlthough the results have a huge variance and does not really show one choice of tokenization is fully superior over another, I commend the authors for the effort it took to do this thorough evaluation and believe, the results will be useful for the people in this community working on similar areas. \n\nOne concern I have is that all the evaluation was done with a single model, PLBART. And so, it is not clear if these insights drawn in this paper would apply to other kinds of models/sizes and other coding tasks. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OibP-TVvz4", "reviewer": "Reviewer_kR7n", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors of this paper propose various subtokenization strategies affecting input string length in a way so as to improve the efficiency of LLMs when trained on source code and when applied to downstream tasks such as code generation, code summarization and code clone detection.\nAuthors propose a strategy the restricts vocabulary size as well as compresses lengths of inputs without affecting performance via several combination strategies in the UnigramLM and BPE tokenization schemas.\nThe main contribution of this paper is to study the impact of tokenization for source code based applications.", "summary_of_the_review": "While the problem is interesting, the presented analysis fails to provide insight into how proposed subtokenization strategies exactly help with source code.\nFurthermore, I find it a little puzzling why comparisons against Wordpiece were omitted. Similarly while the point was to demonstrate the strengths of subword tokenizers, commentary on how they compare against other tokenization schema, i.e whitespace etc would be nice to see given that the vocabulary of source code is fairly limited.\nOn the whole while I see the experiments as thorough I am still unsure as to why these results need to be considered as novel.", "strength_and_weaknesses": "Strengths\n- this paper studies an interesting problem.\n- authors consider code from multiple programming languages.\n\nWeaknesses\n- this paper is a little hard to follow at times, writing could be clear and so could the organization of sections in the paper.\n- the paper primarily compares UnigramLM and BPE, with the authors claiming in the motivations that such tokenizers have demonstrated improvements in NL tasks. However, the authors fail to provide qualitative examples explaining why similar strategies achieve improvements on source code.\n- while authors use the Sentencepiece vocabulary, they do not compare against Sentencepiece. I am not sure why the comparisons were restricted to UnigramLM and BPE alone.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AV6eduMUqx", "reviewer": "Reviewer_RLnm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Subtokenization is one of the unsung heroes of application of deep learning to code. The paper fills the gap by systematically investigating the results of several subtokenization approaches: BPE, UnigramLM, punctuation combination, native, and systematically reports statistics.", "summary_of_the_review": "Subtokenization is an important aspect of models for code. The paper systematically explores the area and presents numbers which mostly confirm the the implicit assumptions of current models. The recommendation of punctuation combination approach is helpful information. Not including wordpiece is limiting.", "strength_and_weaknesses": "(+) Studies an important under-studied problem.\n\n(+) Experimental results are convincing\n\n(-) It is not clear WordPiece is not included as several code-related model use it.\n\n(-) While the numbers are interesting, the recommendation (using punctuation combination tokenization) appears somewhat obvious with hindsight – given the punctuation-heavy syntax of programming languages.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0Ao-czB5XAO", "reviewer": "Reviewer_het6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies the impact of various sub-tokenization methods for neural language models trained on source code. Across a range of tasks and datasets, it demonstrates a number of actionable insights, such as that UnigramLM trained models perform better than BPE ones, and that certain types of tokenization (especially ones that fuse tokens across whitespace and newlines) tend to yield poorer downstream task performance in many settings, while others can be safely used to achieve both roughly comparable results while using fewer tokens to represent the same sized input.", "summary_of_the_review": "This work presents actionable insights on a timely topic. It is rather empirical in nature, but comprehensive and well written. It would be good to publish it.", "strength_and_weaknesses": "I reviewed a previous version of this paper that was similar to the current submission. As in my review then, I maintain that this is an interesting and novel paper. While it is perhaps more empirical in nature than most work published in ML venues, it provides many actionable insights and highlights the ways in which modeling source code is distinct from natural languages.\n\nGiven that the authors addressed my comments on the prior version, I have no significant weaknesses to note, other than, perhaps, that the model (PLBART) that all the results are based on is relatively small by current standards (two 6-layer Transformers). Given that this already required nearly 10K GPU hours due to the breadth of tasks and settings the model was trained for, I do not fault the authors.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster