The Effectiveness of Uncased Tokeniziaion for Clinical Notes
Authors: Cory Paik, Katharina von der Wense
Accepted at: acl25

The Effectiveness of Uncased Tokenization for Clinical Notes

Cory Paik1 and Katharina von der Wense1,2
1University of Colorado Boulder   2Johannes Gutenberg University Mainz
{cory.paik, katharina.kann}@colorado.edu

Abstract
The impact of case-sensitive tokenization on clinical notes is not well understood. While clinical notes share similarities with biomedical text in terminology, they often lack the proper casing found in polished publications. Language models, unlike humans, require a fixed vocabulary and case sensitivity is a trade-off that must be considered carefully. Improper casing can lead to sub-optimal tokenization and increased sequence length, degrading downstream performance and increasing computational costs. While most recent open-domain encoder language models use uncased tokenization for all tasks, there is no clear trend in biomedical and clinical models. In this work we (1) show that uncased models exceed the performance of cased models on clinical notes, even on traditionally case-sensitive tasks such as named entity recognition and (2) introduce independent case encoding to better balance model performance on case-sensitive and improperly-cased tasks.

1 Introduction
Casing is one of many ways to increase the readability of written text. However, more often than not it is not a necessity – after all we communicate verbally without specifying, “capital a” in conversation. Language models (LMs) have fixed vocabularies. For them, case sensitivity is a trade-off between retaining case information to resolve case-sensitive ambiguities and improving efficiency as well as robustness by introducing case-invariance.

Case-sensitive tokenization is often seen as useful for resolving ambiguities, e.g., in named entity recognition (NER). However, for some types of text – such as clinical notes – the observed case is merely an interpretation of spoken language during transcription, which should be re-constructable from lower-case text with sufficient context and knowledge. This is distinct from applications where preserving case is absolutely necessary, such as in programming languages, where case is semantically significant. Moreover, when case-sensitive tokenization results in truncated sequences, then the task becomes more challenging due to missing tokens. For this application, it should be obvious that missing case information will always be easier to infer than truncated tokens.

Prior work in the biomedical domain has shown mixed results on the impact of casing on downstream tasks, with some work showing that cased models perform better on NER tasks (Lee et al., 2020), and others showing that uncased models perform better across the board (Beltagy et al., 2019). To the best of our knowledge, no work has been done to investigate the impact of casing on clinical notes. Clinical notes share some similarities with biomedical text such as PubMed articles with regards to their domain and in the sense that both are littered with case-sensitive acronyms. PubMed articles are much more polished and typically have proper casing, whereas clinical notes are often semi-structured and lack proper casing.

In this work we (1) show that uncased models match or exceed the performance of cased models on clinical notes within 1%, even on traditionally case-sensitive tasks such as NER, and (2) introduce independent case encoding to better balance the trade-off of potentially case-sensitive or improperly-cased tasks.

2 Related Work
General-Purpose Models  While the original transformer introduced by Vaswani et al. (2017) used cased tokens, Devlin et al. (2019) introduced both an uncased and a cased variant of BERT. In their work, they do not explicitly discuss the trade-offs between the two, but used the cased version for named entity recognition (NER) tasks.1 Many models have been released since BERT, using a

1This is further detailed in their readme github.com/google-research/bert.
similar (or identical) architecture. While RoBERTa uses cased tokenization (Liu et al., 2019), most models that are not initialized from BERT use uncased tokenization (Lan et al., 2020; Geiping and Goldstein, 2022; Portes et al., 2023).

Note that we are only concerned with encoder models, not decoder models, which typically use cased tokenization for the output (Radford and Narasimhan, 2018; Radford et al., 2019; Brown et al., 2020) due to other motivating factors (e.g., the fact that humans prefer text with proper casing).

Biomedical Models Lee et al. (2020) introduce BioBERT– which is initialized from the BERT cased model and trained on PubMed Abstracts. While Lee et al. (2020) report that a cased vocabulary performs slightly better on downstream tasks, they do not provide any further analyses or ablations. Beltagy et al. (2019) introduce SciBERT, which uses the same architecture as BERT, but was pretrained from scratch on scientific text with a new tokenizer. They use cased models for NER and uncased for all other tasks following the original BERT methodology, but note that the uncased models sometimes performed slightly better in their initial testing than cased ones on NER. Gu et al. (2021) introduce PubmedBERT, which is trained from scratch on PubMed articles and diverges from previous work by using uncased tokenization. They do not provide ablations to support this decision, but note that the cased and uncased versions had similar performance in preliminary experiments.

Clinical Models Alsentzer et al. (2019) introduce ClinicalBERT and BioClinicalBERT, the latter of which is based on BioBERT and, thus, cased. Note that others have also published models using the same ClinicalBERT namesake, so we will denote this ClinicalBERT as ClinicalBERT_A. For example, Huang et al. (2020) train their model which we denote ClinicalBERT_H directly from BERT uncased on MIMIC-III.2 At a high level, ClinicalBERT_H is pretrained longer and with longer sequences than ClinicalBERT_A, but much less than BioClinicalBERT. Both ClinicalBERT_A and BioClinicalBERT are trained for less time than ClinicalBERT_H on MIMIC-III. Moreover, these do not have overlapping downstream tasks, so it is not straightforward to directly compare them. Wang et al. (2023) re-use the ClinicalBERT name to train a cased model from scratch on clinical notes, but they do not provide much detail about the training process or data.

3 Independent Case Encoding

We introduce a novel model extension by adding independent case encoding to the existing uncased BERT model. This is implemented similar to the existing positional and token type embedding in that is an additive embedding with 4 possible values: uppercase, lowercase, capitalized, or mixed case. This case information is found during tokenization. While training from scratch may provide better results, in our experiments we simply add this embedding to the existing BERT uncased models. For this reason, we only provide independent casing results on models that continue pre-training on MIMIC-III.

The motivation for independent case encoding is to retain some useful case information without being overly burdened when the case is not important. This gives the model the ability to easily ignore case information, unlike the cased tokenization.

4 Experiments

4.1 Dataset Statistics

To analyze the prevalence of improper casing in clinical data, we count the total number of tokens and the number of sequences with greater than 128 tokens for MIMIC-III and the downstream datasets. We also include these metrics for the C4 common crawl dataset (Raffel et al., 2020) to serve as a baseline of tokenizing a different open-domain corpus.

4.2 Pretrained Models

To investigate the impact of case-sensitive tokenization on clinical notes, we train 2 models initialized from BERT base cased (BERT_C) and BERT base uncased (BERT_U) on MIMIC-III (Johnson et al., 2015, 2016; Goldberger et al., 2000), which is a freely available database with thousands of patients from an ICU between 2001 and 2012. It contains patient data, such as demographics, as well as structured timeseries data, such as vital signs, medications, imaging reports, and free-text notes by the nurses and doctors. We preprocess the MIMIC-III dataset using scripts provided by Alsentzer et al. (2019) and create pretraining data using the parameters used by Huang et al. (2020) using the original BERT implementation.3 Following Huang et al. (2020) we train the models for 100k steps with a

2These are concurrent work.

3github.com/google-research/bert
<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="3">Examples</th>
    <th colspan="3">Sequence Length > 128</th>
    <th colspan="3">Num Tokens (k)</th>
  </tr>
  <tr>
    <th>Cased</th>
    <th>Uncased</th>
    <th>↓ %</th>
    <th>Cased</th>
    <th>Uncased</th>
    <th>↓ %</th>
  </tr>
  <tr>
    <td>c4</td>
    <td>Train</td>
    <td>364868892</td>
    <td>269147123</td>
    <td>263073464</td>
    <td>2.257</td>
    <td>178.56B</td>
    <td>171.21B</td>
    <td>4.116</td>
  </tr>
  <tr>
    <td></td>
    <td>Validation</td>
    <td>364608</td>
    <td>269112</td>
    <td>263040</td>
    <td>2.256</td>
    <td>177.87M</td>
    <td>170.61M</td>
    <td>4.081</td>
  </tr>
  <tr>
    <td>MIMIC-III</td>
    <td>Train</td>
    <td>2083112</td>
    <td>250761</td>
    <td>159363</td>
    <td>36.448</td>
    <td>1.28B</td>
    <td>1.13B</td>
    <td>11.376</td>
  </tr>
  <tr>
    <td>MedNLI</td>
    <td>Train</td>
    <td>11232</td>
    <td>120</td>
    <td>98</td>
    <td>18.333</td>
    <td>446.017</td>
    <td>417.671</td>
    <td>6.355</td>
  </tr>
  <tr>
    <td></td>
    <td>Validation</td>
    <td>1395</td>
    <td>26</td>
    <td>16</td>
    <td>38.462</td>
    <td>58.369</td>
    <td>54.562</td>
    <td>6.522</td>
  </tr>
  <tr>
    <td></td>
    <td>Test</td>
    <td>1422</td>
    <td>15</td>
    <td>15</td>
    <td>0.000</td>
    <td>53.799</td>
    <td>50.690</td>
    <td>5.779</td>
  </tr>
  <tr>
    <td>i2b2/2006</td>
    <td>Train</td>
    <td>44948</td>
    <td>92</td>
    <td>58</td>
    <td>36.957</td>
    <td>699.924</td>
    <td>594.722</td>
    <td>15.030</td>
  </tr>
  <tr>
    <td></td>
    <td>Validation</td>
    <td>4994</td>
    <td>10</td>
    <td>6</td>
    <td>40.000</td>
    <td>78.670</td>
    <td>66.682</td>
    <td>15.238</td>
  </tr>
  <tr>
    <td></td>
    <td>Test</td>
    <td>18096</td>
    <td>32</td>
    <td>25</td>
    <td>21.875</td>
    <td>306.989</td>
    <td>268.836</td>
    <td>12.428</td>
  </tr>
  <tr>
    <td>i2b2/2010</td>
    <td>Train</td>
    <td>14684</td>
    <td>21</td>
    <td>16</td>
    <td>23.810</td>
    <td>248.120</td>
    <td>219.393</td>
    <td>11.578</td>
  </tr>
  <tr>
    <td></td>
    <td>Validation</td>
    <td>1632</td>
    <td>1</td>
    <td>1</td>
    <td>0.000</td>
    <td>27.584</td>
    <td>24.390</td>
    <td>11.579</td>
  </tr>
  <tr>
    <td></td>
    <td>Test</td>
    <td>27625</td>
    <td>39</td>
    <td>31</td>
    <td>20.513</td>
    <td>484.395</td>
    <td>433.451</td>
    <td>10.517</td>
  </tr>
  <tr>
    <td>i2b2/2014</td>
    <td>Train</td>
    <td>45794</td>
    <td>668</td>
    <td>591</td>
    <td>11.527</td>
    <td>960.979</td>
    <td>872.000</td>
    <td>9.259</td>
  </tr>
  <tr>
    <td></td>
    <td>Validation</td>
    <td>5088</td>
    <td>80</td>
    <td>74</td>
    <td>7.500</td>
    <td>107.213</td>
    <td>97.167</td>
    <td>9.370</td>
  </tr>
  <tr>
    <td></td>
    <td>Test</td>
    <td>32587</td>
    <td>495</td>
    <td>447</td>
    <td>9.697</td>
    <td>690.789</td>
    <td>623.234</td>
    <td>9.779</td>
  </tr>
</table>

Table 1: Datasets statistics for all downstream tasks. For MedNLI, we use the standard train, validation, and test splits provided by (Romanov and Shivade, 2018). For the i2b2 datasets, we split the training data into a 90/10 train/validation split and use the standard test split. For MIMIC-III, we report the sequence length of individual sentences to better represent the number of truncated sequences. All other datasets use the full sequence length as used by the model. For all datasets, note that uncased tokenization results in fewer overall tokens. For MIMIC-III, we report the number of tokens in billions, for all other datasets they are in thousands.

<table>
  <tr>
    <th>MedNLI</th>
    <th>i2b2 2006</th>
    <th>i2b2 2010</th>
    <th>i2b2 2014</th>
    <th>MNLI MM</th>
    <th>MNLI M</th>
  </tr>
  <tr>
    <td>BERT<sub>c</sub></td>
    <td>78.692</td>
    <td><b>94.031 ± 0.668</b></td>
    <td>83.710 ± 0.111</td>
    <td>92.234 ± 0.581</td>
    <td>83.452</td>
    <td>82.985</td>
  </tr>
  <tr>
    <td>BERT<sub>u</sub></td>
    <td>78.129</td>
    <td>90.846 ± 0.605</td>
    <td>85.077 ± 0.081</td>
    <td>93.318 ± 0.405</td>
    <td>83.208</td>
    <td>82.914</td>
  </tr>
  <tr>
    <td>BERT<sub>c</sub> + MIMIC 200k</td>
    <td>80.028</td>
    <td>91.660 ± 1.875</td>
    <td>87.197 ± 0.173</td>
    <td>92.192 ± 1.128</td>
    <td>82.618</td>
    <td>82.354</td>
  </tr>
  <tr>
    <td>BERT<sub>c</sub> + MIMIC 900k</td>
    <td>81.294</td>
    <td>91.324 ± 1.179</td>
    <td>87.812 ± 0.110</td>
    <td>92.881 ± 0.203</td>
    <td>81.906</td>
    <td>81.172</td>
  </tr>
  <tr>
    <td>BERT<sub>u</sub> + MIMIC 200k</td>
    <td>81.224</td>
    <td>91.499 ± 0.755</td>
    <td>88.042 ± 0.149</td>
    <td>93.747 ± 0.072</td>
    <td>83.411</td>
    <td>83.107</td>
  </tr>
  <tr>
    <td>BERT<sub>u</sub> + MIMIC 900k</td>
    <td><b>83.333</b></td>
    <td>92.533 ± 0.321</td>
    <td>87.969 ± 0.152</td>
    <td>93.530 ± 0.091</td>
    <td>82.567</td>
    <td>82.313</td>
  </tr>
  <tr>
    <td>BERT<sub>t</sub> + MIMIC 200k</td>
    <td>80.380</td>
    <td>91.273 ± 1.085</td>
    <td>87.688 ± 0.119</td>
    <td><b>93.901 ± 0.118</b></td>
    <td><b>83.767</b></td>
    <td><b>83.240</b></td>
  </tr>
  <tr>
    <td>BERT<sub>t</sub> + MIMIC 900k</td>
    <td>82.841</td>
    <td><b>92.354 ± 1.776</b></td>
    <td><b>88.192 ± 0.237</b></td>
    <td>93.723 ± 0.151</td>
    <td>82.577</td>
    <td>82.252</td>
  </tr>
</table>

Table 2: Downstream performance of all models on different downstream tasks. Scores are Accuracy for MedNLI and Exact F1 for all i2b2 tasks. The best model is selected based on the validation set. Scores within the standard deviation of the best score are in bold.

sequence length of 128 and a batch size of 64 followed by 100k steps with a sequence length of 512 and a batch size of 16. We denote these models as BERT<sub>c</sub> + MIMIC 200k and BERT<sub>u</sub> + MIMIC 200k, respectively. These initial experiments show the uncased BERT model converging slower than the cased model, so we also train both models for a total of 900k steps: for 100k steps with a sequence length of 128 and a batch size of 64 followed by 800k steps with a sequence length of 512 and a batch size of 16. With this extended training, all models converge to a masked language modeling loss of approximately 0.25. We denote these models as BERT<sub>c</sub> + MIMIC 900k and BERT<sub>t</sub> + MIMIC 900k. The uncased BERT models with independent case encoding are denoted as BERT<sub>t</sub> + MIMIC 200k and BERT<sub>t</sub> + MIMIC 900k. The BERT base uncased models with independent case embedding are denoted as BERT<sub>t</sub> + MIMIC 200k and BERT<sub>t</sub> + MIMIC 900k.

4.3 Downstream Tasks

Tasks  We evaluate the performance of the models on 4 downstream tasks in the clinical domain: MedNLI (Romanov and Shivade, 2018), the i2b2 2006 de-identification task (Uzuner et al., 2007), the i2b2 2010 concept extraction task (Uzuner et al., 2011), and the i2b2 2014 de-identification task (Stubbs and Özlem Uzuner, 2015; Stubbs et al., 2015; Kumar et al., 2015). We use the preprocessing scripts provided by Alsentzer et al. (2019) for the i2b2 tasks. We additionally evaluate the models on the Multi-Genre Natural Language Inference Corpus (MNLI; Williams et al., 2018).

For MedNLI, we use the standard training, validation, and test splits provided by Romanov and
Shivade (2018). For the i2b2 datasets, we split the training data into 90% for training and 10% for validation and use the standard test splits. For MNLI, we use the standard train and validation splits. We provide statistics for all datasets in Table 1.

Postprocessing We diverge slightly from Alsentzer et al. (2019) in i2b2 prediction postprocessing. All i2b2 tasks are trained using the IOB format and the exact F1 metrics are computed on predicted spans after post-processing. We utilize a simple post-processing scheme to limit predictions to valid sequences and select the prediction with highest probability out of each valid option. For example, given the invalid prediction “O I-problem,” we select the tag with the next highest probability, e.g., “O B-problem” or “O O” rather than always selecting “O B-problem.”

Finetuning For all tasks, we finetune the models using the same sweep of hyperparameters as Alsentzer et al. (2019): AdamW with a learning rate \( \in \{2 \cdot 10^{-5}, 3 \cdot 10^{-5}, 5 \cdot 10^{-5}\} \), a batch size \( \in \{16, 32\} \), and number of epochs \( \in \{3, 4, 5\} \). We use a sequence length of 150 for all tasks.

Metrics We report accuracy for MedNLI and MNLI and exact F1 for all i2b2 tasks. Exact F1 is computed as the number of exact matches between the predicted and true NER spans.

4.4 Computational Budget
All pretraining and finetuning experiments use a single A100 40GB GPU and 16 threads on an AMD EPYC 7543 CPU. Dataset preprocessing is CPU-intensive: upwards of 24 hours for MIMIC-III (not including downstream tasks). Pretraining on MIMIC-III takes approximately 4-6 hours per model to train the 200k models, plus 24 hours per model for the 900k models.

5 Results

Dataset Statistics As shown by Table 1, there is a trade-off between more information by maintaining casing and more efficiency in terms of shorter sequence lengths. The cased tokenizer requires more tokens than the uncased tokenizer, especially on clinical datasets such as MIMIC-III: While uncased tokenization yields 11.38% fewer tokens and 36.45% fewer sequences with length greater than 128 on MIMIC-III, on C4, uncased tokenization only yields 4.08-4.12% fewer tokens and 2.26% fewer documents with length greater than 128. This demonstrates the impact that improper casing and formatting in clinical notes has compared to generic open-domain datasets.

Note that the model with independent case encoding uses the uncased tokenizer and vocabulary.

Downstream Tasks The results for all downstream tasks are provided in Table 2. On nearly all tasks, an uncased or independently cased model performs best – even on NER, which is traditionally considered a case-sensitive task. The only exception to this rule is i2b2 2006, where BERT_c outperforms BERT_u before training on MIMIC-III.

The standard BERT models are generally worse than the models trained on MIMIC-III, but this is to be expected, as the standard BERT models are not trained on in-domain text. We are more interested in the relative performance of the cased and uncased models. In this case, we see that the uncased models have a more clear advantage once trained on MIMIC-III. For example, on MedNLI, BERT_c has a slightly higher accuracy of 78.692 than BERT_u which has 78.129. This advantage disappears once both models are trained on MIMIC-III, where the accuracy of BERT_c improves to 81.294 and BERT_u to 83.333. The independently cased model, BERT_i, had the second highest score on MedNLI at 82.841. Similar to Alsentzer et al. (2019), we find that the i2b2 2006 de-identification task is the most challenging task for models trained on MIMIC-III, likely due to the difference in formatting. However, out of the models trained on MIMIC-III, the uncased and independently cased model both outperform the cased model.

6 Analysis: Robustness of the Independent Case Encoding
To demonstrate the robustness of the independent case encoding, we construct 2 evaluation scenarios by transforming the evaluation dataset either to all upper-case or all lower-case. These transformations naturally do not affect the uncased model, but have a profound impact on the cased model. The upper part of Table 3 shows the difference in performance with respect to the original results for lower-cased data. We see a small but measurable impact across all tasks and a light increase in performance for the independently cased model on MedNLI and MNLI Mismatched.

However, the results on the upper-cased evaluation data in the bottom part of Table 3 are much more profound. Across all tasks, the cased model
<table>
  <tr>
    <th></th>
    <th>MedNLI</th>
    <th>i2b2 2006</th>
    <th>i2b2 2010</th>
    <th>i2b2 2014</th>
    <th>MNLI MM</th>
    <th>MNLI M</th>
  </tr>
  <tr>
    <td>BERT<sub>c</sub></td>
    <td>1.617</td>
    <td>19.341</td>
    <td>4.077</td>
    <td>14.358</td>
    <td>0.529</td>
    <td>0.632</td>
  </tr>
  <tr>
    <td>BERT<sub>c</sub> + MIMIC 200k</td>
    <td>0.985</td>
    <td>14.976</td>
    <td>2.128</td>
    <td>11.152</td>
    <td>0.498</td>
    <td>0.143</td>
  </tr>
  <tr>
    <td>BERT<sub>c</sub> + MIMIC 900k</td>
    <td>0.352</td>
    <td>13.688</td>
    <td>1.696</td>
    <td>10.565</td>
    <td>0.559</td>
    <td>0.000</td>
  </tr>
  <tr>
    <td>BERT<sub>c</sub> + MIMIC 200k</td>
    <td>-0.352</td>
    <td>4.113</td>
    <td>0.311</td>
    <td>1.468</td>
    <td>-0.081</td>
    <td>0.071</td>
  </tr>
  <tr>
    <td>BERT<sub>c</sub> + MIMIC 900k</td>
    <td>0.141</td>
    <td>6.257</td>
    <td>0.239</td>
    <td>1.592</td>
    <td>-0.081</td>
    <td>0.224</td>
  </tr>
  <tr>
    <td>BERT<sub>u</sub></td>
    <td>38.115</td>
    <td>10.774</td>
    <td>37.781</td>
    <td>19.899</td>
    <td>36.381</td>
    <td>37.779</td>
  </tr>
  <tr>
    <td>BERT<sub>u</sub> + MIMIC 200k</td>
    <td>33.896</td>
    <td>9.962</td>
    <td>23.092</td>
    <td>13.454</td>
    <td>32.242</td>
    <td>32.888</td>
  </tr>
  <tr>
    <td>BERT<sub>u</sub> + MIMIC 900k</td>
    <td>21.519</td>
    <td>9.833</td>
    <td>16.387</td>
    <td>12.484</td>
    <td>27.278</td>
    <td>27.224</td>
  </tr>
  <tr>
    <td>BERT<sub>u</sub> + MIMIC 200k</td>
    <td>-0.844</td>
    <td>1.293</td>
    <td>0.591</td>
    <td>0.799</td>
    <td>0.203</td>
    <td>0.245</td>
  </tr>
  <tr>
    <td>BERT<sub>u</sub> + MIMIC 900k</td>
    <td>0.070</td>
    <td>2.518</td>
    <td>0.228</td>
    <td>0.800</td>
    <td>-0.122</td>
    <td>0.234</td>
  </tr>
</table>

Table 3: Difference in performance on the lowercased (top) or uppercased (bottom) development set with respect to the original results. A positive number implies reduced performance. Smaller absolute values mean the model is more robust to the change in case. Uncased models are not shown as they are unaffected.

is impacted by 9.833-38.115 percentage points. In contrast, the independent case model is impacted by less than a percent across all tasks except for i2b2 2006, for which its performance is reduced by 1.29 and 2.52 for the 200k and 900k variants, respectively. Compared to the cased model, this demonstrates a much better balance of retaining case information where possible while ignoring it where not necessary.

7 Conclusion

Our paper provides an analysis of the impact of different casing strategies on clinical notes. We find that uncased models outperform cased models on downstream tasks while requiring fewer tokens. Our findings highlight the trade-off of case-sensitive tokenization, especially as it pertains to clinical data and other text that may not have proper casing. Finally, we introduce independent case encoding to better balance performance on case-sensitive and improperly-cased tasks.

Limitations

Language Models For language models, especially large open-domain language models such as GPT, cased tokenization is, for good reason, the default. This is justifiable because the model needs to produce text that will be read by humans — and humans prefer text with proper casing. There are also applications where preserving case is important, such as in programming languages, where case is semantically significant. Beyond the obvious cased vs. uncased conflicts if case was dropped (e.g., vector x vs. matrix X), case is regularly used to distinguish between functions, types, and variables and in Go is even used to distinguish between public and private. Conceptually, this makes sense – using text-to-speech for code is hardly as effective for code as text-to-speech is for prose.

Scope We intentionally limit the scope of our work to clinical data and, more specifically, to the datasets used by existing ClinicalBERT models. While all of the tasks we evaluated on showed strong performance of uncased tokenization this is in no way an exhaustive list of tasks. While our findings may transfer to other domains where improper casing negatively effects the performance of cased models, more experimentation would be required in those domains and we did not investigate any such tasks. However we hope this work motivates others to think more carefully about the decision of cased vs. uncased tokenization both in clinical applications and more generally.

References

Emily Alsentzer, John R. Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew B. A. McDermott. 2019. Publicly Available Clinical BERT Embeddings. Preprint, arxiv:1904.03323.

Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A Pretrained Language Model for Scientific Text. Preprint, arxiv:1903.10676.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. Preprint, arXiv:2005.14165.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Jonas Geiping and Tom Goldstein. 2022. Cramming: Training a language model on a single gpu in one day. Preprint, arXiv:2212.14034.

Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung-Kang Peng, and H Eugene Stanley. 2000. PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals. Circulation, 101(23):e215–e220.

Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing. Preprint, arxiv:2007.15779.

Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2020. ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission. Preprint, arxiv:1904.05342.

Alistair Johnson, Tom Pollard, and Roger Mark. 2015. MIMIC-III Clinical Database.

Alistair E.W. Johnson, Tom J. Pollard, Lu Shen, Liwei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. 2016. MIMIC-III, a freely accessible critical care database. Scientific Data, 3(1):160035.

Vishesh Kumar, Amber Stubbs, Stanley Shaw, and Özlem Uzuner. 2015. Creation of a new longitudinal corpus of clinical narratives. Journal of Biomedical Informatics, 58:S6–S10. Supplement: Proceedings of the 2014 i2b2/UTHealth Shared-Tasks and Workshop on Challenges in Natural Language Processing for Clinical Data.

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.

Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. BioBERT: A pre-trained biomedical language representation model for biomedical text mining. Bioinformatics (Oxford, England), 36(4):1234–1240.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv preprint, abs/1907.11692.

Jacob Portes, Alexander R Trott, Sam Havens, DANIEL KING, Abhinav Venigalla, Moin Nadeem, Nikhil Sardana, Daya Khudia, and Jonathan Frankle. 2023. MosaicBERT: A bidirectional encoder optimized for fast pretraining. In Thirty-seventh Conference on Neural Information Processing Systems.

Alec Radford and Karthik Narasimhan. 2018. Improving language understanding by generative pre-training.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Preprint, arxiv:1910.10683.

Alexey Romanov and Chaitanya Shivade. 2018. Lessons from natural language inference in the clinical domain. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1586–1596, Brussels, Belgium. Association for Computational Linguistics.

Amber Stubbs, Christopher Kotfila, and Özlem Uzuner. 2015. Automated systems for the de-identification of longitudinal clinical narratives: Overview of 2014 i2b2/uthealth shared task track 1. Journal of Biomedical Informatics, 58:S11–S19. Supplement: Proceedings of the 2014 i2b2/UTHealth Shared-Tasks and Workshop on Challenges in Natural Language Processing for Clinical Data.

Amber Stubbs and Özlem Uzuner. 2015. Annotating longitudinal clinical narratives for de-identification: The 2014 i2b2/uthealth corpus. Journal of Biomedical Informatics, 58:S20–S29. Supplement: Proceedings of the 2014 i2b2/UTHealth Shared-Tasks and Workshop on Challenges in Natural Language Processing for Clinical Data.

Özlem Uzuner, Yuan Luo, and Peter Szolovits. 2007. Evaluating the State-of-the-Art in Automatic De-identification. Journal of the American Medical Informatics Association, 14(5):550–563.

Özlem Uzuner, Brett R South, Shuying Shen, and Scott L DuVall. 2011. 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. Journal of the American Medical Informatics Association, 18(5):552–556.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. Preprint, arxiv:1706.03762.
Guangyu Wang, Xiaohong Liu, Zhen Ying, Guoxing Yang, Zhiwei Chen, Zhiwen Liu, Min Zhang, Hongmei Yan, Yuxing Lu, Yuanxu Gao, Kanmin Xue, Xiaoying Li, and Ying Chen. 2023. Optimized glycemic control of type 2 diabetes with reinforcement learning: a proof-of-concept trial.

Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.

Reviewer paper summaries: This paper investigates the impact of cased versus uncased tokenization on clinical notes, which often lack consistent or proper casing. Traditionally, cased tokenization has been seen as advantageous for tasks like named entity recognition (NER). However, the authors show that in the clinical domain—where text casing can be irregular—uncased tokenization not only improves downstream performance (including NER) but also reduces the number of tokens generated. They introduce an “independent case encoding” approach that encodes case information separately from the token itself. This strategy allows the model to leverage case signals when relevant, while still benefiting from the efficiency of uncased tokenization. Empirical results demonstrate that uncased and “independent case encoding” models outperform cased models across multiple clinical benchmarks (e.g., i2b2 tasks, MedNLI) and even general tasks (MNLI). The findings highlight that uncased tokenization—or case-agnostic approaches—are particularly effective for semi-structured text like clinical notes.

Authors investigate the effect of cased and uncased text on downstream task performance for clinical text. 
They experiment with Bert cased and uncased further pretraining it with MIMIC.

This paper investigates the impact of cased tokenization on clinical note text processing. The authors used MIMIC clinical notes with BERT model on different clinical downstream tasks. The results show uncased model achieves better results.
Summary of strengths: 1. The paper provides comprehensive experimental results on multiple clinical NLP tasks, showing consistent improvements with uncased tokenization.
2. The work addresses a real-world challenge in clinical text, which often has inconsistent casing. The authors’ conclusions have direct implications for model development in clinical settings.
3. By separating case embeddings from tokens, the approach offers a flexible middle ground that preserves case information without incurring the usual costs of cased tokenization.
4. The study points out that uncased tokenization can significantly reduce sequence length, leading to cost savings in terms of memory and computation.

independent case encoding is an interesting idea but there are no substantial benefits it seems.

1. The paper is in general well-written and easy to understand. The authors provide sufficient experimental detail which is helpful for reproducing the results.
2. The proposed independent case encoding though simple, achieves better performance and makes more clinical sense.
Summary of weaknesses: 1. Although the authors mention programming languages and certain acronyms, there is limited exploration of situations where casing is unequivocally crucial (e.g., drug brand names vs. generic names).
2. The paper could benefit from more detailed ablations showing how each aspect of independent case encoding contributes to performance improvements, especially on tasks heavily reliant on case distinctions.

the results do not show a very significant difference between the cased and uncased tokenizers. 
The difference is rather small. 
That uncased is better than cased models for clinical text was already observed from Gu et al. (Pubmedbert 2021), where they state that the uncased version performs more or less the same and therefore they choose the uncased version.

1. The work has limited novelty and experiments. it would be better if the author can try different model archetecture, like encoder-based, decoder-based, encoder-decoder-based, to show whether the approach can improve through different models and tasks. Moreover, the independent case encoding approach is conceptually similar to existing token type embeddings. The 4-state case encoding scheme appears arbitrary without empirical justification. The modification to BERT is relatively straightforward and doesn't introduce fundamental architectural innovations. The approach might not be easily generalizable to more advanced transformer architectures
2. How does the medical abbreviations benefits from the proposed uncased setting? The paper lacks sufficiently address how uncased tokenization affects medical abbreviation disambiguation (e.g., "CA" could mean "cancer" or "California"). How the model handles case-sensitive medical acronyms (e.g., "HF" for Heart Failure vs "hf" for "high-frequency")?
3. Only tested on MIMIC-III dataset, raising questions about generalizability to other clinical text sources. Lack of comparison with alternative approaches to handling case information (e.g., special tokens, hierarchical encoding). No exploration of dynamic or learned case encoding strategies. Limited analysis of computational overhead introduced by the additional embedding layer.
Comments, suggestions and typos: Comments and Suggestions
1. Deeper Ablation Studies: Provide a breakdown of how each component in the independent case encoding affects performance. Are certain types of case embeddings (e.g., capitalized vs. uppercase) more critical?
2. Additional Use Cases: Evaluate tasks where casing might be especially important, such as handling brand names, abbreviations that overlap with English words, or eponymous medical terms.
3. Comparisons with Other Tokenization Schemes: Consider subword or byte-level tokenization approaches to see if they might be more robust or efficient in handling irregular casing.

Typos 
1. Line 034: “ambiguties” should be “ambiguities.”
2. Line 109: “prelimenary” should be “preliminary.”
3. Line 135: “capitlized” should be “capitalized.”

page 4 "independentlycased" needs whitespace here. 
your work reminds me of that paper although they investigate the tokenization of mbert in different languages and their effect on downstream tasks https://aclanthology.org/2021.acl-long.243.pdf

Maybe you can test it on more tasks. 
You could e.g. take PubMedBert and compare it to PubMedBert with Independent encoding on all  the BLUE Benchmark or sth. similar.

see comments above
Overall assessment scores: 4

2.5

2