From Characters to Words: Hierarchical Pre-trained Language Model for Open-vocabulary Language Understanding
Year: 2023

Abstract:
Current state-of-the-art models for natural language understanding require a preprocessing step to convert raw text into discrete tokens. This process known as tokenization relies on a pre-built vocabulary of words or sub-word morphemes. This fixed vocabulary limits the model’s robustness to spelling errors and its capacity to adapt to new domains. In this work, we introduce a novel open-vocabulary language model that adopts a hierarchical two-level approach: one at the word level and another at the sequence level. Concretely, we design an intraword module that uses a shallow Transformer architecture to learn word representations from their characters, and a deep inter-word Transformer module that contextualizes each word representation by attending to the entire word sequence. Our model thus directly operates on character sequences with explicit awareness of word boundaries, but without biased sub-word or word-level vocabulary. Experiments on various downstream tasks show that our method outperforms strong baselines. We also demonstrate that our hierarchical model is robust to textual corruption and domain shift.

heading: 1 Introduction
text: Pre-trained language models with Transformers have achieved breakthroughs in many natural language processing (NLP) tasks (Devlin et al., 2019; Liu et al., 2019). One of the key advantages of Transformers over traditional feature-engineered NLP pipelines is that Transformers enable endto-end training from vast amount of data to automatically learn the optimal language representation (Mikolov et al., 2013b). However, most recent language models still require a separate preprocessing stage known as tokenization. Tokenization is a process that splits raw text parts into a list of discrete tokens from a fixed vocabulary. This
†Work done during internship at Microsoft. PyTorch-like pseudocode is available in Appendix A.3.
pre-defined vocabulary remains as an important bottleneck preventing truly end-to-end training of language models (Tay et al., 2021; Islam et al., 2022).
Based on the granularity of the basic token units, tokenization methods can be divided into three categories: character-based, subword-based and wordbased. A word-based tokenizer segments sentence into smaller chunks of words. Due to language complexity and memory limit, a word-based vocabulary can not represent all possible words. Wordlevel tokenization thus frequently runs into the issue of out-of-vocabulary words. A character-based tokenizer simply splits the text into a sequence of its characters. It is flexible to encode arbitrary words, but character-level tokenization produces long sequences, which is undesirable as the computational cost of Transformers grows quadratically with the sequence length. To strike a good balance between time and space complexity, most stateof-the-art pre-trained language models thus adopt sub-word tokenization. Data-driven sub-word tokenizers (Kudo and Richardson, 2018; Schuster and Nakajima, 2012; Kudo, 2018) are typically pre-trained on a general text corpus to learn a subword vocabulary based on the frequency of word fragments.
Despite their popularity, sub-word tokenizers
3605
limit the robustness and generalizability of the language models built upon them. First, sub-word tokenizers are sensitive to small textual perturbations (Xue et al., 2022). While humans can still comprehend text with subtle misspellings and capitalization variants (Rawlinson, 2007; Davis, 2003), these perturbations can drastically change the tokenization results, potentially leading to a suboptimal text representation. Second, the sub-word vocabulary is pre-built and remains frozen during the language model pre-training and task-specific fine-tuning. Therefore, when adapting a pre-trained language model into a new language context (e.g. biomedical texts and business documents), the tokenizer is prone to excessive fragmentation of subword pieces (Yasunaga et al., 2022; Islam et al., 2022), as illustrated in Table 1. While this issue could be partially remedied by further task-specific pre-training or by collecting more fine-tuning data, such mitigation would be costly and not always possible.
We aim to bring the best of both character-based and word-based models to address the challenges discussed above. To this end, we propose a novel pre-trained language model with a hierarchical twolevel architecture. At the word level, we split the text sequence by characters, and introduce an intraword module that uses Transformers to learn a representation for each word in the sequence from the embeddings of their respective characters. At the sequence level, we introduce an inter-word module that contextualizes the embedding for every words in the text sequence. Our method does not require explicit sub-word or word-level vocabulary, and can thus be considered as an open-vocabulary approach (Mielke et al., 2021). By limiting the attention range to characters within the same word rather than the full sequence in the intra-word module, our model remains computationally efficient.
In order to validate our model, we comprehensively compare our method with various baseline methods, including the most popular sub-word based model BERT (Devlin et al., 2019), some state-of-the-art character-based models (Clark et al., 2022a; Boukkouri et al., 2020), and an hybrid character/sub-word model (Ma et al., 2020). Besides standard benchmarking, we also test the robustness of the various models in two ways: by introducing spelling noise into the validation set and by testing on cross-domain tasks.
Our contributions can be summarized as follows:
• We introduce a novel open-vocabulary pretrained language model with a hierarchical two-level architecture. Our method does not rely on pre-defined word or sub-word vocabulary.
• We propose a novel adaptive and learnable aggregation method to summarize characterlevel features into word-level representations. An ablation study highlights its effectiveness.
• We show that our method outperforms strong baselines on multiple benchmarking datasets, while being computationally efficient.
• We perform quantitative experiments and a case study to show that our model is robust to textual corruption and domain shift.

heading: 2 Related Work

heading: 2.1 Word-level Models
text: Word embedding methods including Word2vec (Mikolov et al., 2013a) and GloVe (Pennington et al., 2014) led to many early NLP breakthroughs. These methods learn vector space representations of words from large-scale unlabeled corpora, and encode semantic relationships and meanings (Goldberg and Levy, 2014). In order to generalize to rare words, Bhatia et al. (2016) proposed to use LSTM to learn word embedding from both morphological structure and word distribution. While early methods only learned a context-independent word representation, ELMo (Peters et al., 2018) proposed to use a deep bidirectional language model to learn contextualized word representations. In more recent studies, Transformer-XL (Dai et al., 2019) enhanced the Transformer architecture with a recurrence mechanism to learn contextualized word embedding through language modeling. Despite the recent progress, word-level models still face the out-of-vocabulary challenge for noisy text and non-canonical word forms (Eisenstein, 2013).

heading: 2.2 Character-level Models
text: Character-level language models emerged in the early years thanks to their simplicity and ability to better address out-of-vocabulary words compared to word-level models (Elman, 1990; Graves, 2013; Kalchbrenner et al., 2016). While sub-word based
approaches gained popularity in language modeling due to their superior performance, recent studies (Choe et al., 2019; Xue et al., 2022) show that character/byte-level models can match the performance of their sub-word counterpart when provided with sufficient parameter capacity. In addition, character-level models have been shown to be more robust to text corruptions (Tay et al., 2021), adversarial attacks, and domain shifts (Aguilar et al., 2020).
Character-level models also show promising results in multilingual settings. While sub-word or word tokenizers require a huge vocabulary to adequately cover various languages, multilingual character-based vocabulary can remain comprehensive and small. The text embedding layer does not eat up most of the model’s parameter budget as in the multilingual BERTBase model for instance (up to 52%). More parameters can then be dedicated to the Transformer layers in characterbased approaches. Character-level models have also been shown to perform better on low-resource languages (Islam et al., 2022).
An important drawback of character-level models is that they typically require more computations than sub-word and word-level models. This is because character-level tokenization produces longer token sequences compared to sub-word or word based approaches, and the computational and memory demands of the self-attention mechanism grow quadratically with the sequence length. In order to address this challenge, CANINE (Clark et al., 2022b) leverages strided convolution to downsample the character sequence, while Charformer (Tay et al., 2021) uses average pooling. Although these methods improve the computational efficiency of character-level models, they require a predefined static downsampling rate. Such downsampling operation often breaks the boundary of basic linguistic units, including morphemes and words.

heading: 2.3 Hybrid Models
text: Vanilla character-level models do not explicitly extract word or sub-word morpheme representations, which might negatively impact their performance on word-level downstream tasks, including namedentity recognition and extractive question answering. In order to address this issue, there have been efforts to combine character-level and word/subword level approaches to build hybrid models. These works propose to use information from char-
acter spelling to inform word representation. For example, Flair (Akbik et al., 2018) proposed to use the internal states of a pre-trained character language model to produce word-level embeddings. CharBERT (Ma et al., 2020) combined sub-word tokens and character tokens and fused their heterogeneous representations. CharacterBERT (Boukkouri et al., 2020) used a CNN to learn word-level representations from the embeddings of their characters, but still requires a word-level vocabulary for pretraining. Char2Subword (Aguilar et al., 2020) proposed a similar approach, where character embeddings are used to mimic pre-trained representation of sub-word tokens with Transformer encoder.

heading: 3 Method
text: Most character-level Transformer encoder models are sub-optimal for two reasons: (1) Dense self-attention on long character sequence is computationally expensive; (2) They do not leverage word boundary, which is an important inductive bias in linguistics. To overcome these challenges, we propose to decompose dense character-level Transformer encoder into two parts: intra-word Transformer encoder and inter-word Transformer encoder. Our hierarchical language model (HLM) adopts an hourglass structure (Nawrot et al., 2022) and contains three main components: (1) an intraword module that learns word embeddings from their characters; (2) an inter-word module which contextualizes the word representations by attending to all words in the input sequence; (3) an intra-word prediction head for character-level pretraining. The overall architecture of our model is shown in Fig. 1. In the following sections, we discuss each component separately.

heading: 3.1 Intra-word Module
text: We aim to learn word-level representations from the embeddings of their characters. An ideal approach should be able to handle words of arbitrary lengths, attend to every character rather than a local window, and remain computationally efficient. Therefore, we choose a shallow (4 layers in our experiments) Transformer encoder to learn contextualized character embeddings, rather than a CNN or a LSTM used by previous methods (Boukkouri et al., 2020; Peters et al., 2018). Either average or max pooling (Boukkouri et al., 2020; Clark et al., 2022b) is commonly used to aggregate contextualized character embeddings and thus reduce the
sequence length. However, such simple pooling tends to wash out strong signals from particular morphemes (Fathi and Maleki Shoja, 2018). To address this challenge, we propose a novel adaptive and learnable aggregation method. Inspired by the approach of using the hidden state of the [CLS] token as the aggregate sequence-level representation, we insert a special [WORD_CLS] token at the beginning of every word. The embeddings of the [WORD_CLS] tokens are then used as wordlevel representations. Formally, for the i-th word of Ci characters in the sequence, we extract its word-level representation ri as:
hi = fθ(e i 0 ⊕ ei1 ⊕ . . .⊕ eiCi) ri = hi0,
where fθ is the intra-word Transformers that produces a contextualized representation hi for each character of the i-th word, ei0 is the embedding of the special [WORD_CLS] token, eic is the c-th character embedding of the i-th word, and ⊕ denotes concatenation along the sequence dimension.
In Sec. 4.4, we conduct an ablation study to show that the proposed aggregation method outperforms the standard average or max pooling. By aggregating character-level tokens into word-level tokens, the token sequence length is greatly reduced for the subsequent inter-word module.

heading: 3.2 Inter-word Module
text: After obtaining word-level features, we apply an inter-word module consisting of deep transformer encoder layers to extract contextualized word-level representation by attending to all words in the sequences. Formally, the contextualized representation wi of the i-th word of the sequence of N words is given as:
wi = fϕ(r 0 ⊕ . . .⊕ rN−1),
where fϕ denotes the inter-word Transformers. We set the depth of the inter-word Transformer encoder stack to 12 in order to match the settings of BERTBase (Devlin et al., 2019) and CANINE (Clark et al., 2022b). The inter-word module contributes the most to the total model parameters.

heading: 3.3 Intra-word Prediction Head
text: Since we adopt an open-vocabulary approach, we propose to use character-level masked language modeling as pre-training task. To restore the character-level token sequence, we concatenate the contextualized character representations from the intra-word module (the initial [WORD_CLS] token is omitted) with the word-level features from the inter-word module along the sequence dimension. Finally, we apply a lightweight intra-word prediction head to get the posterior token probabilities.
Formally, the prediction of the Ci characters from the i-th word are given by:
ci = fσ(w i ⊕ hi1 ⊕ . . .⊕ hiCi),
where fσ is the intra-word prediction head, consisting of a single Transformer layer, a fully-connected layer and a Softmax layer. Note that the intra-word prediction head is only used during pre-training for the masked character modeling task. During downstream fine-tuning, similar to CANINE, we concatenate initial word embedding ri and contextualized word representation wi along the feature dimension, and subsequently employ a small feedforward network to integrate both low-level and high-level information for prediction.

heading: 3.4 Pre-training Task
text: Following the practice of BERT, we pre-train our model on English Wikipedia and BookCorpus dataset (19G) (Zhu et al., 2015). We pre-train the model for 3 epochs (3.9M steps with batch size set as 16) on a server with 8 NVIDIA Tesla V100 GPUs, and each epoch takes 137 hours. We adopt whole-word masked language modeling as pre-training task. In detail, we randomly select 15% of words from the input sequence, and mask every characters in the selected word. We replace the character tokens in 80% of the selected masked word with the [MASK] token. For 10% of the selected masked words, we replace their characters with randomly selected characters drawn from our character vocabulary. The remaining 10% words are unchanged. The three main components of our model are jointly trained in end-to-end fashion.

heading: 3.5 Implementation Details
text: We use spaCy (Honnibal et al., 2020) to split sentences into words, which is rule-based using space, punctuation and special rules (e.g. splitting don’t into do and n’t). We use a case-sensitive character vocabulary of size 1024, which consists of letters, digits and symbols. The maximum sequence length is set to 20 characters for the intra-word module and 512 words for the inter-word module. A [CLS] and a [SEP] token are inserted at the beginning and end of each sequence respectively. The hidden size is set to 768, the number of attention heads is set to 12, the feed-forward dimension in the Transformer encoder is set as 1536 and 3072 for intra-word and inter-word modules respectively. We leverage relative position (He
et al., 2021) in our model, and we do not use token type embedding. GELU (Hendrycks and Gimpel, 2016) is used as activation function. Our model contains 125M parameters. We use the AdamW optimizer (Loshchilov and Hutter, 2018) for model pre-training and fine-tuning. For the pre-training, the weight decay is set to 0.01 and the number of warmup steps is set to 10,000. A linear learning rate decay schedule is used, starting at 5e-5. The dropout rate is set to 0.1. More algorithm details can be found in Appendix A.3.

heading: 4 Experiments
text: We evaluate the performance of our pre-trained model on a wide range of downstream tasks. We compare the performance of our pre-trained hierarchical language model (HLM) with various baseline methods, including the popular sub-word based BERT model, three recent byte/characterlevel models, as well as a hybrid model referred to as CharacterBERT. For BERT, we use the cased BERTBase model (108M parameters) to match our inter-word Transformers module setup. For CANINE, we adopt CANINE-C (132M) which also uses a character-level pre-training task. For CharacterBERT, we use the general version (105M) which is pre-trained on English Wikipedia and OpenWebText. For those baseline models, we use the pretrained weights hosted on Huggingface† or released by the authors. For Charformer (203M) and Bytelevel T5 (200M), we use results of the base version from the original paper as pre-trained weight is not available.

heading: 4.1 Evaluation on Standard Benchmarks
text: In order to assess our model’s performance on general domain, we evaluate our methods on standard English NLP benchmarks, including Stanford Question Answering Dataset (SQuAD) task (Rajpurkar et al., 2016, 2018) and GLUE tasks (Wang et al., 2018). For the SQuAD task, we benchmark on both SQuAD 1.1 and 2.0 versions. SQuAD 1.1 dataset contains 100,000+ questions with associated context documents, and every question is answerable given the context. SQuAD 2.0 dataset contains an additional 50,000 unanswerable questions. We fine-tune the models for 2 epochs with a batch size of 16, and a learning rate of 3e-5. The evaluation on the validation set is shown in Table 2 (left). We use exact match (EM) and F1 scores as
†https://huggingface.co/models
the two evaluation metrics. Our method outperforms all the baseline methods on both SQuAD versions.
We also benchmark our model on three text classification tasks from the widely adopted GLUE tasks (Wang et al., 2018), including MNLI (Williams et al., 2018), MRPC (Dolan and Brockett, 2005) and QNLI (Rajpurkar et al., 2016). The MNLI dataset contains 393k training samples with textual entailment annotations. Given a sentence pair containing a premise and an hypothesis, the task is to predict whether the premise entails the hypothesis, contradicts the hypothesis, or neither. We conduct evaluation in both matched and mismatched settings. The MRPC dataset contains 3.7k of training sentence pairs, and the task is to predict whether the two sentences are semantically equivalent. The QNLI dataset contains 108k training samples of question-paragraph pairs, and the task is to predict whether the context sentence contains the answer to the question. We fine-tune the models on the datasets described above for 5 epochs, with a batch size of 16, and a learning rate of 2e-5. We use the accuracy as the evaluation metric. As shown in Table 2, our proposed method outperforms the baseline methods on all tasks.
In order to investigate the model’s performance when the size is scaled up, we increase the size of our HLM to match BERTLarge and benchmark the performance. Our HLMLarge performs competitively with the baseline method given a limited computational budget. More details and preliminary experimental results can be found in Appendix A.2.

heading: 4.2 Robustness to Textual Corruptions
text: Humans are prone to making spelling mistakes. For example, 10-15% of web search queries contain misspellings (Dalianis, 2002; Cucerzan and Brill,
†Use results from Tay et al. (2021)
2004). In order to test our model’s robustness to misspellings, we add synthetic noise to the finetuning and evaluation set of downstream tasks and re-evaluate all the models.
Following the practice of Xue et al. (2022), we experiment with three types of noises: (1) Random drop: We randomly delete 10% of characters (spaces and punctuation are included) from the input sequence; (2) Random repeat: We randomly select 20% of characters, then append 1-3 repetitions (with equal probability) after the the selected original characters; (3) Random case: We randomly set the case for each character (upper or lower) in the input sequence.
We perform the perturbation experiments on two representative downstream tasks: text classification on MNLI dataset and question answering on SQuAD 2.0. For the MNLI dataset, we add noise to both premise and hypothesis sentences. For the SQuAD 2.0 dataset, we only apply the perturbations to the question sentence, but not to the context paragraph, in order to avoid copying corrupted answer from the context for extractive QA mod-
els. The evaluation results are shown in Table 3. We found that BERT’s performance significantly drops under perturbation, one explanation being that even subtle misspellings would greatly change the sub-word tokenization results. In comparison, character-level models including CANINE degrade less in the presence of noise. We also present the results for unseen perturbation setting in Appendix A.4. Overall, our proposed HLM is robust to different kinds of perturbation and achieves the best performance.
In order to access the model’s robustness to various magnitude of perturbations, we add different amounts of noise to the QNLI dataset and perform the evaluation. In practice, we randomly sample 5%, 10%, 15%, 20% of characters for each example in the finetuning data and validation set. For each selected character, we either drop the character or repeat the character as mentioned above (equal probability). The accuracy on the validation set is shown in Fig. 2.

heading: 4.3 Robustness to Domain Shift
text: Most generic language models are pre-trained on web-crawled text corpora including Wikipedia and Common Crawl. But in real world deployments, models are often used in a different domain, an issue referred to as domain shift. In order to evaluate the robustness to domain shift, we finetune and evaluate the pre-trained models on downstream tasks from specialized domains including biomedicine and social media. For the biomedical field, we perform the evaluation on the NCBIdisease dataset (Crichton et al., 2017; Gu et al.,
2021), which contains 7,287 sentences annotated with disease mentions from PubMed abstracts. The task is framed as a named entity recognition (NER) problem where the entities are the disease mentions. We fine-tune the models for 20 epochs, with a batch size of 16, and a learning rate of 2e-5. For the social media experiment, we leverage the W-NUT16 NER shared task (Strauss et al., 2016). This dataset contains 7,244 tweets annotated with 10 NER categories, including person, location, company and others. We fine-tune the models for 5 epochs. The evaluation results on the test sets are shown in Table 4. We use the F1 score as the evaluation metric. As observed, the proposed HLM outperforms the baseline methods, highlighting its higher robustness to domain shift.
Case study In order to understand the performance gain of our model over sub-word based BERT on cross-domain tasks, we look into the cases where BERT makes incorrect predictions. We found that many of these cases contain excessively fragmented words. Table 5 shows two examples from the NCBI-disease NER task. The word fragility in case 1 is segmented into f, ##rag, ##ility, and the word rupture in case 2 is segmented into r, ##up, ##ture. We think these tokenization results
are sub-optimal as they break word morphemes, which possibly explains BERT’s mispredictions. In comparison, we use BertViz (Vig, 2019) to visualize the behavior of our HLM model. Specifically, we visualize the attention patterns of the [WORD_CLS] token of the last Transformer layer of our intra-word module. As shown in Fig. 3, the [WORD_CLS] token for the word fragility and rupture are primarily attended by the character string fragil and rupt respectively, which are the stems of the words.

heading: 4.4 Ablation Study
text: In this section, we perform an ablation study to compare the effect of different word-level aggregation methods. Specifically, we replace the proposed special token learning-based aggregation with standard aggregation methods such as average pooling and max pooling. We did not implement the strided convolution proposed in CANINE as it can not handle the variable word lengths. We report the validation accuracy on MRPC and the test F1 score on NCBI-disease in Table 6. Our learned aggregation outperforms the standard pooling strategies. Note that average and max pooling are usually performed on a fixed-length window of characters in previous studies (Tay et al., 2021), not adaptively at the word-level as in our ablation study.

heading: 4.5 Computational Efficiency
text: In this section, we benchmark the computational efficiency of the proposed model. Specifically, we measure the inference throughput (number of processed samples per second) on the test set of the MRPC dataset, a sub-task of the GLUE benchmark.
We evaluate different models on the same server with one NVIDIA Tesla V100 GPU. The batch size is set to 32 and we use single precision. The evaluation results are shown in Table 7. While BERT is the most computationally efficient, our HLM also performs competitively, the performance gap being smaller compared to other character-level baseline models. We speculate that this performance gain comes from our hierarchical architecture. By aggregating character tokens into word-level tokens, the sequence length is drastically reduced for the inter-word module which has the deepest Transformer stack. We provide more analysis on the computational complexity in Appendix A.1.

heading: 5 Conclusion
text: In this work, we propose a novel hierarchical language model for open-vocabulary language understanding. Our method does not rely on explicit sub-word or word vocabulary. We demonstrate that our HLM model outperforms baseline methods on standard benchmarks, and highlight its robustness to spelling errors and domain shifts. In future work, we will expand our language support and explore incorporating a decoder for generative tasks.
Limitations
This work has two main limitations. First, we only consider baseline models with similar amount of parameters, and pre-trained on similar scale of text corpus for comparison. While we are aware of recent models including T5 (Raffel et al., 2020) and PaLM (Chowdhery et al., 2022), they either use huge corpus like C4 (745GB text) for pre-training or contain significantly more parameters than ours. In the future, we will try to find additional computational resources to scale up our model and pre-train on larger text corpus. Second, we leverage spaCy to segment sentences into words, which is rule-based using spaces, punctuations and other rules. This approach works well on English and many other common languages such as French, German and Spanish. But for a few languages that do not use spaces to split words (e.g. Chinese and Japanese), it will be challenging to retrieve word boundaries. To address this issue, we consider either falling back to character splitting for these languages (similar to multilingual BERT) or employing a more sophisticated word boundary detector in future work.

heading: Acknowledgements
text: This work was partially supported by NIH Award Number 1R01HL141813-01 and the Pennsylvania Department of Health. We are grateful for the computational resources provided by Pittsburgh Super Computing grant number TGASC170024.

heading: A Appendix
text: A.1 Analysis on Computational Complexity Let N denotes the character length of input sequence. Without loss of generality, we assume the words in the sequence are of the same length M . The multi-head self-attention module is the major component of Transformer. While it provides global receptive field, the computational cost and memory footprint grow quadratically with input sequence length (Zeng et al., 2021). Therefore, for a vanilla character-based Transformers with dense self-attention, the computational and space complexity is O(N2).
For our proposed HLM, the input sequence is still at the character level. But we sparsity the dense self-attention by introducing a hierarchical architecture. For the intra-word module, each character token only attends to characters from the same word. Since there are NM words in the sequence, the computational and space complexity of the intra-word module is
O
( N
M ·M2
) = O(NM) (1)
For the inter-word module, since it only operates on word-level tokens, the computational and space complexity is
O
( N2
M2
) (2)
Since typically N ≫ M , and we have a shallow intra-word module and a deeper inter-word module, Eq. 2 dominates the computational and space complexity of the full model, which is significantly lower than the vanilla character-level model.
In comparison to sub-word based models like BERT, our inter-word module operates on wordlevel token sequence, which is always equal or shorter than sub-word level token sequence. Therefore, despite our model has an extra intra-word module, we empirically observe in Table 7 that our HLM is competitive in terms of computational efficiency comparing to sub-word based models.
A.2 Preliminary Evaluation of Scaled Model In this section, we scale up our model size and benchmark the performance. In order to match BERTLarge, we set the number of layers in the inter-word Transformer encoder to 24 and the feedforward dimension of Transformer encoder is set as 2048 and 4096 for intra-word and inter-word modules respectively. We set the number of attention
heads as 16 and the hidden size as 1024. The batch size is set as 128. Other hyperparameters are set as the same as HLMBase, described in Section 3. Due to limited access to computational resources, we could only pre-train the model for 370k steps at the camera-ready deadline. In comparison, BERTLarge was pretrained for 1M steps with a batch size of 256. Therefore, our computational budget is about 1/6 of BERT’s. We benchmark our model’s performance on SQuAD 1.1 and 2.0 datasets. The evaluation results on the validation set are shown in Table 8. We use exact match (EM) and F1 scores as the two evaluation metrics. Our model performs competitively compared with BERTLarge, despite that our HLMLarge has significantly less computational budget for pre-training.
A.3 Algorithm Details
In this section, we provide algorithm details for our input pre-processing and model algorithm. Our pre-processing consists the following steps. First, we split each sentence into a list of words. Next, we map characters to codepoint indexes using a character-level vocabulary, and insert [WORD_CLS] token at the start of each word. Next, we insert a [CLS] token at the start, and a [SEP] token at the end for each sequence. Then we truncate the token sequence based on both character-level (20 characters for each word) and word-level (512 words per sentence) limits. Next, we compute the maximum number of characters for words in the batch, and pad all words to this length. We also determine the maximum number of words in the sequence batch, and pad all sequences to this length. The pre-processed batch can then be represented as a matrix of shape [batch_size, max_num_word, max_num_char]. Our unique representation of text batch enables us to efficiently switch between performing intraword self-attention and inter-word self-attention by simply reshaping, which is shown in Algorithm 1.
We provide pseudocode for pre-training of our HLM in Algorithm 1. For better readability, we
omit implementation details including utilizing attention mask which avoids performing attention on the [PAD] tokens and handling for padding words. We recommend padding the input matrix to multiples of 8 for better acceleration on GPU. We also found a residual connection between initial word embedding ri and contextualized word embedding wi improves the performance in a subsequent study.
A.4 Robustness to Unseen Perturbations In this section, we benchmark the model’s robustness to unseen noise. Specifically, we only add noise to the evaluation set, while using the original fine-tuning data. We experiment with three types of perturbation as introduced in Section 4.2. The results are shown in Table 9. In all three scenarios, our proposed HLM outperforms baseline methods, showing better robustness.
Algorithm 1 Pseudocode for HLM, PyTorch-like
# embeddings: character-level embedding lookup table # intra_word_encoder: Intra-word Transformer encoder # inter_word_encoder: Inter-word Transformer encoder # intra_word_head: Intra-word prediction head
for input_ids, labels in loader: # load a minibatch with n samples input_embeds = embeddings(input_ids) batch_size, num_word, num_char, hidden_size = input_embeds.shape
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
We discuss the limitations in Section 7.
7 A2. Did you discuss any potential risks of your work? We think there is a minimal potential risk in our work, since our model is not a generative language model.
3 A3. Do the abstract and introduction summarize the paper’s main claims? We summarize the main claims in Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Not applicable. Left blank.
C 3 Did you run computational experiments? In section 4.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? In section 3.5.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? In section 3.5.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? In section 4.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? In section 3.5.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: In future work, we will expand our language support and explore incorporating a decoder for generative tasks. Limitations
This work has two main limitations. First, we only consider baseline models with similar amount of parameters, and pre-trained on similar scale of text corpus for comparison. While we are aware of recent models including T5 (Raffel et al. , 2020) and PaLM (Chowdhery et al. , 2022), they either use huge corpus like C4 (745GB text) for pre-training or contain significantly more parameters than ours. In the future, we will try to find additional computational resources to scale up our model and pre-train on larger text corpus. Second, we leverage spaCy to segment sentences into words, which is rule-based using spaces, punctuations and other rules. This approach works well on English and many other common languages such as French, German and Spanish. But for a few languages that do not use spaces to split words (e. g. Chinese and Japanese), it will be challenging to retrieve word boundaries. To address this issue, we consider either falling back to character splitting for these languages (similar to multilingual BERT) or employing a more sophisticated word boundary detector in future work