## 2 Literature Review
### Domain-Specific BERT Models  
Different works already showed that BERT models' performances on many natural language processing tasks, including intent recognition, drop dramatically on held-out data when a significant percentage of out-of-vocabulary (OOV) words do not appear in the training data [22; 23].  
For instance, in [24], [PERSON] et al. investigated two different approaches aimed at enlarging the vocabulary size when fine-tuning a pretrained multilingual BERT model on a variety of NLP tasks, thus addressing the OOV problem: joint mapping and mixture mapping. Their experimental results show that using mixture mapping is more promising.  
Alternatively, [PERSON] et al. suggested a different approach for domain adaptation [25]. Here, authors propose to train Word2 Vec on target-domain text and align the resulting word vectors with the wordpiece vectors of a general-domain pretrained language model (PTLM). Authors claim their method is cheaper compared to that of successful but expensive unsupervised pretraining on target-domain text. Besides testing their approach on several biomedical named entity recognition (NER) tasks, they also proved their method to be effective when adapting an existing general-domain question answering (QA) model to an emerging domain: the COVID-19 pandemic.  
Similarly, the work authored by [PERSON] et al. [26] presents different biomedical and clinical transformer-based pretrained language models for Spanish and analyzes the impact of the model's vocabulary on the NER performances. Authors experimented with different pretraining choices, such as masking at word and subword level, varying the vocabulary size and testing with domain data, looking for better language representations. Mixed-domain pretraining and cross-domain transfer approaches were confirmed as a valid alternative to generate suitable target models in the absence of enough data to train a model from scratch. Their results also confirmed that domain-specific pretraining is fundamental to achieving higher performances in downstream NER tasks, even within a midresource scenario.  
[PERSON] et al. recently introduced exBERT [27], a training method to extend BERT pretrained models from a general domain to a specific domain with a new additive vocabulary under constrained training resources (i.e., constrained computation and data). exBERT uses a small extension module to learn to adapt an augmenting embedding for the new domain in the context of the original BERT's embedding of a general vocabulary. Authors demonstrate that exBERT consistently outperforms prior approaches when using limited corpus and pretraining computation resources.  
In [28], [PERSON] et al. evaluate different methods for extending an Indonesian BERT model with additive domain-specific vocabulary. They specifically focused on efficient model adaptation under vocabulary mismatch and benchmarked different ways of initializing the BERT embedding layer for new word types. The work suggests that initializing with the average BERT subword embedding may be more effective than other initialization methods and help to make training five times faster.  
Finally, although recently proposed methods such as vocabulary expansion [25], vocabulary expansion, and continual pretraining [27] overcome the issue of OOV words, these methods also increase the size of vocabulary due to the addition of new terms in the vocabulary. In this regard, [PERSON] et al. recently proposed the Adapt-and-Distill approach [29] to adapt general models to a specific domain using vocabulary expansion and knowledge distillation. Different from existing adaptation methods, this approach not only adapts general models to specific domain but also reduces the size of the model.