## 1 Introduction  
Pre-trained large language models (LLMs) are models that are trained using a large amount of text data from our daily life, allowing the models to learn the occurrence probabilities of words or characters within this text [1]. As a result, LLMs can model the distributions present in these text data. The training data for LLMs consists of textual contexts, and these models can be trained with virtually unlimited amounts of text, enabling LLMs to possess powerful text processing abilities and exhibit excellent performance in various downstream tasks. Early LLMs mainly focused on learning word embeddings, but these embeddings only vectorize words without having practical effects in downstream tasks. Therefore, training models with word embeddings often results in poor prediction accuracy, as observed in models such as Skip-gram [2] and GloVe [3]. Moreover, these models fail to capture long-range contextual relationships. Starting in 2016, the main development trends for LLMs have included establishing relationships between long-distance texts and pre-trained models on large-scale corpora. [PERSON] et al. used language modeling and sequence autoencoders to improve the sequence learning of recurrent neural networks (RNNs), marking the beginning of modern LLMs [4]. They introduced the idea of using pre-trained models for downstream tasks and demonstrated the effectiveness through several classification tasks. Ever since then, the pre-trained models and their fine-tuning for downstream tasks have gradually gained attention. With the increase in computing capability, deep learning has continuously evolved. [PERSON] et al. used unidirectional LSTM [5], which can only capture unidirectional semantic information from the context [3]. [PERSON] et al. made a pioneering contribution by introducing bidirectional LSTM, which can learn information from both contexts [6]. They showed for the first time that semantic information can be embedded in word embeddings trained on a large-scale unlabeled data corpus, where the vectors representing words contain the semantic information of those words. Thus, this approach can achieve significantly better performance compared to traditional word embedding models. In 2018, ELMo trained a model to predict the current word using contextual information, with a two-layer LSTM to extract syntactic and semantic information [7]. This model shows outstanding performance in multiple NLP tasks, which can tackle the problem of word ambiguity. Subsequently, pre-trained models such as GPT [8; 9] and BERT [10] came into being, leading to remarkable advancements in natural language processing tasks. Following the success of GPT [8] and BERT [10], many pre-trained models have been proposed, showcasing the exceptional performance of pre-trained models in downstream tasks. Examples include ERNIE [11], SpanBERT [12], RoBERTa [13], XLNet [14], ALBERT [15], and ELECTRA [16]. More recently, the generic large language models, pre-trained on text-only corpora, have become surprisingly effective at encoding text for image synthesis, i.e., the text-to-image diffusion model [17; 18]. As a result, these pre-trained models have achieved impressive results in various NLP downstream tasks. At the same time, these pre-trained large language models require a lot of computing and memory resources for both model pre-training and fine-tuning. By taking the BERT (**B**idirectional **E**ncoder **R**epresentations from **T**ransformers) model as an example, we aim to answer the following three research questions. (1) How can we port the BERT model onto an ARM-based multi-core CPU? (2) How can we parallelize the BERT model, and how well does it run on the multi-core CPU? (3) How do the hyper-parameters impact the performance of BERT? In this article, we port the BERT model to an ARM multi-core processor and investigate how to accelerate the model fine-tuning by using multi-core parallelism to tuning hyper-parameters.