## 4 Fine-Tuning the BERT Model on Multi-Core CPUs
### Performance Results
#### 4.2.1 Batch Size  
Batch size refers to the number of samples included in each gradient calculation during training. When using small batch sizes, parallel computation cannot be not easily achieved. Using a larger batch size allows for more effective parallel training by splitting training examples across different processor cores, significantly accelerating model training. The choice of batch size primarily affects the direction of gradient descent during model training. When the dataset is relatively small, we can perform a full-batch learning, where the entire dataset is treated as a single batch. In such a case, the gradient descent direction is determined by the entire dataset. When the dataset size is sufficiently large, loading the entire dataset into memory at once becomes impractical due to memory and computational limitations. If the selected samples are sufficiently representative, using half of the dataset (or even smaller) will yield a gradient descent direction that is comparable to that obtained with the full dataset. Therefore, selecting an appropriate batch size can accelerate model training without sacrificing prediction accuracy. Based on the above analysis, we run the parameter tuning experiments on different datasets by setting batch sizes to 16, 32, 64, 128, 256, and 512. We evaluate the performance impact of using various batch sizes on training time and prediction accuracy for the downstream tasks. The performance results in terms of training time and prediction accuracy are shown in Figure 16.  
In Figure 16a, we see that, as the batch size increases, the training time significantly decreases. This particularly holds for tasks with large datasets, such as CoLA and SST-2. In terms of prediction accuracy, we see that the impact of batch size on prediction accuracy is uncertain, as shown in Figure 16b. For the CoLA dataset, increasing the batch size results in a slight decrease in accuracy, which aligns with our previous analysis that using a larger batch size requires more epochs to achieve the same level of accuracy. Subsequent experiments also confirm that increasing the number of epochs can compensate for the accuracy loss caused by larger batch sizes. On the other hand, for datasets like RTE, WNLI, and MRPC, increasing the batch size leads to a slight improvement in accuracy. This is due to the fact that using larger batch sizes can better represent the direction of gradient descent for the dataset, increasing the likelihood of finding the optimal point. To summarize, we conclude that, for small datasets like RTE, WNLI, and MRPC, increasing the batch size brings the downstream task training closer to training on the entire dataset, leading to better identification of the correct gradient descent direction, but for large datasets like CoLA, the impact of increasing the batch size on the overall gradient descent direction is major. Although increasing the batch size can reduce the model training time, it requires more epochs to achieve the same level of accuracy. More epochs mean more training time. Thus, there exists a trade-off between batch size and the number of epochs. In resource-rich environments, increasing the batch size can reduce training time for downstream tasks and have a positive impact on accuracy for small datasets, but for large datasets, a balance needs to be achieved between batch size and the number of epochs. In a nutshell, using a larger batch size within a reasonable range has the following advantages. First, it can improve memory utilization and allow for more parallelism. Second, it can reduce the number of training iterations required to complete one epoch, leading to shorter training time for the same amount of data. Third, within a certain range, using a larger batch size provides a more accurate direction for gradient descent, leading to smaller training oscillations. On the other hand, blindly increasing the batch size can have the following side effects. First, it may exceed the memory capacity, consuming substantial memory resources.