## 3 Fine-Tuning BERT Parallelization
### Parallelizing the BERT Model
#### 3.2.2 Strategies for Accelerating BERT Fine-Tuning  
Given that Phytium 2000+ has 64 CPU cores, investigating how to fully exploit the computational resources on this processor to accelerate the training process becomes necessary. For the TensorFlow-based BERT, we configure the number of CPU cores used for model training, thereby meeting the computational resource requirements. TensorFlow provides a configuration function called tf.ConfigProto(), which facilitates resource utilization and expedites the training process of the model. This function allows for various parameter settings, which are shown in Figure 7.  
The _device_count_ parameter sets an upper limit on the number of CPU cores to be used, while _inter_op_parallelism_threads_ and _intra_op_parallelism_threads_ indicate the level of thread parallelism for operations in a session. By default, these values are set to 0, allowing the system to automatically determine the correct values. When _log_device_placement_ is set to _True_, the messages about which operations and tensors are assigned to specific devices will be printed in the terminal, displaying their device assignments during model training.  
On parallelism, _inter_op_parallelism_threads_ and _intra_op_parallelism_threads_ are the two important parameters of parallelizing the BERT model. The parameter _intra_op_parallelism_threads_ determines the degree of parallelism for computations within each operator (e.g., matrix multiplication). On the other hand, _inter_op_parallelism_threads_ determines the level of parallelism for unrelated operations within the TensorFlow computation graph.  
For the BERT model, _intra_op_parallelism_threads_ impacts the parallelism of matrix multiplications during training. Figure 8 shows that one of the main components that can be parallelized in the BERT model is the self-attention mechanism. First, calculating the matrix of \(\mathbf{Q}\), \(\mathbf{K}\), and \(\mathbf{V}\) can be obtained by multiplying the input matrix with the matrices \(\mathbf{W}_{q}\), \(\mathbf{W}_{k}\), and \(\mathbf{W}_{v}\), respectively. This operator can be parallelized. Second, the attention matrix \(\mathbf{A}\), computed by multiplying the \(\mathbf{K}\) and \(\mathbf{Q}\) matrices, can also be parallelized. Finally, the output matrix \(\mathbf{O}\), obtained by multiplying the value matrix \(\mathbf{V}\) with the attention matrix \(\mathbf{A}\), can be parallelized as well. Thus, the parallelization of these three sets of matrix multiplications is controlled by _intra_op_parallelism_threads_. Additionally, the feed-forward neural network in the encoder is also parallelizable, as it involves matrix multiplication-related operations.  
Figure 7: Usage of tf.ConfigProto().  
On the other hand, _inter_op_parallelism_threads_ determines the level of parallelism between unrelated operations, particularly in the structure of the BERT model, where it primarily relates to the parallelism between multiple attention heads. Each attention head in the multi-head attention mechanism has its own set of matrices (\(\mathbf{W}_{q}\), \(\mathbf{W}_{k}\), \(\mathbf{W}_{v}\)), allowing each attention head to obtain different \(q\), \(k\), and \(v\) vectors for the same input \(x\). Since the operations between attention heads are independent, the parallelism can be controlled by changing _inter_op_parallelism_threads_.