# A Novel Multi-Task Performance Prediction Model for Spark
## 3 Proposed Model
### MHAC Model
#### 3.3.1 Custom Multi-Head Attention Networks  
In our Spark performance prediction model MHAC, the data after PCA dimensionality reduction are fed into our customized multi-head attention network for processing. Different from the conventional multi-head attention network, our model does not use the traditional K, V, and Q settings, but instead uses specially designed \(h\) \"heads\" to better adapt to our application requirements, and we default to \(h=4\) in this paper. These \(h\) \"heads\" are able to pay attention to each feature of the input data at the same time, learn the weights and attention distributions among them, and capture the complex dependencies among the features. This processing helps extract valuable semantic information from input data and job features, which can then be used for performance prediction. The architecture of the customized multi-head attention network is shown in Figure 2.  
The specific implementation and computational process of the customized multi-head attention network are shown in Figure 2. Initially, for the input feature matrix \(Q\in\mathbb{R}^{n\times k}\), we generate the new feature matrices \(Q_{i}\in\mathbb{R}^{n\times d}(i=1,2,3,4)\) through multiple (4 by default) independent linear transformations. Each of these new feature matrices is derived using Equation (13). The expansion of \(Q_{i}\) is detailed in Equation (14). \[Q_{i}=QW_{i} \tag{13}\]\[Q_{i}= \begin{pmatrix}q_{i11}&\cdots&q_{i1d}\\ \vdots&\ddots&\vdots\\ q_{i71}&\cdots&q_{ijd}\\ \vdots&\ddots&\vdots\\ q_{i21}&\cdots&q_{i2d}\\ \vdots&\ddots&\vdots\\ q_{i1n1}&\cdots&q_{iind}\end{pmatrix} _{n\times d} \tag{14}\]  
where \(n,j,d,z\in\mathbb{Z}^{+}\), \(W_{i}\in\mathbb{R}^{k\times d}\) represents the corresponding weight matrix, and \(i\in\{1,2,3,4\}\). These four feature matrices represent four different \"heads\", each focusing on different aspects of the input features. During the training process of the entire multi-head attention network, these weight matrices \(W_{i}\) are updated to minimize the prediction error, thereby optimizing the model. \(Q_{i}\) is a feature matrix with \(n\) rows and \(d\) columns, where \(n\) denotes the number of data points and \(d\) represents the dimension of the new feature space. Next, the second step involves computing the dot-product attention scores. For each pair of features, we calculate their dot-product attention scores across each head. Specifically, for any two data point features \(Q_{li}\), \(Q_{lj}\in\mathbb{R}^{1\times d}\) from \(Q_{l}\) (where \(i,j\in[1,n]\)), the attention score att_score\({}_{ij}\) on head \(l\) (with \(l\in[1,4]\)) is computed as shown in Equation (15). Figure 2: The structure of the multi-head attention network. The specific expansions of the data point features \(Q_{li}\) and \(Q_{lj}\) are given in Equation (16) and Equation (17), respectively. \[att\_score=softmax(\frac{Q_{li}*Q_{lj}^{T}}{sqrt(k)}) \tag{15}\]  
\[Q_{li}=(q_{li1},q_{li2},\ldots,q_{lid})^{1\times d} \tag{16}\]  
\[Q_{lj}=(q_{lj1},q_{lj2},\ldots,q_{lid})^{1\times d} \tag{17}\]  
Here, \(att\_scorehead_{ij}\) denotes the attention score between features \(Q_{li}\) and \(Q_{lj}\) within the \(l\)-th head. Both \(Q_{li}\) and \(Q_{lj}\) are features of data points \(i\) and \(j\) in the \(l\)-th head after the transformation by the weight matrix. The \(softmax(*)\) function is an activation function, transforming any real-valued vector into a probability distribution. Its function ensures that all attention scores lie between 0 and 1, and their cumulative sum equals 1. The dot product \(Q_{li}Q_{lj}^{T}\) between features \(Q_{li}\) and \(Q_{lj}\) is a prevalent method for gauging their similarity. Its magnitude reflects the semantic similarity between feature \(i\) and feature \(j\), with a larger dot product indicating greater similarity. The normalization factor \(\sqrt{k}\), where \(k\) represents the dimensionality of the feature vector, is incorporated to tackle the issue of the dot product rapidly increasing with dimensionality.