# A Novel Multi-Task Performance Prediction Model for Spark
## 3 Proposed Model
### MHAC Model
#### 3.3.1 Custom Multi-Head Attention Networks
Without this normalization, the magnitude of the dot product could become excessively large, causing the gradient of the softmax function to approach zero and severely hampering the model's training. Through this formula, we can determine the attention score between features \(Q_{li}\) and \(Q_{lj}\) for each head. This score signifies the strength of the relationship between \(Q_{li}\) and \(Q_{lj}\), aiding the model in capturing and leveraging intricate dependencies among features. Upon computing the dot-product attention scores, the third step involves leveraging these scores to compute the weighted sum of features for each data point, yielding the novel feature representation for each head. For head \(l\) (where \(l\in\{1,2,3,4\}\)) and data point \(i\), the new feature representation \(Q_{li}^{\prime}\) can be acquired using Equation (18). \[Q_{li}^{\prime}=\sum_{j=1}^{n}att\_score_{lj}*Q_{lj} \tag{18}\]  
Herein, \(att\_score_{ij}l\) denotes the attention score between data points \(i\) and \(j\) in head \(l\). \(Q_{lj}\) signifies the feature representation of the \(j\)-th data point in head \(l\) after transformation by the weight matrix \(W\). Through this weighted summation process, we acquire the new feature representation \(Q_{li}^{\prime}\) for each data point across all heads. This enhanced representation apt captures the dependencies between data points and features, rendering it beneficial for performance prediction tasks. After completing the computation of new feature representations for each head, we then proceeded to aggregate these feature representations. We did this by concatenating the new feature representations from the four heads and passing them through a learnable linear transformation \(W^{O}\). The final feature representation was obtained, as shown in Equation (19). \[MultiHead=Concat(Q_{1}^{\prime},Q_{2}^{\prime},Q_{3}^{\prime},Q_{4}^{\prime}) W^{O} \tag{19}\]  
Here, MultiHead denotes the ultimate feature representation, while \(W^{O}\) signifies the linear transformation matrix to be optimized during the training phase. The term \(Concat(Q_{1}^{\prime},Q_{2}^{\prime},Q_{3}^{\prime},Q_{4}^{\prime})\) represents the column-wise concatenated feature vectors from the four distinct heads. Upon completion of this step, we secure a novel feature representation encapsulating information from all heads. This derived representation is subsequently input into the Convolutional Neural Network (CNN). * Perform convolution operation using kernel \(h\), with \(b\) as bias term, resulting in output \(Y\). The size of \(h\) is \(3\times 3\) as shown in Figure 3.
* Apply batch normalization to standardize output \(Y\) from step 3. Normalize \(Y\) using scaling factor \(\gamma\), add bias factor \(\beta\), yielding \(Y^{\prime}\). Here, \(\epsilon\) is a small constant to prevent division by zero. * Apply the ReLU activation function, known for its simplicity and effectiveness in mitigating gradient vanishing during neural network training. * For the second or deeper convolutional layers, execute residual connection. Add the current layer's output \(Y\) to the previous layer's input \(X\). This design enables learning of residual mappings, avoiding direct input-output mapping. \(X_{l+1}\), \(Y_{l}\), and \(X_{l}\) represent inputs and outputs. This enhances the deep network training and model performance. * If all convolutional layers are iterated, proceed; otherwise, return to step 2. * For each pooling window \(W_{i}\), perform average pooling. \(X^{\prime}_{i}\) corresponds to new feature matrix \(X^{\prime}\). * Execute fully connected layer for prediction results. \(W\) is the weight matrix, \(b\) is the bias vector--both learned during training. \(X^{\prime}\) is the feature matrix after pooling, \(Y_{\text{final}}\) is the model's prediction. After passing through the CNN, the model carries out backpropagation to continuously iterate to improve the performance of the model.