# Attention Transfer Reinforcement Learning for Test Case Prioritization in Continuous Integration
## 4 ATRL-TCP: Attention Transfer Reinforcement Learning for TCP
We input the observation vector into the full connected layer, using Relu as the activation function between layers and using the tanh activation function as the output activation function to limit the value of noise to constrain its effect on the guide vector:  
\[\text{noise}=\text{tanh}(\omega\text{State}+b) \tag{4}\]  
The guide vector (the current guide is batch data) is calculated using Equation (5), where \(\odot\) denotes the Hadamard Product. The values of mask and bias are shown in Equations (6) and (7), and the number of rows of both matrices is equal to the batch size:  
\[\text{guide}=\text{mask}\odot\text{{State}}+\text{bias} \tag{5}\]  
\[\text{mask}= \begin{bmatrix}0&\cdots&0&1&\cdots&1\\ 0&\cdots&0&1&\cdots&1\\ \vdots&\ddots&\vdots&\vdots&\ddots&\vdots\\ 0&\cdots&0&1&\cdots&1\end{bmatrix}  \tag{6}\]  
\[\text{bias}= \begin{bmatrix}0.5&\cdots&0.5&0&\cdots&0\\ 0.5&\cdots&0.5&0&\cdots&0\\ \vdots&\ddots&\vdots&\vdots&\ddots&\vdots\\ 0.5&\cdots&0.5&0&\cdots&0\end{bmatrix}  \tag{7}\]  
The structure of Guide Attention is illustrated in Figure 2. The _State_ represents the input observation matrix, which undergoes dimensional expansion to generate two matrices: _State_\({}^{\prime}\) and _State_\({}^{\prime\prime}\). The _State_ matrix is first element-wise multiplied (Hadamard product) with the _mask_ matrix, ensuring selective retention of specific elements. After this operation, a _bias_ term is added to obtain the _guide_ matrix, which serves as the primary guiding component in the attention mechanism. Simultaneously, the _State_\({}^{\prime}\) matrix is processed through a fully connected layer, transforming it into the _noise_ matrix. The _noise_ matrix maintains the same dimensions as the original input, ensuring alignment with subsequent operations. To regulate its impact, a _Tanh_ activation function is applied, constraining the values of the _noise_ matrix within a fixed range. This guarantees that when summing with the _guide_ matrix, the overall guiding influence of the _guide_ remains dominant. Next, the modified _guide_ matrix, now influenced by the controlled noise, is processed row-wise using the \(Softmax\) function. This normalization ensures that each row sums to 1 while emphasizing the most significant features in the data. The result of this operation is the _score_ matrix, which encodes the attention distribution. Finally, the _score_ matrix undergoes a Hadamard product with the \(State^{\prime\prime}\) matrix, allowing it to modulate the original observations dynamically. The output of this operation constitutes the observation matrix that has been processed through the Guide Attention module, incorporating both structured guidance and feature-weighted refinement. Building on the design of the Guide Attention module, we developed the Attention Transfer Reinforcement Learning for TCP. The model still employs an Actor-Critic architecture to enhance training stability. We referenced the TD3 method in designing the model structure in Figure 3.  
The model includes two Actors: Actor and Actor_T. The first is used to generate the data required for model training and make the final decisions, while the second is used to calculate the target value needed for the Critic's loss calculation. The model also includes four Critics: Critic1, Critic2, Critic_T1, and Critic_T2. The first two calculate their losses using the action values generated by the Actor, while the latter two supervise each other and collaborate with Actor_T to produce the minimal target values required by Critic1 and Critic2. By employing two sets of Critic networks and using the smaller value when calculating the target value, the model suppresses the issue of network overestimation. The Critic networks are updated multiple times before updating the Actor network, ensuring more stable training for the Actor network. As shown in Figure 4, Long Short-Term Memory (LSTM) networks are utilized to process the historical execution data of test cases, extracting useful sequential dependencies  
Figure 3: Overall architecture of the model.