# Test Suite Prioritization Based on Optimization Approach Using Reinforcement Learning
## 3 Materials and Methods
Faults are injected in particular locations in the workflow through the fault injector, and then for evaluation, test the program after faults are injected in the program [30]. The fault data assembler assembles the related data for analysis, such as the number of injected faults, detected faults, and undetected faults. The result evaluator obtains the data from the fault data assembler and generates the results according to the data. In Q-learning, the agent learns from its environment. Figure 3 presents the workflow of agent learning in the proposed approach. The agent starts from the start state that becomes the current state, and the agent receives a reward from the current state. Then, the agent analyzes valid actions from the valid action array for the current state and the reward values against valid actions from the reward matrix. After analysis, the agent takes action, moves to the next state, and updates its Q-Table. Then, the next state becomes the current state, and the entire process is repeated according to the defined episodes. For instance, in our experimental configuration, the agent repeats the episode 1000 times, and the agent navigates from the initial state to the final state in the matrix representing the testing environment. The more agents explore the environment, the better the \(Q\) value achieved. \[Q(s,a)=r+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime}) \tag{1}\]  
Equation (1) (known as the Bellman equation) is used to compute the maximum future reward. It is the reward the agent received for entering the current state s plus the maximum future reward for the next state s\({}^{\prime}\). The \(Q\) value is iteratively approximated based on the Bellman equation. The Q-learning equation is given by  
\[Q_{t+1}(s_{t},a_{t})=Q_{t}(s_{t},a_{t})+\alpha(r_{t}+\gamma\max_{a}Q_{t}(s_{t} +1,a)-Q_{t}(s_{t},a_{t})) \tag{2}\]  
In Equation (2), \(\alpha\) is the learning rate that controls the difference between previous and new \(Q\) values. As the agent explores more and more of the environment, and the approximated \(Q\) values start to converge to \(Q\). The final updated Q table is obtained from 1000 episodes of training from the start state to the end state. After obtaining the updated Q-Table, the algorithm extracts the sequence of states from the start state to the final state with maximum reward. Finally, the proposed prioritization algorithm prioritizes the test cases based on this extracted sequence. For instance, if two or more test cases have the same similarity index, the length of test cases is another selection parameter. The test case with greater length is given priority among test cases with the same similarity index. The proposed Q learning algorithm-based TSP mechanism initializes Q-matrix with zeros. The value of \(\gamma\) representing the learning rate is set between 0 and 1. The reward matrix is constructed and initialized with the reward value of each state. To make the agent understand its environment well, the agent is trained with more episodes as they say, the more the agent explores the environment, the better the \(Q\) value obtained. For each episode, the agent starts from state-0, navigates to the goal state with a random navigation path, and selects one action among all possible actions for the current state (S) with the highest \(Q\) value. After performing an action on the state (S), the agent moves to the next state (S'), and the Q-table is updated using Equation (1). The proposed Q-Learning based TCP algorithm is presented in in Algorithm 1. Figure 3: Flow of the learning methodology of the proposed for test suite prioritization. ```
0: Initial Q-Table, Valid Action Array, Reward Matrix, Transition Matrix, Policy, and testing environment matrix. Updated Q-Table.