# Test Suite Prioritization Based on Optimization Approach Using Reinforcement Learning
## 4 Results and Discussion
### TSP Example as Proof of Concept  
In this sub-section, the proof of concepts of the proposed model for TSP are presented for a clear understanding of the proposed TSP mechanism using the theorem. In this example, we consider an application that utilizes some users' and testers' logs. First, the Q-learning algorithm takes users' and testers' logs, testing environment matrix, reward matrix, transition matrix, initial Q-Table, and valid action matrix as the input and calculates the updated Q-Table. Next, another algorithm extracts a sequence of states with the highest reward value from the initial state to the final state. Finally, the prioritization algorithm prioritizes the test cases based on the sequence with the highest reward value. Testers' logs are recorded while implementing the test cases; the test case log is \"S0(C), S3(A), S8(A), S10,\" where S0(C) means the tester performs action C on the state S0 which leads to state S3 and then tester performs action A on the state S3 which leads to state S8 and so on. After tester logs collection, the actions from states are separated to obtain a sequence of states for constructing the testing environment matrix. Table 2 presents an example of tester logs based on the states S.  
After the testers' logs collection, the user logs are collected. For instance, different user logs are automatically recorded using case study applications. Each entry in the user log consists of state numbers, and action values such as \"S0(B), S3(B), S8(A), S10\", where S0 is the state name, B is the action, and ',' separates the events of user logs. Then preprocessing function is applied to the user logs, separating actions from activity names of all the user logs. After preprocessing and data preparation, the sequence of states of user logs is prepared. The user logs state sequences is presented in Table 3.  
The agent learning environment can be presented as a grid or matrix. In the testing environment matrix, an agent can perform only a few correct actions; the rest are invalid ones. Valid action array includes left, right, up, down, and no action. In addition, the same action can be invalid in the case of border states, resulting in an agent position outside the testing environment states. From the above users' and testers' logs, there are 11 states obtained, ranging from S0 to S10. The algorithm converts them into a testing environment matrix. For 11 states, a 3 \(\times\) 4 matrix is needed, where every cell represents a state. Table 4 presents an example of the testing environment matrix in which every cell symbolizes a state. The total number of states is 11, but the 3 \(\times\) 4 matrix has 12 states, as shown in the example in the Table 4. Thus the 12 th state of the testing environment matrix is considered as a block state. The block state means the agent cannot move to this state. The reward is allocated to every state of the agent navigation testing environment. The agent learns from the testing environment matrix based on these reward values. Agent prefers valid actions which maximize its reward value. The following functions are used to calculate the reward values of each state. \[f(r)=f(x)*c\] \begin{table}
\begin{tabular}{c c c c} \hline S0 & S1 & S2 & S3 \\ S4 & S5 & S6 & S7 \\ S8 & S9 & S10 & Block \\ \hline \end{tabular}
\end{table} 
Table 4: Agent navigation testing environment matrix. \begin{table}
\begin{tabular}{c c} \hline \hline
**Test Case No.** & **Sequence** \\ \hline
1 & S0, S1. \\
2 & S0, S1, S5. \\
3 & S0, S1, S5, S10. \\
4 & S0, S3. \\
5 & S0, S1, S4, S10. \\
6 & S0, S3, S9, S10. \\
7 & S0, S2, S7. \\
8 & S0, S1, S5, S10. \\
9 & S0, S3. \\
10 & S0, S3, S8, S10. \\ \hline \end{tabular}
\end{table} 
Table 3: User logs states sequences. \begin{table}
\begin{tabular}{c c} \hline \hline
**Test Case No.** & **Sequence** \\ \hline
1 & S0, S1. \\
2 & S0, S1, S5. \\
3 & S0, S1, S5, S10. \\
5 & S0, S1, S5, S10. \\
6 & S0, S3, S8, S10. \\ \hline \end{tabular}
\end{table} 
Table 2: Tester logs states sequences.