# Test Suite Prioritization Based on Optimization Approach Using Reinforcement Learning
## 4 Results and Discussion
### TSP Example as Proof of Concept
In Equations (4) and (5), \(x\) represents the state frequency value, \(r\) denotes the reward value, \(c\) corresponds to the complexity of the state defined by the programmer, \(me\) represents the mean value, and \(md\) denotes the median value. Equation (4) shows that the reward of each state is calculated using \(f(x)\) multiplied by the complexity of the state. \[f(x)=\left\{ \begin{array}{ll}0&\text{invalid}\\ -\lambda&<me\\ +1&me\leq x\geq md\\ +\lambda&>md\end{array} \right. \tag{5}\]  
Equation (5) represents that function \(f(x)\) assigns 0 to invalid actions of the agent and 10 to \(\lambda\) value, and \(+1\) where the frequency is in between the mean value and median value. The block state is the state in which the agent cannot move. S0 is the initial state, and S10 is the goal or final state shown in Table 4. To encourage the agent to navigate into the final goal, the reward with a negative number means to avoid the action, and the reward in a positive number encourages to do the action to maximize the reward. For example, the final state 'S10' is given three times the highest positive reward amount to encourage the agent to reach the final state. Block states are given a reward amount of five times more negative than the highest reward to discourage the agent from moving there. Reward values of each action on a state are stored in the reward matrix. The reward matrix contains the reward values of every state; for example, the reward values of state S0 are 0, 1, 0, 10, \(-1\). The transition matrix is necessary to make the agent understand the transitions for each state. The value \(-1\) in the matrix means that the transition is impossible, as it is not possible on the first state S0 for the top action. The values of state S0 in the transition matrix are \(\{-1\), S4, \(-1\), S1, S0\(\}\). If the agent performs an action, it move to state S4. The transition matrix is constructed based on the states and actions. It shows by taking action 'a' what would be the agent's next state's'. The initial Q-Table is initialized with zeros, and its size is M X N, where M is the number of states and N is the number of actions. The agent takes an initial Q-Table to maintain the history of reward values in the learning process. When the agent takes any action in the testing environment, the reward value is stored in the initial Q-Table. After every episode, the agent updates its initial Q-Table, and the final updated Q-Table is obtained at the end of the learning process. Valid action matrix is for providing the agents with valid actions it can practice on each state during its learning episodes:  
* Valid actions are (Up = 0), (Down = 1), (Left = 2), (Right = 3) and (NA = 4). * \(-1\) represents invalid actions that the agent cannot perform and restricts the agent while programming the agent to avoid \(-1\). In Q-Learning, the valid action matrix is essential because it contains valid actions for every state for the agent navigation over different states in a learning environment. For example, valid actions of State S0 are \(\{-1,1,\)\(-1,3,4\}\), stating that the agent cannot take up and left actions in State S0, and the agent can take only down, left, and no action in State S0. The up and left actions are invalid for the agent on the State S0, and the rest of the actions are valid. During the learning process, the agent updates its initial Q-Table according to the reward values against actions. Then, after learning the complete environment, the agent gets an updated Q-Table. In this example, the agent learns the environment in 1000 episodes; after 1000 episodes, the agent's updated Q-Table is given in Table 5.  
For extracting the sequence with highest reward value from updated Q-Table, the reward allocation algorithm considers the testing environment, updated Q-Table and actions. In the testing environment matrix, the initial state is S0. In the updated Q-Table for State S0, the highest reward value is on action 3, so the agent performs action 3, which denotes action to the right.