# A Reinforcement Learning Algorithm for Automated Detection of Skin Lesions
## 3 Proposed Method
### MDP for the Segmentation of Skin Lesion  
The segmentation agent is used to find the ROI in this algorithm, with the skin lesion segmentation mechanism modeled as an MDP process. State, space S, operation A, and the reward features are the three main features of the MDP. This method's three-dimensional explanation is proposed as follows:  
Figure 5: (**a**) The segmentation executor. (**b**) The actor and (**c**) The critic. The following are the network’s architecture. The segmentation executor produces various segmentation results depending on the parameters taken for action. The action set is output by the actor as a series of five actions at a time, each with 8 parameters. The outputs of the critic are single valued for estimation. ResNet-18 is the architecture used. State: many of the agent's activities in the environment are included in the state space. The agent's decision will be based on this information. The state in this work consists of the image \(I\), the current segmentation mask \(S_{mt}\) and the step-index t, which is defined as \(S_{t}\) = (\(S_{mt}\), \(I\), \(t\)). \(S_{mt}\) is a mask for the segmentation process with a pixel scale of 0 or 255. The background pixels are 255 and 0. The original segmentation default value mask is 0. \(I\) is a representation of the lesion that requires segmentation. The index at step \(t\) is used to differentiate between the various phases of the segmentation. The state of the segmentation phase terminates in our multi-step segmentation process. The steps are maximized and performed until the training process is completed. Until the maximum number of stages has been achieved, the agent can enter the terminal state and continue to execute the final segmentation task. Action: the action area includes any operation that the segmentation executor can conduct. In a state, the agent makes a policy declaration \(\pi\) in the space of action. The action is then used to adjust the brushstroke direction and shape, described as several parameters. Reward function: The reward function defines the state to reward mapping in the RL task. Regularly, the agent's job is to maximize the amount of discounted future rewards R. This function signifies the immediate state reward when an action is changed, which helps evaluate the result's effectiveness for the decision taken by the agent. The M's segmentation mask changes at every step during the process of training. Consequently, the mask's _Acc_ is measured by the comparison of the truth ground mask for every step. The \(L2\) mean square error is used as an arbitrary metric. \(R_{l2}\) is the default \(L2\) reward mode. To better reflect each step's effect, we need an essential reward feature to take advantage of the \(L2\) change pattern. The resemblance between the two images calculates the value of \(L2\). If two images are the same, the loss of \(L2\) is equivalent to 0. The reward function can be modeled using the difference in RL2 between the two adjacent steps. \(R_{diff}\)'s reward function is shown in the Equation (1) below, where \(S_{m}A_{t}\) denotes the last segmentation mask, and \(S_{mt}\)denotes the mask. \[R_{diff}=L2(S_{mt}-1,G)-L2(S_{mt},G) \tag{1}\]  
The reward function sends a favorable signal when the loss of \(L2\) decreases and vice versa. Therefore, it is essential to predict Q's long-term return value at each point to enhance the learning trends. The reward function underlines the effectiveness of each action selected. The consistency of the operation chosen over the entire duration of the segmentation is an essential function. \(Q(S_{t},A_{t})\) is the \(A_{t}\) value defined in the \(S_{t}\) state. The Bellman Theorem is used to measure \(Q(S_{t},A_{t})\) and use the reward function. The reward function is denoted by \(R(S_{t},A_{t})\).