# A Reinforcement Learning Algorithm for Automated Detection of Skin Lesions
## 3 Proposed Method
### Overview of Our RL Method  
In this article, we propose a groundbreaking multi-stage segmentation strategy based on DRL to detect skin lesions. In each step, a segmentation agent is trained to find the best segmentation technique based on the previous step's evaluation results. In this article, the DDPG algorithm trains the segmentation agent to solve the MDP problem. DL and deterministic policy gradient (DPG) are combined in the DDPG algorithms [70]. The actor  
Figure 3: Segmentation of skin lesions by our proposed neural network architecture. uses the low-dimensional state space to make choices. The advantage of the DDPG is that it classifies policies that outperform the actor. Since DDPG is an off-policy algorithm, it provides a huge replay buffer, enabling it to learn from a wide variety of unrelated transformations. The predicted gradient of the action-value function is a more appealing version of the DPG. Because of its simplicity, the DPG can be calculated far more accurately than the traditional stochastic policy gradient. In high dimensional action spaces, DPG algorithms outperform their stochastic equivalents drastically. DDPG is a non-policy model-based algorithm for studying continuous action. It uses DQN's ERM and slow learning goal networks and is built on DPG, which operates through continuous action spaces. Compared to traditional methods, such as the level set, Chan-Vese, and snakes, the method proposed does not require any technological skills. The neural network optimizes the method proposed based on the segmentation results from the prior step. Thus, the network evolves techniques for segmenting skin lesion images without the need for specialized knowledge. DDPG is used to address problems that require continuous action space based on the AC architecture. It is suggested to solve two challenges: overcoming the delayed RL issue for neural networks and creating a self-learning framework based on a neural network that needs no training or reinforcement from the context. The AC paradigm is a mix of value and policy-based methods. The first uses an implicit system for learning the value function and an action-based value function to achieve the policy. Conversely, policy optimization refers to explicitly defined model capacity, such as the policy gradient (PG) [71]. Our RL-based image segmentation algorithm is shown in Algorithm 1 below:  
```
Randomly initializing actor network \(\mu(s\mid\theta^{\mu})\) and critic network \(Q(s,a\mid\theta^{Q})\) with weights \(\theta^{Q}\) and \(\theta^{\mu}\). Initializing of the target networks \(\mu^{\prime}\) and \(Q^{\prime}\) and weights \(\theta^{\mu^{\prime}}\leftarrow\theta^{\mu}\), \(\theta^{Q^{\prime}}\leftarrow\theta^{Q}\)  Initializing of experience replay memory R for episode \(e=1\), \(N\)do  Initializing a random process \(M\) for exploration of actions  Received \(s_{1}\) initial observation state for\(x=1\), \(T\)do  Select action parameter set \(a_{1}=\mu(s_{1}\mid\theta^{\mu})+N_{t}\) accordingly to the exploration noise and the current policy  Feed the action parameters \((A_{S0},A_{S1},A_{S1},...A_{ST})\) in the segmentation executor. Feed the updated segmentation mask \(S_{int}+1\) and the ground truth for computation of reward function \(r(t)\). Execution of actions \(a_{t}\) and observing reward \(r_{t}\) and observation of new state \(s_{t+1}\)  Storing transition \((s_{t},a_{t},r_{t},s_{t+1})\) in \(R\)  Sampling of a random mini-batch \((s_{i},a_{i},r_{t},s_{i+1})\) of \(N\) transitions from \(R\)  Set \(y_{i}=r_{i}+\gamma Q^{\prime}(s_{i+1},\mu^{\prime}(s_{i+1}\mid\theta^{\mu^{ \prime}})\mid\theta^{Q})\)  Feed the ground truth \(S_{int}\) in the critic network  Feed the reward \(r(t)\) and long term expected return \(Q\) to the evaluation network. Evaluation of the segmentation policy focused on reward \(r(t)\) and the long-term return \(Q\).