# An Intelligent Shooting Reward Learning Network Scheme for Medical Image Landmark Detection
## 3 Our Proposed Shooting Reward Learning Network
### Markov Decision Process Formulation  
Inspired by the reinforcement learning for face-alignment tasks, in this work we propose medical landmark detection as a Markov decision process [49]. We define \(L=[L_{1},L_{2},\cdots,L_{I}]\in R^{2\times I}\) as a location vector of \(I\) points, where \(L_{i}\) denotes for the horizontal and vertical coordinates of the \(i\)-th landmark, given a medical image \(I\). All ground truth landmarks should be represented by the vector \(L^{GT}=[L_{1}^{GT},L_{2}^{GT},\cdots,L_{I}^{GT}]\). In this work, a Markov decision process is realized through the definition of an agent. We constrain the full procedure to have both an action space \(A\) and a discrete, finite state space \(S\). The agent can select from a finite set of actions, \(A(s_{i})\subseteq A\), for each state \(s_{i}\in S\). With probability, an agent in state \(s_{i}\) will transition to state \(s_{j}\) after performing action \(\alpha\subseteq A(s_{i})\). The agent is rewarded at each time step \(t\), depending on the transition from state \(s\) to state \(s^{\prime}\) and the action \(\alpha\). Maximizing the total anticipated reward along the whole available structure approaches is the agent's purpose. Starting from a certain state \(s_{i}\), perform action \(\alpha\) in accordance with policy \(\pi\). [PERSON]'s Equation, often known as the recursive maximization equation, can be articulated as  
\[Q_{\pi}^{*}(s_{i},\alpha)=E_{s_{j}|s_{i},\alpha}[E_{r|s_{j},\alpha,s_{j}}[r|s_ {i},\alpha,s_{j}]+\gamma\max_{\alpha^{\prime}\in A(s_{j})}Q_{\pi}^{*}(s_{i}, \alpha^{\prime})]. \tag{2}\]  
The Bellman equation could frequently be formulated as an iterative update [41]:  
\[Q_{t+1}(s_{i},\alpha)=(1-\beta)Q_{t}(s_{i},\alpha)+\beta[R_{t}+\gamma\max_{ \alpha^{\prime}\in A(s_{j})}Q_{\pi}^{*}(s_{i},\alpha^{\prime})]. \tag{3}\]  
where \(R_{t}\) stands for the agent's reward and \(Q^{*}(s_{i},\alpha)\) is the maximum total expected reward. Consequently, \(Q^{*}(s_{i},\alpha)\) is referred to as the \(Q\)-values. The weight provided to fresh knowledge over old information is determined by the Q-learning rate (\(\beta\)), while the weight given to immediate rewards over long-term rewards is determined by the discount factor (\(\gamma\)).  
We evaluate each candidate's quality value using the searched network and choose a course of action based on policy \(\pi\):  
\[\pi_{\theta_{c}}(s)=\max_{\alpha\in A(s_{j})}Q_{\pi}^{*}(s_{i},\alpha) \tag{4}\]  
where \(\theta_{c}\) denotes the network architecture parameters for the entire search.  
**Shooting reward.** We modified the distance between the anticipated regression landmark and the target in different places after assessing the model's performance, and we assigned various rewards depending on where the regression landmark was located. If we utilize ground truth as the target for the landmark detection task, landmark detection becomes identical to a shooting game that is repeatedly played to achieve the goal of getting nearer to the target. Therefore, to concurrently maximize both processes of learning a landmark loss function and routing a reliable regression network architecture, we designed a learnable incentive (shooting reward Figure 3). In particular, the reward function is intended to quantify the SDR evaluation index, which is defined as follows:\[d_{i}=\theta|\overrightarrow{L_{i}}-\overrightarrow{L_{i}^{GT}}|. \tag{5}\]  
\[R=\left\{\begin{array}{ll}10&d_{i}<2.0\;\mathrm{mm}\\ 8&2.0\;\mathrm{mm}\leq d_{i}<2.5\;\mathrm{mm}\\ 6&2.5\;\mathrm{mm}\leq d_{i}<3.0\;\mathrm{mm}\\ 3&3.0\;\mathrm{mm}\leq d_{i}<4.0\;\mathrm{mm}\end{array}\right.\]  
\[R_{t}=\frac{1}{n}\sum_{i=1}^{n}R. \tag{6}\]  
where \(\theta\) is the pixel spacing and \(L_{n}^{o_{x}},L_{n}^{o_{y}}\) are the horizontal and vertical differences between the predicted landmarks and the ground truth, respectively.