## 3 Design and Training Methods for DQN Feature Constrained Control
### Training Method and Result  
Based on the previous section, the DQN Agent (point feature) selects the linear velocity vector in the x and y directions of the camera's coordinate system as the output action, and five discrete actions, labeled a\({}_{1}\) to a\({}_{5}\), are designed around the velocity vectors in these two directions. Additionally, the virtual camera FOV environment used for training the Q-network is also a discrete grid space, and discrete point feature displacement actions are employed. This enables the DQN model, based on a discrete action space, to efficiently learn the feature constraint strategy through the Q-learning mechanism. The Q-network is essentially trained to learn using neural network learning to obtain the action value function  
\[Q^{\pi}(s_{t},a_{t})=r_{t}+\gamma Q^{\pi}(s_{t+1},\pi(s_{t+1})) \tag{7}\]  
This equation represents the Q-value of taking the action \(a_{t}\) at the time \(t\) for the Agent's state \(s_{t}\). \(\pi(s_{t+1})=\underset{a_{t+1}}{\text{max}}\ Q^{\pi}(s_{t+1},a_{t+1})\) represents the optimal action \(a_{t+1}\) taken at the time \(t+1\) for the Agent's state \(s_{t+1}\). In order to realize the feature constraint control of the DQN method, we construct a Q-network to estimate the function \(Q^{\pi}(s_{t},a_{t})\), while a target network is used to update the weight parameter \(\theta\) of the Q-network, facilitating the convergence of the Q-network gradients. The target network is a replica network of the former, and we specify the weight parameter of the target network as \(\theta^{-}\) to distinguish it. The control of hand-eye camera movements based on Agent (feature) FOV states \(S\) is based on the \(Q^{\pi}(s_{t},a_{t})\) function, which is actually outputted by the Q-network as \(Q^{\pi}_{\theta_{i}}(s_{t},a_{t})\), and by the target network as \(Q^{\pi}_{\theta_{i}}(s_{t+1},a_{t+1})\). During the training of the two networks, the output of the target network \(Q^{\pi}_{\theta_{i}^{-}}(s_{t+1},a_{t+1})\) is used to calculate the target value \(y=r_{t}+\delta Q^{\pi}_{\theta_{i}^{-}}(s_{t+1},\pi(s_{t+1}))\), while the output of the Q-network \(Q^{\pi}_{\theta_{i}}(s_{t},a_{t})\) serves as the predicted value. Then, the process of updating the Q-network weight \(\theta\) is as follows:  
\[L(\theta_{i})=\frac{1}{N}\underset{i=1}{\overset{N}{\sum}}\Big{(}y-Q^{\pi}_{ \theta_{i}}(s_{t},a_{t})\Big{)}^{2} \tag{8}\]  
\[\theta_{i}\leftarrow\theta_{i}+\alpha\
abla L(\theta) \tag{9}\]  
\[\
abla L(\theta_{i})=\frac{1}{N}\underset{i=1}{\overset{N}{\sum}}\
abla\Big{(} y-Q^{\pi}_{\theta_{i}}(s_{t},a_{t})\Big{)}^{2} \tag{10}\]  
Another important technique to address the issue of non-independent and non-identically distributed training samples is experience replay. By utilizing a replay buffer (the experience pool) that stores past experiences (namely \(S\), \(A\), \(r\), \(S^{\prime}\)), this method allows for the mixing of past and current experiences. This reduces the correlation between data points and enables the reuse of samples, ultimately enhancing the learning efficiency. In the above Equations (8) and (10), \(N\) denotes the number of sample batches obtained from the experience pool, and \(\alpha\) is the learning rate. For the target network, parameter \(\theta^{-}\) is not updated in real time; the target network is assigned the parameter after the Q-network parameter is updated for a period of time; then, it can be expressed by Equation (11):  
\[\theta_{i}^{-}\leftarrow\theta_{i} \tag{11}\]The Q-network interacts with the environment multiple times through the above process and learns to obtain the optimal action-value function, \(Q_{\theta_{i}}^{\pi}(s_{t},a_{t})\). With function \(Q_{\theta_{i}}^{\pi}(s_{t},a_{t})\), the visual feature as Agent can use the \(\varepsilon\)-greedy policy to select the optimal action strategy \(\pi(s_{t})=\underset{a_{t}}{\text{argmax}}Q_{\theta_{i}}^{\pi}(s_{t},a_{t})\) and control the camera movement to keep visual features within the camera's safe FOV with strategic action \(\pi(s_{t})\).