## 3 Design and Training Methods for DQN Feature Constrained Control
### Training Method and Result
The training method for the Q-network based on the offline camera FOV environment is as follows (Algorithm 1):  
```
Define the camera FOV environment, state space \(S\), action space \(A\), and \(r\) reward functions;  Set up and initialize Q-network (primary network) and target network parameter \(\theta_{i},\theta_{i}^{-}\);  Initialize learning rate \(\alpha\), discount factor \(\gamma\), exploration rate \(\varepsilon\); for episode = 1, \(\ldots\), \(n\)do  Randomly initialize the feature's state (positions) in the camera FOV  and observe the initial state \(s_{(x,y)}\); for\(t\) = 1, \(\ldots\), \(m\)do  Acquire state \(s_{t}=s_{(x,y)}\) and select actions from \(A\) based on \(\varepsilon\)-greedy policy;  Get reward \(r_{t}\) and reach new state \(s_{t+1}\);  Store sample \((s_{t},a_{t},r_{t},s_{t+1})\) into the replay buffer \(D\); if\(s_{t+1}\) is located in VS area then  break; if\(D\) is full then  Extract \(N\) samples from \(D\) according to Equations (8)-(11)  Update network Q-network and target network parameters \(\theta_{i},\theta_{i}^{-}\); endif endfor  endfor  Optimal action value function Q based on FOV feature constraints (trained Q-network)
```  
**Algorithm 1**Q-Network Training  
In order to realize the Q-network to control the displacement of features according to the feature state in the python-based PyCharm environment, construct a Q-network, as well as target networks with the same structure. The structure of these two networks is a two-layer fully connected neural network with a hidden layer containing 50 neurons, each followed by a ReLU activation function. Then, set the network learning rate to \(\alpha\) = 0.01 and the reward discount factor to \(\gamma\) = 0.99. To ensure feature path exploration and optimal action selection, set the greedy policy to \(\varepsilon\) = 0.2; set the replay buffer \(D\) to size 150 and the training extracted data batch \(N\) to size 32 for training learning of the Q-network. Meanwhile, to simulate the actual camera FOV environment, we followed the scheme outlined in Section 3. The 640 \(\times\) 480 camera plane was further segmented, and a virtual camera FOV environment of size 16 \(\times\) 16 was constructed within the PyCharm environment. In this environment, based on the design principles of the action set \(A\) and reward function \(r\) in Section 3, we trained the Q-network to guide the movement of the feature. Specifically, the purpose of our offline training is to guide the Agent (point features) in the FOV environment to follow the principle of reward function into VS mission area. The specific offline training process is shown in Figure 2. During the training process in the grid environment based on the virtual camera FOV, the position of the Agent is defined as the state \(s\), and its movement direction is defined as the action \(a\). Partitions with different reward values, \(r\), are set up according to the point feature FOV rules. During training, the Agent interacts with the environment in the simulation to generate experience data \((s,a,r,s^{\prime})\), which are stored in the experience memory pool. Both the target network and the Q-network sample data from this pool to compute the target Q-value and the current Q-value. These are used to construct the loss function, which incorporates the immediate reward \(r\). Ultimately, the Q-network optimizes the Q-value function through gradient descent to learn the policy that constrains the Agent's behavior. In this training experiment, the Q-network was trained for 500 iterations, resulting in the averaged controller rewards shown in Figure 3. From the figure, it can be observed that the reward value continues to increase after several training sessions, and the average cumulative reward curve tends to asymptotically converge, which indicates that the Q-network can be trained to constrain the features based on the offline camera FOV environment to ensure the visibility of the features in the camera FOV.