## Experimental Results
### Comparison of Motion Information Representation in the State Space
#### 5.2.2 Results  
All experiments were carried out on a server with a 1080 Ti graphics card, 48 GB memory, and an Intel i7-8700k CPU. Tensorflow was used to build the models, and Tensorboard, a visualization tool that comes with Tensorflow, was used to export the charts. Generally, rewards are used as the judgment index of the model performance of reinforcement learning. In this paper, game score was used as the index to compare the proposed model with other existing models.  
Figure 6 presents the training scores of the four image state spaces. These four curves show that the more images that were superimposed, the higher the game score. With one exception, the calculation of the influence map includes the object motion, and the score was therefore similar to that of the four-frame image overlay method. After training a certain number of times, the influence map scored even higher. As shown in Figure 6, at the abscissa of 1.2 M, the four curves from top to bottom are those of the influence map superimposed image, four consecutive frames, two consecutive frames, and single-frame image. The corresponding ordinates of the four curves are presented in Table 6.  
The four curves shown in Figure 6 all exhibit upward trends; the more frames superimposed, the faster the upward speed. However, when one, two, and four frames are superimposed reach a certain score, the curve almost stops increasing and tends to be flat. The score of the influence map superimposed image continues to increase, showing that the dynamic information carried by the influence map enables agents to learn more environmental knowledge.  
Table 7 reports the corresponding ordinates of the four curves at the abscissa of 1.4 M. The comparative experimental results presented in Figure 5 show that the state space of  
\begin{table}
\begin{tabular}{l l l l} \hline \hline
**State Space** & **Smooth Value** & **Actual Value** & **Training Time** \\ \hline Single-frame image & 3079 & 3443 & 4 h 35 m 1 s \\ Two consecutive frames & 4119 & 4343 & 5 h 20 m 56 s \\ Four consecutive frames & 5509 & 6103 & 8 h 17 m 20 s \\ Influence map superimposed image & 6201 & 6840 & 5 h 10 m 33 s \\ \hline \hline \end{tabular}
\end{table}
Table 6: Ordinate comparison of four training scores at 1.2 M gradient descent.  
Figure 6: Training scores of four different image state spaces.  
the influence map superimposed image achieved better performance; the performance was 11.8-12.6% higher than that of the four consecutive frames method. In addition, as shown by the comparison of the use of the two consecutive frames and the influence map superimposed image, the reason for the better performance is not the overlay of consecutive images but the role played by the influence map. The influence map can indicate the trend and focus of the current map. In addition to expressing certain motion information, it also plays a role similar to that of an attention mechanism. The brightness contrast of the influence map may cause the convolution kernel of the CNN to pay more attention to brighter localities, which allows the neural network to take into account the information provided by the influence map when selecting actions.