## 4 Theoretical Analysis of the Condition When DRL-Based Algorithm Does Not Converge  
A DRL algorithm is a combination of a deep learning (DL) algorithm and reinforcement RL algorithm [36], it combines the perception ability of the DL algorithm and decision-making ability of RL. DL is used to extract environmental features from the complex environment, RL learns from features and optimizes itself by trial and error. Neural network is a nonlinear function approximator, through data training, to find the best-optimized function of a certain situation [37], which in this paper is to approximating the Q-function of RL. Motivated by [38], in this paper we prove that the DRL-based algorithm does not converge when facing with certain probability of random actions in steady-state dynamics. Then we verify our proof by simulation. Anti-jamming process of DRL algorithm does not converge when facing with certain probability of random actions. The following proof follows the lines given in [38]. To start with, note the anti-jamming process of DRL is a Markov process [24], first consider a Markov process with finite states \(S=\{s_{1},s_{2},...,s_{n}\}\). We assume that the cost of transition between states and the discount factor \(\alpha\in(0,1)\). Let the function approximator be parametrized by a single scalar \(r\). \[\bar{J}(r)=(\bar{J}(1,r),\bar{J}(2,r),...,\bar{J}(n,r))^{T}. \tag{8}\]  
We let the \(\bar{J}(0)\) be a nonzero vector that satisfies with \(e^{T}\bar{J}(0)=0\), where the \(e=(\overbrace{1,1,...,1}^{n})^{T}\). \(\bar{J}(r)\)is a unique solution to a linear differential equation [38]:  
\[\tfrac{d\bar{J}}{dr}(r)=(Q+\sigma I)\bar{J}(r)\enspace, \tag{9}\]  
where \(I\) is a unit matrix of \(n\times n\), and \(\sigma\) is a tiny positive constant. \(Q\) matrix is average direction of the consequence of constant probability of random actions. It can be noted by:  
\[Q=\left[ \begin{array}{cccc}q_{1,1}&q_{1,2}&...&q_{1,n}\\ q_{2,1}&...&...&q_{2,n}\\...&...&...&...\\ q_{n,1}&q_{n,2}&...&q_{n,n}\end{array} \right]. \tag{10}\]  
According to our definition of \(\bar{J}\), it is easy to know that all functions represented by \(\bar{J}\) can map to three-dimensional space \(\{J\in\Re^{3}\left|e^{T}J=0\right.\}\). Set the Markov chain transition probability matrix is \(P\), and can be denoted as:  
\[P=\left[ \begin{array}{cccc}p_{1,1}&p_{1,2}&...&p_{1,n}\\ p_{2,1}&...&...&...\\...&...&...&...\\ p_{n,1}&p_{n,2}&...&p_{n,n}\end{array} \right]. \tag{11}\]  
Since the transition cost is set to \(0\), the \(TD(0)\) operator [39] can be defined as \(T^{(0)}=\alpha P\), for any \(J\in\Re^{3}\), there is a certain angle \(\theta\), and a scalar \(\beta\in(0,1)\), for any \(r\), \(T^{(0)}\bar{J}(r)\) is equal to the vector \(\bar{J}(r)\) extends \(\beta\) and rotates the \(\theta\) angle. Concretely, the update equation of \(TD(0)\) can be addressed as:  
\[r_{t+1}=r_{t}+\gamma\tfrac{d\bar{J}}{dr}(r)(\alpha\bar{J}(i_{t+1},r_{t})-\bar{J }(i_{t},r)), \tag{12}\]  
where the \(i_{t}\) is the state at time \(t\). The expectation of the update direction formula as follows:\[\sum\limits_{i=1}^{n}\frac{d\bar{l}}{d\tau}\left(\alpha\sum\limits_{i=1}^{n}p_{i,j} \bar{J}(j,r)-\bar{J}(i,r)\right), \tag{13}\]  
where \(j\) is the state that the \(i\) state is going to visit next. \(\alpha\sum\limits_{i=1}^{n}p_{i,j}\bar{J}(j,r)-\bar{J}(i,r)\) is equivalent to \((\alpha P-I)\bar{J}(r)\). From Equation (9), and as the \(r_{t+1}-r_{t}\) becoming small enough, the algorithm update process can be approximated by another differential equation. The average direction is given by the parameter, the differential equation describing this process can be:  
\[ \begin{array}{l}\frac{d\tau}{dt}=((Q+\sigma I)\bar{J}(r))^{T}(\alpha P-I)\bar {J}(r)\\ =\bar{J}^{T}(r)(Q^{T}+\sigma I)(\alpha P-I)\bar{J}(r),\end{array}  \tag{14}\]  
for \(\sigma=0\) (the process of neural network convergence is slow, then the \(\sigma\) can approximate to \(0\)), then we have:  
\[