## 2 System Model
### DT Mode  
In direct transmission mode, the source directly transmits the data to the destination (without a help from the relay). According to whether the source receives ACK from destination, the following two observations can be described as follows.  
Observation 3 (\(\Phi_{3}\)): The source transmits the data directly to the destination and receives an ACK. The achieved throughput is given by  
\[R\left(e_{t}^{re},p_{t}^{b}\left|\Phi_{3}\right.\right)=\frac{T-\tau_{\text{s }}}{T}C_{0}. \tag{34}\]  
The probability that the case \(\Phi_{3}\) occurs is computed as  
\[\Pr\left(\Phi_{3}\right)=\frac{\left(1-p_{t}^{b}\right)P_{f}}{p_{t}^{b}P_{d}+ \left(1-p_{t}^{b}\right)P_{f}}. \tag{35}\]  
The belief that the jammer will be present in the next time slot can be updated as  
\[p_{t+1}^{b}=P_{\bar{J}\bar{J}}. \tag{36}\]  
In this case, although the relay does not receive and forward data to the destination, it still harvests energy for future use. The updated remaining energy of the relay for the next time slot is  
\[e_{t+1}^{re}=\min\left(e_{t}^{re}+E_{h}\left(t\right),E_{ca}\right). \tag{37}\]  
The transition probability that the case \(\Phi_{3}\) happens is computed as  
\[\Pr\left(e_{t}^{re}\to e_{t+1}^{re}\left|\[PERSON]_{3}\right.\right)=\Pr\left[E_{h }\left(t\right)=e_{k}^{h}\right]\Pr\left(\Phi_{3}\right), \tag{38}\]_Observation 4_ (\(\Phi_{4}\)): The source transmits data directly to the destination, but it does not receive an ACK. In this case, there is no achieved throughput such that we have \(R\left(e_{t}^{re},p_{t}^{b}\left|\[PERSON]_{4}\right.\right)=0\). The probability that case \(\Phi_{4}\) happens can be calculated as follows:  
\[\Pr\left(\Phi_{4}\right)=\frac{p_{t}^{b}P_{d}}{p_{t}^{b}P_{d}+\left(1-p_{t}^{b} \right)P_{f}}. \tag{39}\]  
The updated belief for the next time slot can be given as  
\[p_{t+1}^{b}=P_{JJ}. \tag{40}\]  
The remaining energy in the relay is updated in the same way as Equation (37) under \(\Phi_{3}\). The transition probability that case \(\Phi_{4}\) occurs is given by  
\[\Pr\left(e_{t}^{re}\to e_{t+1}^{re}\left|\Phi_{4}\right.\right)=\Pr\left[E_{h} \left(t\right)=e_{k}^{h}\right]\Pr\left(\[PERSON]_{4}\right), \tag{41}\]  
for \(k=1,2,3...,\
u\).  
Based on these observations, we can calculate the expected value function, and further we can find the optimal operation mode, \(a_{k}\). Therefore, the value function in (22) can be rewritten as follows  
\[V_{\left(e^{re},p^{b}\right)}=\max\limits_{a_{k}}\left\{\begin{array}{l} \sum\limits_{t=k}^{\infty}\alpha^{t-k}\sum\limits_{\[PERSON]_{i}\in a_{t}}\Pr\left( \[PERSON]_{i}\right)\\ \sum\limits_{e_{t+1}}\Pr\left(e_{t}^{re}\to e_{t+1}^{re}\left|\Phi_{i} \right.\right)\\ R\left(e_{t}^{re},p_{t}^{b},a_{t}\left|\Phi_{i}\right.\right)\left|e_{k}^{re} =e^{re},p_{k}^{b}=p^{b}\end{array}\right\}\right. \tag{42}\]  
In order to solve problem in Equation (42), a numerical method is used [30]. The solution to the problem provides the optimal policy of the system. The complexity of algorithm can be analyzed based on the amount of computation space such as number of states, actions, transition probabilities and observations. Based on the Bellman's equation, the optimal policy is chosen by solving the value function using iteration-based dynamic programming. Let us denote \(\mathbb{Z}\), \(\mathbb{S}\) be the action set and the possible state set at the beginning of each time slot, respectively. The algorithm complexity can be defined according to the action and state space of the system. In the POMDP, the agent has to control the process at each time step to maximize the long-term reward. Therefore, the number of \(O\left(\left|\mathbb{Z}\right|\left|\mathbb{S}\right|^{2}\right)\) operations is required in each iteration to calculate total number of the transition probabilities from one state \(s\left(t\right)\) to other state \(s^{\prime}\left(t\right)\) after performing an action \(a\left(t\right)\).