# Power Control and Link Selection for Wireless Relay Networks with Hybrid Energy Sources
## 3 Optimal Control for Expected Total Rewards
### Problem Simplification  
From Equation (10), we can see that the optimized variables of \(\text{P}_{1}\) include the 0-1 variable \(I_{j}^{i}\) and the continuous variable \(p_{SR}^{i}\), \(p_{SD}^{i}\) and \(p_{R}^{i}\). Thus, it is very difficult to optimize so many different typesof variables at the same time. Therefore, we do some simplification of P\({}_{1}\) to reduce the optimized variables. And, the problem of P\({}_{1}\) can be expressed as:  
\[P_{2}:\underset{p_{R}^{i}}{\max}E\left\{\sum_{i=1}^{N}\left[\beta_{i}D_{i}-(1- \beta_{i})H_{i}\right]\right\} \tag{16}\]  
\[\text{s.t. }C_{R}^{i}\leq E(i) \tag{17}\]  
\[p_{R}^{i}\leq p_{R}^{\max}\forall i\in\mathbb{N}. \tag{18}\]  
And \(\beta_{i}\) is the 0-1 variable, which can be calculated as:  
\[\beta_{i}=\left\{\begin{array}{cc}1\;p_{R}^{i}\
eq 0\\ 0\;p_{R}^{i}=0\end{array}\right.\;\;\forall i\in\mathbb{N}. \tag{19}\]  
That is, when the transmission power of relay is non-zero, \(\beta_{i}=1\) and relay link is chosen to transmit data. While the power of relay is zero, \(\beta_{i}=0\) and direct link is selected to deliver bits. In addition, \(D_{i}\) is the network total reward when relay link is chosen, which is given by:  
\[\text{D}_{i}=\omega_{t}R_{RD}^{i}-\omega_{g}G_{SR}^{i}, \tag{20}\]  
where \(R_{RD}^{i}\) is the throughput, and \(G_{SR}^{i}\) is the grid energy consumed by the source node. We assume that the amount of data bits transmitted in the two stages of relay link should be equal. So, there is  
\[R_{SR}^{i}=R_{RD}^{i}=B\frac{T}{2}\log(1+h_{SR}^{i}g_{0}d_{SR}^{-\alpha}\sigma ^{-2}p_{SR}^{i})=B\frac{T}{2}\log(1+h_{RD}^{i}g_{0}d_{RD}^{-\alpha}\sigma^{-2}p _{R}^{i}). \tag{21}\]  
From equation (21), we can know that  
\[h_{SR}^{i}d_{SR}^{-\alpha}p_{SR}^{i}=h_{RD}^{i}d_{RD}^{-\alpha}p_{R}^{i}. \tag{22}\]  
According to Equation (20) to (22), \(D_{i}\) can be further given as:  
\[D_{i}=w_{t}B\frac{T}{2}\log(1+h_{RD}^{i}g_{0}d_{RD}^{-\alpha}\sigma^{-2}p_{R}^ {i})-w_{g}\frac{T}{2}\frac{h_{RD}^{i}d_{RD}^{-\alpha}p_{R}^{i}}{h_{SR}^{i}d_{SR }^{-\alpha}}. \tag{23}\]  
\(H_{i}\) is the reward when direct link is selected, and the problem of solving \(H_{i}\) can be expressed as:  
\[P_{H}:\underset{p_{SD}^{i}}{\max}H_{i}=\underset{p_{SD}^{i}}{ \max}[\omega_{t}R_{SD}^{i}-\omega_{g}G_{SD}^{i}]=\omega_{t}TB\log(1+h_{SD}^{i }g_{0}^{\prime}d_{SD}^{-\alpha}\sigma^{-2}p_{SD}^{i})-\omega_{g}Tp_{SD}^{i} \tag{24}\]  
\[\text{s.t. }p_{SD}^{i}\leq p_{S}^{\max}\forall i\in\mathbb{N}. \tag{25}\]  
**Proposition 1**.: _The reward of the direct link \(H_{i}\) is convex in transmission power \(p_{S}^{i}\)._  
**Proof:** The second derivative of formula (24) is  
\[f(p_{SD}^{i})^{\prime\prime}=-\frac{w_{t}TB(h_{SD}^{i}g_{0}d_{SD}^{-\alpha} \sigma^{-2})^{2}}{\left(1+h_{SD}^{i}g_{0}d_{SD}^{-\alpha}\sigma^{-2}p_{SD}^{i }\right)^{2}\ln 2}<0, \tag{26}\]  
which means that \(f(p_{SD}^{i})\) is a convex function.  
According to the properties of convex functions, we can easily know that \(H_{i}\) gets the maximum value at \(p_{\text{SD}}^{j}=\frac{\omega_{l}}{\omega_{g}}-\frac{1}{h_{SD}^{i}8d_{SD}^{i} \omega^{-2}}\). Therefore, the optimal \(p_{\text{SD}}^{j}\) and \(H_{i}\) are known once \(h_{\text{SD}}^{i}\) is given. However, \(p_{\text{SD}}^{j}\) has practical significance only on [0, \(p_{S}^{\text{max}}\)] in this work. Consequently, when \(\frac{\omega_{l}}{\omega_{g}}-\frac{1}{h_{SD}^{i}8d_{SD}^{i}\omega^{-2}}<0\), \(P_{H}\) decreases monotonously on [0, \(p_{S}^{\text{max}}\)] with maximum value \(p_{H}(0)\). While \(\frac{\omega_{l}}{\omega_{g}}-\frac{1}{h_{SD}^{i}8d_{SD}^{i}\omega^{-2}}>p_{S} ^{\text{max}}\), \(P_{H}\) increases monotonously on [0, \(p_{S}^{\text{max}}\)] with maximum value \(p_{H}(\text{max})\).  
It can be seen from Equation (16) to (26), that the values of \(\beta_{i}\) and \(D_{i}\) are determined by \(p_{R^{\prime}}^{i}\), and the maximum value of \(H_{i}^{*}\) is a fixed value in each time slot which is only related to \(h_{SD}^{i}\). Therefore, the optimal variable of problem \(P_{2}\) is only \(p_{R}^{i}\).