## 3 Problem Descriptions and Optimization
For each EH-ST, the state in the chain is defined by \(\{E_{k}\}\), where \(\{0,1,2,\ldots,e_{max}\}\) represents the available energy units stored in frame \(k\). The transition matrix of the Markov chain is denoted by \(P=[p_{mn}]\), where \(p_{mn}\) is the probability of one-step transition given by:  
\[\Pr\left\{E\left(k+1\right)=e_{n}\mid E(k)=e_{m}\right\}. \tag{14}\]  
According to the energy harvesting model, energy consumption model, and the defined policy, the transition probability from state \(e_{m}\) to state \(e_{n}\) is formulated by:  
\[p_{mn}= \begin{cases}\beta\left(e_{m},b_{k}\right),&\text{if }e_{m}=0\text{ and }e_{n}\
eq e_{ max}\\ 1-\sum\limits_{n=0}^{e_{max}-1}\beta\left(n,b_{k}\right),&\text{if }e_{m}=0 \text{ and }e_{n}=e_{max}\\ \eta\left(e_{m}\right)\beta\left(e_{m},b_{k}\right),&\text{if }e_{m}\
eq 0 \text{ and }e_{m}\leq l_{max}\text{ and }e_{m}>e_{n}\\ \eta\left(e_{m}\right)\beta\left(e_{m},b_{k}\right)+[1-\eta\left(e_{m}\right)] \beta\left(e_{n}-e_{m},b_{k}\right),&\text{if }e_{m}\
eq 0\text{ and }e_{m}\leq l_{max} \text{ and }e_{m}\leq e_{n}\text{ and }e_{n}\
eq e_{max}\\ \eta\left(e_{m}\right)\left[\sum\limits_{n=e_{max}}^{\infty}\beta\left(n,b_{k }\right)\right]+[1-\eta\left(e_{m}\right)]\left[\sum\limits_{n=e_{n}-e_{m}}^{ \infty}\beta\left(n,b_{k}\right)\right],&\text{if }e_{m}\
eq 0\text{ and }e_{m}\leq l_{max} \text{ and }e_{m}=e_{max}\\ \eta\left(e_{m}\right)\beta\left(e_{n}-e_{m}+l_{max},b_{k}\right),&\text{if }e_{m}\
eq 0 \text{ and }e_{m}>l_{max}\text{ and }e_{m}>e_{n}\\ \eta\left(e_{m}\right)\beta\left(e_{n}-e_{m}+l_{max},b_{k}\right)+[1-\eta\left( e_{m}\right)]\beta\left(e_{n}-e_{m},b_{k}\right),&\text{if }e_{m}\
eq 0 \text{ and }e_{m}>l_{max}\text{ and }e_{m}\leq e_{n}\text{ and }e_{n}\
eq e_{max}\\ \eta\left(e_{m}\right)\left[\sum\limits_{n=e_{n}-e_{m}+l_{max}}^{\infty}\beta \left(n,b_{k}\right)\right]+[1-\eta\left(e_{m}\right)]\left[\sum\limits_{n=e_ {n}-e_{m}}^{\infty}\beta\left(n,b_{k}\right)\right],&\text{if }e_{m}\
eq 0 \text{ and }e_{m}>l_{max}\text{ and }e_{m}=e_{max}\\ 0.&\text{otherwise}\end{cases}  \tag{15}\]  
which indicates the Markov chain is irreducible under this transmission policy. Therefore, there exists a unique steady-state probability distribution, which is denoted by \(\pi_{\mathbf{\eta}}(\mathbf{e}),\mathbf{e}\in E^{U}\), and it satisfies both \(\left(P^{\prime}-I\right)\pi_{\mathbf{\eta}}^{\prime}(\mathbf{e})=0\) and \(\sum_{\mathbf{\tau}=0}^{e_{max}}\pi_{\mathbf{\eta}}\left(e\right)=1\), where \(P\) and \(I\) are transition probability and identity matrices [37]. Thus, the steady-state probability \(\pi_{\mathbf{\eta}}(\mathbf{e})\) can be acquired by solving the linear equations. Moreover, the steady-state probability distribution is independent of the initial state \(\mathbf{e_{0}}\), so Equation (10) can be further deduced as:  
\[T_{\mathbf{\eta}}^{\left(u\right)}=\mathbb{E}\Bigg{[}\sum\limits_{e\in E^{U}}\pi_ {\mathbf{\eta}}(\mathbf{e})g\left(\eta_{u}\left(e_{u}\right)\right)\binom{U}{M}\prod \limits_{i\
eq u,j}\eta_{i}\left(e_{i}\right)\times\prod\limits_{j\
eq u,i} \left(1-\eta_{j}\left(e_{j}\right)\right)\Bigg{]}. \tag{16}\]  
Since the packet transmission action \(Q_{u,k}\) is based only on the current available energy unit in EH-ST \(u\), the energy harvesting process is i.i.d for all EH-STs. Moreover, the energy level of each EH-ST is independent, so \(\pi_{\mathbf{\eta}}(\mathbf{e})\) can be represented as:  
\[\pi_{\mathbf{\eta}}(\mathbf{e})=\prod\limits_{u}\pi_{\eta_{u}}(\mathbf{e}_{u}), \tag{17}\]  
where \(\pi_{\eta_{u}}(\mathbf{e}_{u})\) is the steady-state probability of EH-ST \(u\) at energy level \(e_{u}\). Letting:  
\[ \begin{split} G(\eta_{u})&=\sum\limits_{e=1}^{e_{ max}}\pi_{\eta_{u}}(\mathbf{e}_{u})g\left(\eta_{u}\left(e\right)\right)\\ P(\eta_{u})&=\sum\limits_{e=1}^{e_{max}}\pi_{\eta_{u}}( \mathbf{e}_{u})\eta_{u}(e),\end{split}