# Energy Efficient Policies for Data Transmission in Disruption Tolerant Heterogeneous IoT Networks
## 4 Transmission Policy  
To deal with the dynamic program presented in Section 3 we capitalize on the computational methods provided by the framework of MDP The problem at hand constitutes a MDP since the state and decision spaces are finite, the distributions of the random system parameters are independent of each other, independent of their past values and invariant over time, i.e., the system is stationary and as a result state transitions are independent of past states and controls, given the current state \(s_{k}\) and control \(u_{k}\), and finally, the cost function is additive over time. The MDP can be described by the probability of transition between states,  
\[P\{s_{k+1}=j|s_{k}=i,\ u_{k}=u\}=p_{ij}(u)=P\{x_{k+1}|x_{k}\}\cdot P\{y_{k+1}| y_{k}\}\cdot P\{m_{k+1}|m_{k},u_{k}\}. \tag{5}\]  
From this point on we will utilize the Markov Chain notation introduced in Equation (5), whereby \(p_{ij}(u)\) is the probability that the system will make a transition to state \(j\) given that the system is in state \(i\) and decision \(u\) was made. For the MDP under consideration, given that \(0<\gamma<1\), there exists an optimal stationary policy \(\pi=\{\mu,\mu,\dots\}\), i.e., a policy that applies the same control function \(\mu\) at all stages ([30] Section 2.3). What is more, the control function \(\mu\) will be independent of the initial state of the system and deterministic [30], i.e., each time the system is in state \(i\), \(\mu(i)\) will make the same decision \(u\). We will refer to a stationary policy \(\pi=\{\mu,\mu,\dots\}\) as stationary policy \(\mu\). Our objective is to find a stationary policy \(\mu^{*}\), from the set of all admissible stationary policies \(\mathcal{M}\subseteq\Pi\), that minimizes total cost in Equation (4), i.e.,  
\[\mu^{*}=\arg\min_{\mu\in\mathcal{M}}J_{\mu}(i),\qquad\text{for all $i\in S$}. \tag{6}\]  
Let \(J^{*}\) be the total cost attained when the optimal policy \(\mu^{*}\) is used, then, for the MDP at hand, \(J^{*}\) satisfies the Bellman equation,  
\[J^{*}(i) =\min_{u\in U(i)}\sum_{j=1}^{n}p_{ij}(u)\left[g(i,u,j)+\gamma J^{*} (j)\right],\text{ for all $i\in S$},\] \[=\min_{u\in U(i)}\left[g(i,u)+\gamma\sum_{j=1}^{n}p_{ij}(u)J^{*}(j )\right],\text{ for all $i\in S$}, \tag{7}\]  
where \(n\) is the cardinality of the state space. Equation (7) actually describes a system of \(n\) non-linear equations, the right hand side of which is a contraction with a unique fixed point located at \(J^{*}(i)\). Due to the contraction property one can derive both \(J^{*}\) and \(\mu^{*}\) via iterative methods. In this work we utilize the Optimistic Policy Iteration (OPI) algorithm [30; 31; 32] to approximate the optimal policy \(\mu^{*}\) and the optimal horizon cost \(J^{*}\). The OPI algorithm depends on the Approximate Policy Evaluation (APE) algorithm thus we begin with the description of the latter algorithm and then proceed with the description of the OPI algorithm itself. Given a policy \(\mu\) we can approximate the horizon cost \(J_{\mu}\) by use of the APE algorithm presented in recursive form in Algorithm 1. Formally, Algorithm 1 converges to \(J_{\mu}\) only after an number of recursive calls, i.e., when \(r_{\max}\rightarrow\infty\). In practice, however, a finite value for \(r_{\max}\) must be selected and heuristically chosen values for \(r_{\max}\) lead to an accurate calculation of \(J_{\mu}\) as indicated by analysis and computational experience [30]. Another popular variation of Algorithm 1 stops the sequence of recursive calls when \(\max\limits_{i\in S}|J_{\mu}^{\prime}(i)-J_{\mu}(i)|\) becomes smaller than a predefined threshold [31].