# On Hybrid Energy Utilization in Wireless Sensor Networks
## 3 Simulation Results  
In this section, the performance of the proposed hybrid energy utilization scheme is evaluated by computer simulation based on a value iteration algorithm, as shown in Algorithm 1. Two performance metrics--(1) a constant energy utilization ratio and (2) a packet dropping probability--are taken into consideration. Based on [10], we assume the discount factor (\(\gamma\))= 0.99, stopping criterion \(=10^{-6}\), and the cost value for the transmission of one packet or dropping one packet is one energy unit individually. For simplicity, we assume that the packet arrival to the wireless sensor buffer can be zero or one. In each time slot, the proposed MDP model uses both battery units and the constant energy source to transmit a number of packets such as zero, one, two, three, and so forth. ```
Input: Initial Cost Values \(C_{A}(S)\) Output: Optimal Expected Cost Value \(V_{\pi^{*}}(S)\) Begin the loop;  Calculate expected cost values for first state \(V_{1}(S)\)  Calculate expected cost values for second state \(V_{2}(S)\) while\(|V_{i+1}(S)-V_{i}(S)|\geq\epsilon\)do  Calculate expected cost values for next state \(V_{i+1}(S)\) repeat \(i\gets i+1\); until; \(|V_{i+1}(S)-V_{i}(S)|<\epsilon\) end the loop  end while \(V_{\pi^{*}}(S)=V_{i+1}(S)\)
```  
**Algorithm 1**Value Iteration  
Figure 2: Markov chain for the Markov decision process \((N_{Q}=N,N_{B}=N)\). Figure 3 presents a flowchart for the value iteration algorithm, which finds the optimal transmission policy based on the proposed MDP model. In a conventional method (the performance calculations of hybrid energy utilization without using MDP), the expected value is as the summation of cost value for each action that will be taken over the same loop, similar to the value iteration algorithm's loop for the proposed MDP method. The scenario which does not use MDP has no transmission policy, and the action will be taken randomly for each time slot. Besides, it does not have the discount factor and stopping criterion to calculate the expected value. According to the configuration setting, two performance metrics (i.e., power consumption ratio and PDP) are compared with/without using MDP in two different simulation environments. In the first simulation environment, the model sizes of the battery and packet buffer are two by two, three by three, or four by four. Additionally, the solar energy unit arrival is assumed to be only zero or one (Case 1), which fits the aforementioned model sizes as depicted in Figures 4 and 5. In Figure 4, the difference of the power consumption ratio between the MDP scenario and the scenario without MDP can reach up to 40%. In both scenarios, the power consumption ratio between the constant energy source and total hybrid power decreases as the model size increases. The reason behind this is that the larger battery or packet buffers accommodate more solar harvested energy for transmitting packets. Consequently, more packets can be transmitted in a timely fashion, avoiding packet dropping. However, in the without-MDP scenario, the average power consumption ratio can be fixed because the weighting factor \(\alpha\) does not affect the constant energy source usage. On the other hand, Figure 5 shows that the PDP decreases for both scenarios as the model size increases. In the MDP scenario, this led to more packets being received and transmitted at the right time based on solar harvested energy. In addition, the percentage of PDP will be fixed when \(\alpha\) is larger than or equal to 0.5 for both scenarios. Needless to say, this percentage does not change, since the cost of PDP  
Figure 3: Flowchart for the value iteration algorithm. will be always less than the cost of constant energy source used in this case. On the other side, \(\alpha\) will be negligible without using MDP and the action will be dropping the packet. In the second simulation environment, the same model size of the battery and packet buffer is used, which is four by four, as shown in Figures 6 and 7.