# An Algorithm for Making Regime-Changing Markov Decisions
## 6 Algorithm Implementations and Performance Analysis
### An illustration  
Let us consider a typical application of [PERSON] decision theory to agricultural management [28] which addresses a stochastic forest growth model in the context of timber harvesting optimization. The work [28] re-considers the classical results of [PERSON] in the framework of random growth and potential ecological hazards (for a derivation of [PERSON]'s result from 1849 and its discussion, we refer the reader to [28]). The idea is that as a reasonable approximation, one supposes that the only stochastic element is the growth of trees: That is, the timber volume per hectare defines the state. To facilitate the analysis, this timber volume (cubic meter per hectare) is discretized as shown in the first row of the Table 2. Furthermore, the costs of action (in USD per hectare) are shown in the second row, assuming that the action means timber harvest for all states (state 2-state 6) followed by re-forestation with the exception of the bare land (state 1), where there is re-forestation only. There are no costs if no action is taken, as seen from the third row of Table 2.  
Given the state space \(P=\{1,\ldots,6\}\) and control costs as defined in Table 2, define the action space \(A=\{1,2\}\), where \(1\) means being idle and \(2\) means acting. Assuming decision \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & **State 1** & **State 2** & **State 3** & **State 4** & **State 5** & **State 6** \\ \hline volume & 0 & 29 & 274 & 530 & 728 & 868 \\ acting & –494 & –117 & 3068 & 6396 & 8970 & 10,790 \\ being idle & 0 & 0 & 0 & 0 & 0 & 0 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Discrete states and their control costs. Figure 1: Stylized implementation of the control flow within a realistic application. intervals of 20 years, a Markov decision problem is determined with the action-dependent stochastic matrices  
\[\alpha^{1}=\left[ \begin{array}{cccccc}1&0&0&0&0&0\\ 0.1&0.1&0.7&0.1&0&0\\ 0.1&0&0.1&0.7&0.1&0\\ 0.1&0&0&0.1&0.7&0.1\\ 0.1&0&0&0&0.1&0.8\\ 0.1&0&0&0&0&0.9\end{array} \right],\ \ \ \ \ \ \ \alpha^{2}=\left[ \begin{array}{cccccc}0.1&0.9&0&0&0 &0\\ 0.1&0.9&0&0&0&0\\ 0.1&0.9&0&0&0&0\\ 0.1&0.9&0&0&0&0\\ 0.1&0.9&0&0&0&0\\ 0.1&0.9&0&0&0&0\end{array} \right].\]  
Note that the transitions to state 1 (bare land) from higher state in \(\alpha^{1}\) may be interpreted as a result of a natural disaster, whereas those to a neighboring state may describe a randomness in the forest growth. The matrix \(\alpha^{2}\) describes the timber harvest followed by re-forestation. The work [28] discusses an optimal policy for this model assuming an infinite time horizon with a discounting which is inferred from the interest rate effects. The optimal policy is compared to that from a non-random growth model, represented by the same parameters with the exception of the free-growth transition \(\alpha^{1}\), being replaced by  
\[\alpha^{1}=\left[ \begin{array}{cccccc}1&0&0&0&0&0\\ 0&0&1&0&0&0\\ 0&0&0&1&0&0\\ 0&0&0&0&1&0\\ 0&0&0&0&0&1\\ 0&0&0&0&0&1\end{array} \right].\]  
This non-random model resembles the well-known results of [PERSON], claiming that the only optimal strategy is a roll-over of harvesting and re-planting after a number of years. In the above Markov decision problem, the discount factor is set to \(\kappa=(1+g)^{20}=0.61\) (corresponding to \(g=2.5\%\) annual interest rates). Here, it turns out that it is optimal to reforest in state 1, do nothing in state 2 and 3, and cut and reforest in states 4, 5, and 6. That is, the resulting roll-over is three times the decision period, 60 years. In contrast, the random model suggests that in the presence of ecological risks, it is optimal to reforest in state 1, do nothing in state 2, and cut and reforest in states 3,4, 5, and 6, giving a shorter roll-over of two decision periods, 40 years. Using our techniques, the optimal forest management can be refined significantly.