## 3 Markov Decision Process  
MDP [15] is described as a discrete-time-state transition stochastic process which gives a mathematical framework for making any rational decision when the outcome is partly random and partly regulated by the decision makers [27]. It is convenient to use MDP formulations to line up a strategy under uncertainty. Here decisions are optimized in either of the following way, i.e., minimization of the expected cost to meet the objective or maximization of the expected reward. MDP process model is presented in Figure 2.  
MDP is expressed by five-tuple: \((X_{t},A_{t},P,R,\gamma^{\prime})\). Here \(X_{t}\) denotes system state at any time instant \(t\) and \(A_{t}\) represents the finite set of actions (\(a_{t}\)) where the corresponding action \(a_{t}\) if performed at \(X_{t}\) drives the system state \(X_{t}\) to one of the probable next states \(X_{t+1}\) as depicted in Figure 2. The state transition probabilities from each system state \(X_{t}\) to next possible state \(X_{t+1}\) depending on the action \(a_{t}\) performed are recorded in matrix P and the corresponding reward generated at each state are documented in matrix R. The structure of P matrix and R matrix are presented in Figure 3a,b respectively for m number of system states subject to an action \(a_{t}\). Both matrices are represented in \(m~{}\times~{}m\) dimensions indicating all possible state changes where each row heading and column heading, i.e., \(State~{}i~{}\exists_{i=0}^{m-1}\) indicate present system state and next possible system state, respectively. For instance, the element of P matrix \(P_{X_{0-2}}\) represents state transition probability from state 0 to state 2 and accordingly the element \(R_{X_{0-2}}\) of R matrix denotes the corresponding reward generated due to state change from state 0 to state 2 depending on the action \(a_{t}\). However, each system state could be represented as a combination of k state variables  
\[X=\{SV^{1},SV^{2},..,SV^{k}\} \tag{1}\]  
Each state variable could have different range of values, say \(m_{1}\),\(m_{2}\),..,\(m_{k}\) then the product of these values gives the order of these matrices, i.e., \(m=m_{1}\times m_{2}\times..\times m_{k}\). For simplicity, let State 0 and State 2 are represented as \(\{SV^{1}_{0},SV^{2}_{0},..,SV^{k}_{0}\}\) and \(\{SV^{1}_{2},SV^{2}_{2},..,SV^{k}_{2}\}\) respectively in Figure 3a,b. In such case, the element of P matrix \(P_{X_{0-2}}\) is given by the product of state transition probabilities of each state variables from State 0 to State 2, i.e., \(P_{X_{0-2}}=P_{SV^{1}_{0-2}}\times P_{SV^{2}_{0-2}}\times...\times P_{SV^{k}_{ 0-2}}\). Accordingly, the reward \(R_{X_{0-2}}\) in R matrix gives the reward value which results due to performing action \(a_{t}\) at \(State0(\{SV^{1}_{0},SV^{2}_{0},..,SV^{k}_{0}\})\) which drives the system to \(State2(\{SV^{1}_{2},SV^{2}_{2},..,SV^{k}_{2}\})\). Hence, a pair of P matrix and R matrix are required for each action. Figure 2: MDP Process Model where \(t_{i}\) denotes \(i\)-th time instant. The series of rewards obtained due to performing sequence of actions at each predicted state starting from the current state yield the utility value. Obtained rewards are simply put in to quantify additive utility, i.e.,  
\[U_{add}([r_{0},r_{1},r_{2}...])=r_{0}+r_{1}+r_{2}+.. \tag{2}\]  
whereas discounted utility is measured using a discount factor (\(\gamma^{\prime}<1\)) where sooner rewards have more impact than later rewards. \[U_{dis}([r_{0},r_{1},r_{2}...],\gamma^{\prime})=r_{0}+\gamma^{\prime}r_{1}+ \gamma^{\prime 2}r_{2}+.. \tag{3}\]  
Discounted utility is particularly suitable for convergence of optimization algorithms where the sequence of actions is predicted to intensify the expected discounted utility. Optimal MDP policy is designed by value iteration process that is repeated for all states \(s\). An arbitrary value \(V_{0}\) assigned at each state.