## 2 Materials and Methods  
Reliability of any complex system is always crucial when one wants to optimize its performance. Various reliability indices can further be used for the planning of potential preventive, corrective, and maintenance actions and to maintain the system's ability for trouble-free operation. Many methods, including fault tree analysis, maximum likelihood estimator, fuzzy membership function, etc., have been used in past to understand the behavior of reliability and related indices. The Markov decision process is also one of such tools that can be used to model an industrial/networking system to understand its various performance measures. During operation, a system can be in any one of these possible states: perfectly working, working with less efficiency (also known as degraded  
Figure 1: Applications of wireless sensor networks. state), and failed state, depending on which components are working perfectly, which are working in a degraded state, and which are components in failed state. More clarity for the system's reliability can be gathered if it is modeled through the various instances of subsystem failures as well as subsequent repairs. The Markov decision process is a stochastic process through which one can model the different states and transition between them on the basis of different failure/repair rates of the subsystems of a system. The Markov process is introduced by the mathematician \"[PERSON]\" in early 1900. It is one of the most important processes out of all stochastic process, due to its use and simplicity. A stochastic process is a collection of random variables \(Y_{t}\) defined over some indexed set \(T\). Then, with full generality, it is assumed that for every \(n\), the distribution of \(Y_{n+1}\) depends on the whole history \((Y_{0}=i_{0},Y_{1}=i_{1},Y_{n}=i_{n})\). In contrast, the Markov process asserts that the distribution of \(Y_{n+1}\) depends only on the current state \(Y_{n}=i_{n}\), not on the whole history. Formally, the process \(Y_{n}\) is called a Markov process if for each \(n\) and every \(i_{0},i_{1},...,i_{n}\) and \(j\in N\)[33],  
\[P[Y_{n+1}=j\mid Y_{0}=i_{0},Y_{1}=i_{1},...,Y_{n}=i_{n}]=P[Y_{n+1}=j\mid Y_{n} =i_{n}] \tag{1}\]  
The Markov process can be used to model real time problems, per the index variable \(t\). There are two possibilities for \(t\), discrete and continuous. In these cases, the discrete-time Markov process and continuous-time Markov process can be used, respectively. In continuous-time Markov processes, the conditional probability of the future state at time \((t+s)\), given the present state at \(s\) and all past states, depends only on the present state and is independent of the past states [34]. \[P[Y_{t+s}=j\mid Y_{s}=i,Y_{u}=k,0\leq u\leq s]=P[Y_{t+s}=j\mid Y_{s}=i]=P_{ij}(t) \tag{2}\]  
Here, \(P_{ij}(t)\) represents the transition state probability from the \(ith\) state to the \(jth\) state. Also, one can understand it as the probability that the Markov process, which is presently in state \(i\), will be in state \(j\) after some time \(t\). Further, Equation (2) can be generalized as  
\[P_{ij}(t+S)=\sum_{k}P[Y_{t+s}=j,Y_{t}=k\mid Y_{0}=i]\]  
\[=\sum_{k}\frac{P[Y_{0}=i,Y_{t}=k,Y_{t+s}=j]}{P[Y_{0}=i]}\]  
\[=\sum_{k}\frac{P[Y_{0}=i,Y_{t}=k]}{P[Y_{0}=i]}*\frac{P[Y_{0}=i,Y_{t}=k,Y_{t+s} =j]}{P[Y_{0}=i,Y_{t}=k]}\]  
\[=\sum_{k}P[Y_{t}=k\mid Y_{0}=i]*P[Y_{t+s}=j\mid Y_{0}=i,Y_{t}=k]\]  
\[=\sum_{k}P[Y_{t}=k\mid Y_{0}=i]*P[Y_{t+s}=j\mid Y_{t}=k]\]  
\[=\sum_{k}P_{ik}(t)P_{kj}(s) \tag{3}\]  
Equation (3) is known as Chapman-Kolmogorov equations for the continuous-time Markov process. Using the concept of the continuous-time and discrete state space Markov process, along with different failures and repairs that occur during the operation, the author modeled a wireless sensor network (WSN) and obtained the different reliability indices of it. The reliability-based modeling of a WSN through the Markov decision process (MDP) has not been conducted so far, which make this work novel.