## 2 Background
### Stochastic Models  
In this paper, we consider Markov chains or Markov processes that are stochastic models describing sequences of possible events in which the probability of each event depends only on the state attained in the previous event. More specifically, we concentrate on Continuous-Time Markov Chains. A Continuous-Time Markov Chain (CTMC) with finite or countable state space \(\mathcal{S}\) is a family \(\{X(t)\mid t\in\mathbb{R}^{+}\}\) of \(\mathcal{S}\)-valued random variables such that  
\[\mathit{Prob}(X(t_{n+1})=s_{n+1}\mid X(t_{1})=s_{1},X(t_{2})=s_{2 },\ldots,X(t_{n})=s_{n})=\\ \mathit{Prob}(X(t_{n+1})=s_{n+1}\mid X(t_{n})=s_{n}).\]  
This condition is the natural continuous-time analogue of the Markov property. It requires that the future behavior of the process is conditionally independent of its past evolution given the present states. We assume that the Markov properties under consideration satisfy the following properties: a Markov process \(X(t)\) is  
* _stationary_ if its statistical properties do not change by time, i.e., the family of random variables \((X(t_{1}),X(t_{2}),\ldots,X(t_{n}))\) has the same distribution as the collection \((X(t_{1}+\tau),X(t_{2}+\tau),\ldots,X(t_{n}+\tau))\) for all \(t_{1},t_{2},\ldots,t_{n},\tau\in\mathbb{R}^{+}\). * _time-homogeneous_ if the conditional probability \(\mathit{Prob}(X(t+\tau)=s\mid X(t)=s^{\prime})\) remains constant regardless of \(t\), i.e., the behavior of the system does not depend on when it is observed. In particular, the transitions between states are independent of the time at which the transitions occur. * _irreducible_ if all states in its state space \(\mathcal{S}\) can be reached from all other states by following the transitions of the process. Within a Markov process, a state is labeled _persistent_ or _recurrent_ if the probability of the process eventually returning to that state is one. Otherwise, the state is called _transient_. In terms of a system, the recurrent states correspond to the behavior which is repeatedly exhibited by the system, whereas transient states correspond to a behavior which will be no longer exhibited after a certain time. A recurrent state is termed _positive-recurrent_ or _ergodic_ if the expected number of steps until the process returns to that state is less than infinity. A Markov process is ergodic if all its states are positive-recurrent. In the context of finite Markov chains, irreducibility alone ensures ergodicity. In this paper, we consider Continuous-Time Markov Chains, which are time-homogeneous, irreducible, and ergodic. An ergodic Continuous-Time Markov Chain (CTMC) possesses an _equilibrium_ (or _steady-state_) _distribution_, defined as the _unique_ collection of positive real numbers \(\pi(s)\) where \(s\in\mathcal{S}\), satisfying the equation:  
\[\lim_{t\to\infty}\mathit{Prob}(X(t)=s\mid X(0)=s^{\prime})=\pi(s)\,.\]  
It is worth noting that the above equation for \(\pi(s)\) is independent of \(s^{\prime}\). We denote by \(q(s,s^{\prime})\) the transition rate from state \(s\) to state \(s^{\prime}\), where \(s\
eq s^{\prime}\). The sum of all transition rates out of state \(s\) to any other state in the chain is denoted as \(q(s)\). A state \(s\) for which \(q(s)=\infty\) is termed an instantaneous state, as it is instantaneously left upon entry. While theoretically possible, we assume throughout that \(0<q(s)<\infty\) for each state \(s\). The infinitesimal generator matrix \(\mathbf{Q}\) of a CTMC \(X(t)\) with state space \(\mathcal{S}\) is defined as the \(|\mathcal{S}|\times|\mathcal{S}|\) matrix. Its off-diagonal elements are the \(q(s,s^{\prime})\)'s, and its diagonal elements are the negative sum of the off-diagonal elements of each row, i.e., \(q(s,s)=-\sum_{s^{\prime}\in\mathcal{S},\ s^{\prime}\
eq s}q(s,s^{\prime})\). For simplicity, we use \(q(s,s^{\prime})\) to denote the components of matrix \(\mathbf{Q}\). For \(s\in\mathcal{S}\) and \(S\subseteq\mathcal{S}\), we write \(q(s,S)\) to denote \(\sum_{s^{\prime}\in S}q(s,s^{\prime})\).