## 5 Solving the Dilemmas
### Anticipatory Troubleshooting Dilemmas as a Markov Decision Process
#### 5.2.1 Fix-Replace Dilemma as an MDP  
To model the fix-replace dilemma as an MDP, we first discretize the time limit \([0,T_{limit}]\) by partitioning it to \(n\) non-overlapping time intervals of size \(\Delta_{T}\). Let \(T=\{T_{0},\ldots,T_{n}\}\) be the time steps such that \(T_{0}=0\), and for all \(i>0\), \(T_{i}\) is the time step that is the upper limit of \(i^{th}\) time interval. Thus, \(T_{n}=T_{limit}\). The number of time intervals (\(n\)) is referred to as the granularity level. The granularity level should be set according to the domain such that it provides a sufficient approximation of real failure times. Finally, we can present our modeling of the fix-replace dilemma for component \(C\). **States. ** A state in our MDP is a tuple \(s=\langle T_{i},IsFaulty,S_{C},C_{Age}\rangle\), where:  
* \(T_{i}\) is a time step. * \(IsFaulty\) is true iff \(C\) is faulty in the time interval \([T_{i-1},T_{i}]\). * \(S_{C}\) is the survival curve of \(C\) at time \(T_{i}\). * \(C_{Age}\) is the age of \(C\) at time \(T_{i}\). States for time \(T_{0}\) represent the first time step. States for time \(T_{limit}+1\) are terminal states. Recall that we only consider single fault scenarios--i.e, at most one component fails. **Actions. ** The set of actions in our MDP consists of two actions: \(Replace(C)\) and \(Fix(C)\). The outcome of applying an action in a state \(s=\langle T_{i},IsFaulty,S_{C},C_{Age}\rangle\) results in one of the two following states: \(s^{\prime}_{fail}=\langle T_{i+1},true,S^{\prime}_{C},C^{\prime}_{Age}\rangle\) and \(s^{\prime}_{ok}=\langle T_{i+1},false,S^{\prime}_{C},C^{\prime}_{Age}\rangle\). \(s^{\prime}_{fail}\) represents the state where component \(C\) fails in the next time interval, and \(s^{\prime}_{ok}\) represents the state where component \(C\) does not fail in the next time interval. The values of \(S^{\prime}_{C}\) and \(C^{\prime}_{Age}\) depend on the performed action. Applying \(Replace(C)\) in state \(s\) does not change the survival curve of \(C\), and sets the age of \(C\) to zero at time \(T_{i}\). Applying \(Fix(C)\) in state \(s\) does not change the age of \(C\) but sets \(S^{\prime}_{C}\) to be the _AfterFix_ curve of \(C\). The columns \(Fix(C)\) and \(Replace(C)\) in Table 2 provide a summary of how \(S^{\prime}_{C}\) and \(C^{\prime}_{Age}\) are set after performing a repair action in a state \(s\). \begin{table}
\begin{tabular}{l l l l} \hline \hline
**Action** & **Fix(C)** & **Replace(C)** & **\(Ignore(C)\)** \\ \hline \(S^{\prime}_{C}\) & \(S_{C}(\cdot,T_{i})\) & \(S_{C}\) & \(S_{C}\) \\ \(C^{\prime}_{Age}\) & \(C_{Age}+\Delta_{T}\) & \(\Delta_{T}\) & \(C_{Age}+\Delta_{T}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The values of \(S^{\prime}_{C}\) and \(C^{\prime}_{Age}\) in the state reached after performing an action in a state \(s=\langle T_{i_{L}}IsFaulty,S_{C},C_{Age}\rangle\). **Transition function. ** The transition function \(\mathit{Tr}(s,a,s^{\prime})\) for our MDP returns zero for any state \(s^{\prime}\
otin\{s^{\prime}_{fail},s^{\prime}_{ok}\}\). The probability of reaching \(s^{\prime}_{fail}\), i.e., the probability that \(C\) will fail in the time interval \([T_{i},T_{i+1}]\), is computed based on \(S^{\prime}_{C}\) and \(C^{\prime}_{Age}\) as follows. \[\mathit{Tr}(s,a,s^{\prime}_{fail})=\mathit{Pr}(C\ \text{ fails in }[T_{i},T_{i+1}])=S^{ \prime}_{C}(C^{\prime}_{Age}-\Delta_{T})-S^{\prime}_{C}(C^{\prime}_{Age}) \tag{5}\]  
which is a standard computation in survival analysis: the probability of surviving until \(T_{i}\) (when the age of \(C\) is \(C^{\prime}_{Age}-\Delta_{T}\)) minus the probability of surviving until \(T_{i+1}\) (when the age of \(C\) is \(C^{\prime}_{Age}\)). As noted above, the outcome of applying any action \(a\) to any non-terminal state \(s\) is either \(s^{\prime}_{fail}\) or \(s^{\prime}_{ok}\); we have that \(\mathit{Tr}(s,a,s^{\prime}_{ok})=1-\mathit{Tr}(s,a,s^{\prime}_{fail})\). **Reward function.