## 2 Partially Observable Markov Decision Process (POMDP)  
Markov decision process (MDP) is a decision-making process where a given state of the system/components is determined based on the immediate predecessor situation (state) and the action taken at this state. The MDP model consists of states (present and succeeding), action, transition probabilities (defines the probability of arriving at the different states or remaining in a given state), and reward (the cost/benefit of taking action under a given state). Even though the MDP has a wide range of applications in pipeline maintenance optimization, the approach has some limitations when applied to real-world problems. For most pipelines, detecting and identifying the defect state (corrosion deterioration, in this case) is challenging due to several factors. For example, ILI technologies are inherently inaccurate due to noise and other factors, which result in erroneous output. Pipe corrosion monitoring systems and inspection technicians can also contribute to these detection errors. The corrosion growth rate can only be estimated due to the stochastic nature of corrosion processes, which contributes to the distortion in the reading of the true deterioration state. When the underlying state cannot be accurately determined (or observed), a partially observable Markov decision process (POMDP) framework can be used instead. A POMDP uses a probabilistic observation model (or function) to relate possible observations to states. The MDP core states are broken down into a finite number of belief states, for most cases, using the observation function for the POMDP model. Actions, as well as rewards, are calculated based on these belief states when solving the problem. A POMDP model consists of states (\(S\)), which include the current state (\(s\), \(V_{s}\in S\)) and succeeding state (\(s^{\prime}\), \(V_{s^{\prime}}\in S\)), action (\(a\), \(Pa\in A\)), transition probabilities (\(P(s^{\prime},s)\), \(V_{s}\), \(s^{\prime}\in S\)), observation (\(o\), \(Fo\in O\)), observation function (\(o(s)\), \(Ps\in S\), \(Po\in O\)), belief state (\(b(s)\), \(Ps\in S\)), and reward (\(r(s\), \(a)\), \(Pa\in A\), \(Ps\in S\)). Bayes\({}^{\prime}\) rules are applied to update the probability of each belief state in each iteration (Equation (1)). The probability of being in a succeeding state \(s^{\prime}\) when observing \(o\) after taking action \(a\) (\(P(o\mid s^{\prime},a)\)) was normalized by dividing it by the observation \(o\) in all belief states for action \(a\) (\(P(o\mid b,a)\)). Multiplying this normalized value by the expected belief of transitioning from the current state to the succeeding state gives the updated belief state (\(b^{\prime}(s)\)). The new belief state is the update of the old belief (\(b(s)\)) based on the probability of observation and transition. \[b^{\prime}\big{(}s^{\prime}\big{)}=\frac{P(o|s^{\prime},a)}{P(o|b,a)}\sum_{s \in S}P\big{(}s^{\prime}|s,a)b(s) \tag{1}\]  
where  
\[P(o|b,a)=\sum_{s^{\prime}\in S}P\big{(}o\big{|}s^{\prime},a\big{)}\sum_{s\in S }P\big{(}s^{\prime}|s,a\big{)}b(s)\]  
For most POMDP problem models, the value (reward) obtained by taking an action at a given stage is computed by adding the expected reward (\(V(b)\)) for the new belief state to the discounted expected rewards from all old belief states accumulated in each iteration. Starting from some initial value, the reward from a belief of being in a succeeding state \(s^{\prime}\) multiplied by the belief was used to compute the expected marginal reward. The discounting factor (\(\gamma\)) is included to control the overall impact of older rewards in the decision-making process. The optimal value (\(V^{*}(b)\)) (action) is the maximum of the value function (\(V(b)\)) for all actions. The value function can be expressed as a piecewise-linear and convex function under a finite time horizon. This optimal solution can also be represented as a set of vectors (\(a\)-vector). The formulation for both approaches is given in Equations (2) and (3), respectively.