## 4 POMDP Model Formulations  
A POMDP model comprises the following elements (Figure 2):  
* _States_ (S)--defines a system's condition (core state) in a dynamic environment. All possible states for the system are collected into a set (\(S\)). The two important states for analysis are the current state (\(s\)) and succeeding state (\(s^{\prime}\)). * _Action_ (\(A\))--is a collection of decisions (action--\(a\)) taken based on the state of the system (defined as a set--\(A\)). * _Transition probabilities_ (\(T(s^{\prime}\mid s,a)\)--is a matrix that defines the probability of landing on a specific succeeding state (\(s^{\prime}\)) from a given current state (\(s\)) after taking action (\(a\)). * _Reward_ (\(R(s,a)\))--is a function that defines the cost (or reward) of taking a given action (\(a\)) on a system in a given current state (\(s\)). * _Observation_ (\(o\)) and _observation probability_ (\(P(o\mid s,a)\)) have been added as elements in the model formulation. The _observation_ is any description that can be obtained through a different mechanism and used to determine the actual state of the system. It is defined as a set (\(O\)). _Observation probability_ is a matrix that defines the probability of observing \(o\) for a system in state \(s\) and after taking action \(a\). * _Belief state_ (\(b(s)\)) is created to replace the states defined in the MDP model. The analysis of POMDP models uses these belief states, which define the degree (probability) of finding the system in a given MDP state (core state). The belief states of a POMDP model can be formulated as a finite number of states (discrete state POMDP--DSPOMDP) or continuous states using different functions (CSPOMDP). Both formulation approaches can help attain deeper core state analysis. The continuous function formulation of the belief state has been extended to the _observation probability_ and _action_, which is collectively called continuous space POMDP. For a large number of (continuous) _states_, _observations_, and _actions_, continuous space modeling methods can be more effective even though finding optimal results could be very difficult due to model complexity (refer [26; 27] for detailed discussion). In DSPOMDP, the system transitions into a succeeding state based on the transitional probability, action, and current state. The system's succeeding belief state is obtained through a belief-updating computation using Bayes' rules. For a CSPOMDP, the update is computed by integrating the expected belief states rather than summing, which is the case for discrete formulation [27; 28]. _P(o!s^{\prime},a\)_ was normalized by dividing it with the probability of observing \(o\) under known action a and current belief state \(b(s)\) (\(P(o\,|\,b,a)\)). This normalizing factor was computed by adding the expected probability of the observation \(o\) for all succeeding states based on the current belief state. The new belief state (\(b^{\prime}\) (\(s^{\prime}\))) is computed by multiplying this normalized value by the expected belief of transitioning from the current state to the succeeding state, an integral function (Equation (1)). An MDP model with infinite dimensional core states (to constitute the belief states) can represent a CSPOMDP [27]. \[b^{\prime}(s^{\prime})=\frac{P(o|s^{\prime},a)}{P(o|b,a)}\int_{s\in S}P\big{(} s^{\prime}|s,a\big{)}b(s) \tag{1}\]  
where  
\[P(o|b,a)=\sum_{s^{\prime}\in S}P\big{(}o|s^{\prime},a\big{)}\sum_{s\in S}P \big{(}s^{\prime}|s,a\big{)}b(s)\]  
The total reward (\(V(b)\)) is computed at each step of POMDP. For a given action, it is the sum of the integral of the expected marginal reward from a belief state and the discounted expected rewards of all preceding belief states. The discounting factor (\(\gamma\)) controls the impact of older belief states values on the latest value. The optimal value (\(V*(b)\)) is determined by comparing the values obtained from all actions (Equation (2)).