Optimal Planning and Learning in Uncertain Environments
for the Management of Wind Farms

Downloaded from ascelibrary.org by HUNT LIBRARY ACQ DEPT on 07/09/14. Copyright ASCE. For personal use only; all rights reserved.

Milad Memarzadeh, A.M.ASCE 1; Matteo Pozzi 2; and J. Zico Kolter 3

Abstract: Wind energy is a key renewable source, yet wind farms have relatively high cost compared with many traditional energy sources.
Among the life cycle costs of wind farms, operation and maintenance (O&M) accounts for 25–30%, and an efficient strategy for managementof turbines can significantly reduce the O&M cost. Wind turbines are subject to fatigue-induced degradation and need periodic inspections
and repairs, which are usually performed through semiannual scheduled maintenance. However, better maintenance can be achieved byflexible policies based on prior knowledge of the degradation process and on data collected in the field by sensors and visual inspections.
Traditional methods to model the O&M process, such as Markov decision processes (MDPs) and partially observable MDPs (POMDPs),
have limitations that do not allow the model to properly include the knowledge available and that may result in nonoptimal strategies for
management of the farm. Specifically, the conditional probabilities for modeling the degradation process and the precision of the observations
are usually affected by epistemic uncertainty. Although MDPs and POMDPs are formulated for fixed transition and emission probabilities,
the Bayes-adaptive POMDP (BA-POMDP) framework treats those conditional probabilities as random variables and is therefore suitable forincluding epistemic uncertainty. In this paper, a novel learning and planning method is proposed, called planning and learning in uncertain
dynamic systems (PLUS), within the BA-POMDP framework that can learn from the environment, update the distributions of model parameters, and select the optimal strategy considering the uncertainty related to the model. Validating with synthetic data, the total management
cost of a wind farm using PLUS is shown to be significantly less than costs achieved by a fixed policy or through the POMDP framework.
The preliminary results show the promise of the proposed methodology for optimal management of wind farms. DOI: 10.1061/(ASCE)CP.1943-5487.0000390. © 2014 American Society of Civil Engineers.
Author keywords: Optimal planning and learning; Sequential decision making; Wind farm management; Markov Chain Monte Carlo;
Reinforcement learning.

Introduction
Wind energy is playing an ever-increasing role worldwide as arenewable source. As a result, there will be an increasing demandfor careful management of costs associated with operation andmaintenance (O&M) of wind turbines, which on average accountfor approximately 25–30% of the overall energy generation costs(Marquez et al. 2012). To make wind energy sustainable andcompetitive with other sources, accurate risk assessment and effective management of wind farms are necessary. Farms are made upof many similar turbines, each of which is a complex system, including structural, mechanical, and electrical components; theirconditions degrade because of aging, fatigue load, and exposure
to environmental risks. Managing a wind farm includes selecting
appropriate operation and maintenance levels for the turbines,
scheduling visual inspections, and performing maintenance/repair/1

Doctoral Candidate, Dept. of Civil and Environmental Engineering,
Carnegie Mellon Univ., Porter Hall A7B, 5000 Forbes Ave., Pittsburgh,
PA 15213 (corresponding author). E-mail: miladm@cmu.edu
2Assistant Professor, Dept. of Civil and Environmental Engineering,
Carnegie Mellon Univ., Porter Hall 111, 5000 Forbes Ave., Pittsburgh,
PA 15213. E-mail: mpozzi@cmu.edu
3
Assistant Professor, School of Computer Science, Carnegie Mellon
Univ., Gates Hillman Center 7115, 5000 Forbes Ave., Pittsburgh, PA
15213. E-mail: zkolter@cs.cmu.edu
Note. This manuscript was submitted on September 22, 2013; approvedon March 4, 2014; published online on July 8, 2014. Discussion period
open until December 8, 2014; separate discussions must be submitted
for individual papers. This paper is part of the Journal of Computing
in Civil Engineering, © ASCE, ISSN 0887-3801/04014076(10)/$25.00.
© ASCE

replacement actions. A rational manager has to find a reasonable
tradeoff between a conservative maintenance policy, profitably exploiting the farm, and exploring the interaction of the turbineswith the environment. Thus, a robust decision making tool isneeded to automatically evaluate the uncertainties related to theenvironment. In this context, the overall goal is to find an optimal
policy that maximizes the total expected reward of the systemover a finite or infinite time horizon, making use of probabilistic
models for predicting the degradation of the system and the effects
of rehabilitations.
In the literature, methods based on the partially observable
Markov decision process (POMDP) framework have been recently
proposed for optimal management of wind farms (Byon et al. 2010;
Byon and Ding 2010; Nielsen and Sorensen 2012), fixing themodel parameters based on the historical data and finding the optimal policy based on them. The purpose of this paper is to address
the main limitation of POMDPs. In a POMDP framework, theprobabilities defining the state transitions and the accuracy of theobservations are fixed as if known with certainty. On the contrary,
in most real-world management problems, the transition probabilities (modeling the degradation process and the effectiveness of themaintenance actions) and the emission probabilities (modelingthe precision of sensors and visual inspection) are themselvesalso affected by large uncertainty. However, these models canbe learned from the data collected. Specifically, the frameworkof the Bayes-adaptive partially observable Markov decision process(BA-POMDP) (Ross et al. 2011) allows treatment of the statetransition and emission probabilities as random variables, whose
distribution can be learned and updated during the process ofmonitoring and management.

04014076-1

J. Comput. Civ. Eng.

J. Comput. Civ. Eng.

Downloaded from ascelibrary.org by HUNT LIBRARY ACQ DEPT on 07/09/14. Copyright ASCE. For personal use only; all rights reserved.

This paper makes two contributions: (1) a BA-POMDP modelfor O&M processes is developed that accurately captures the
epistemic uncertainties in this setting; and (2) a new algorithm
is proposed for approximate learning and planning in the
BA-POMDP framework, PLUS, which is demonstrated to performbetter than existing methods in the present setting.
The remaining parts of the paper include the notation and theprimary frameworks for sequential decision making, the proposed
methodology, the application of the methodology to a numericalexample, and conclusions.

Related Work
This section introduces the classical methodologies of robust decision making and reinforcement learning upon which the PLUS
algorithm is built.
Markov Decision Process
A fundamental model for sequential decision making is the Markov
decision process (MDP) [introductions are provided in the textbook
of Bertsekas (1996) and Sutton and Barto (1998)]. In an MDP, theenvironment is modeled as a finite set of states and actions that an
agent can take. The goal is to choose actions that maximize the total
expected reward. A typical MDP model is shown in Fig. 1. Figs. 1, 2,
and 4 are based on the “two time slice Dynamic Bayesian network” notation adopted by the textbook of Koller and Friedman

Fig. 1. MDP model(2009). The boxes with dashed edges include the variables for eachtime step. Subscript t indicates that a variable is referred to time t.
Circles refer to random variables, squares to decision variables,
diamonds to utility variables, and dots to model parameters.
Variables outside the boxes are time-independent. Shaded nodes
indicate variables observed by the agent. Arrows indicate dependencies among variables by reporting how each variable in the rightbox (i.e., at time t þ 1) is related to other variables. The notation
effectively provides a template for unrolling the model into agraphical model, where the random variables, actions, and rewards
are modeled at each time step.
An MDP is defined by a 5-tuple (S, A, T, R, γ), in which thefollowing are true:• S is a finite set of states of the system.
• A is a finite set of possible actions that an agent can take. These
actions affect the system evolution and the reward that the agent
receives.
• T∶S × A × S → ½0; 1 is a transition probability function. It
models the uncertainty in the prediction of the next state stþ1given the current state st and action at . Formally, it is definedas Tðs; a; s 0 Þ ¼ Pðstþ1 ¼ s 0 jst ¼ s; at ¼ aÞ, where PðXjYÞ
indicates the conditional probability of event X given event
Y. In the MDP, the Markov property holds the following:
given the current state of the system and the action that an
agent has taken, future states are independent of the past,so that Pðstþ1 ¼ s 0 jāt ; s̄t Þ ¼ Pðstþ1 ¼ s 0 jat ; st Þ, where s̄t ¼
fs0 ; s1 ; s2 ; : : : ; st g and āt ¼ fa0 ; a1 ; a2 ; : : : ; at g indicate thesequence of states and actions, respectively, from the beginningof the process up to time t.
• R∶S × A → R is a reward function. Based on the current state s
and the action a, the agent receives reward Rðs; aÞ. It can represent an immediate payoff (positive reward) or a cost (negative
reward).
• γ ∈ ½0; 1Þ is a discount factor that discounts the future rewards
and relates them to present value.
In the MDP framework, the agent starts in an initial state, s0 .
At any time step t, the agent observes the current state of the system, st , performs an action at , receives a reward Rðst ; at Þ, andmoves to the next state stþ1 with probability Tðst ; at ; stþ1 Þ. This
process is iterated up to a finite time horizon or indefinitely inthe so-called infinite horizon problem.
A policy, π∶S → A, is a mapping from state space to actions.
The value of a policy is the corresponding expected sum of discounted rewards when starting in some state and executing actions
according to the policy. The optimal policy π is that achieving the
maximum value. This paper focuses on the infinite horizon problem, in which the optimal policy is stationary and its value is described by Bellman’s equationXV  ðsÞ ¼ max Rðs; aÞ þ γ Tðs; a; s 0 ÞV  ðs 0 Þ
ð1Þ
a∈A

Fig. 2. POMDP model© ASCEs 0 ∈S

Optimal policy for MDPs can be identified by two classicalmethods: value iteration and policy iteration. The details of thesealgorithms can be found in Sutton and Barto (1998) and Russell
and Norvig (2010).
Recently, researchers have been trying to incorporate uncertainty in the transition probabilities of the MDP framework directlyin the formulations to find policies that are both optimal in termsof maximizing the total expected reward and robust to errors in theparameters (Bagnell et al. 2001; Iyengar 2005; Li and Si 2007; Nilim
and Ghaoui 2005). Bagnell et al. (2001) have proposed a stochastic
dynamic game to solve the problem of MDPs with uncertain transition probabilities. The proposed solution is equilibrium of the game
04014076-2

J. Comput. Civ. Eng.

J. Comput. Civ. Eng.

that corresponds to the value function under the worst model. Li andSi (2007) have proposed a new optimality criterion that is a basis fordevelopment of robust policy iteration to solve MDPs with uncertaintransition probabilities. Nilim and Ghaoui (2005) have solved theuncertain MDP problem in the context of finite and infinite horizon
using robust value iteration. The authors’ considers uncertainty in themodel but does not assume that the state is completely observable.

Downloaded from ascelibrary.org by HUNT LIBRARY ACQ DEPT on 07/09/14. Copyright ASCE. For personal use only; all rights reserved.

Partially Observable Markov Decision Process
One of the primary limitations of a MDP is that it assumes that thestate of the system is fully observable, which is not true in mostreal-world applications. A partially observable MDP (POMDP)
is a generalization of the MDP (Smallwood and Sondik 1973;
Sondik 1978). In the POMDP framework, the exact state of thesystem cannot be observed directly but can only be inferred by indirect observations. A typical POMDP model is shown in Fig. 2.
A POMDP is defined by an 8-tuple (S, A, Z, T, O, R, γ, b0 ), in
which S, A, T, R, and γ are defined as in MDP. The others aredefined as follows:• Z is a finite set of observations the agent has access to.
• O∶S × A × Z → ½0; 1 is the emission probability, which gives,
for each action and resulting state, a probability distribution
over the observations. Oðs; a; zÞ ¼ Pðzt ¼ zjst ¼ s; at−1 ¼ aÞ
defines the probability of observing z given that the agent has
taken action a and landed in state s.
• b0 ¼ Pðs0 Þ is the initial belief state.
The belief state at time t is the posterior probability of the stateof the system, given past and present observations, formally described as bt ¼ Pðst jāt−1 ; z̄t Þ, where z̄t ¼ fz1 ; : : : ; zt g indicatesthe sequence of observations. It is assumed that, in the initial stept ¼ 0, the agent does not receive any observation. The current belief state is a sufficient statistic for all past actions and observations
so that action selection can be based only on the current belief state
without loss of information. The belief state bt belongs to B, thespace of probability distributions over the states of the system S.
Belief at time t þ 1 can be updated from that at the previous stepand observation ztþ1 , using the Bayes rulePOðs 0 ; at ; ztþ1 Þ s∈S Tðs; at ; s 0 Þbt ðsÞPð2Þ
btþ1 ðs 0 Þ ¼ P
00
00
s 0 0 ∈S Oðs ; at ; ztþ1 Þs∈S Tðs; at ; s Þbt ðsÞ
A POMDP is equivalent to an MDP in the belief state, as shown
by Aoki (1965) and Astrom (1965). In the context of POMDPs, apolicy π∶B → A is a mapping from the space of belief states toactions. Bellman’s equation for optimal policy π can be formulated asXX
V  ðbÞ ¼ max
bðsÞRðs; aÞ þ γ Pðzjb; aÞV  ðb 0 Þ
ð3Þ
a∈A

z∈Z

s∈S

where distribution b 0 is updated belief btþ1 , according to Eq. (2),with a ¼ at , z ¼ ztþ1 , b ¼ bt , and conditional probabilityPðzjb; aÞ is computed as
XXPðzjb; aÞ ¼
Oðs 0 ; a; zÞ Tðs; a; s 0 ÞbðsÞð4Þ
s 0 ∈Ss∈S

In principle, a POMDP is solved by applying the methods tosolve MDPs to the belief state. However, as the belief state is a
probability distribution, it is defined on an infinite space, and soexact solution for the POMDP is not generally available. In reacting
to the observations collected, an agent can select one conditional
plan among the many available (Russell and Norvig 2010).
© ASCE

Fig. 3. Simple example of value function for two-state POMDP model(adapted from Kaelbling et al. 1998)

The conditional plan can be interpreted as a policy function definedon the domain of the sequence of observations.
The number of possible conditional plans, nc , grows exponentially with the time horizon assumed for the problem. Let αi ðsÞ
define the value of executing the ith conditional plan startingfrom perfect knowledge that the system is in state s. The valueof following
P that plan is linearly related to belief state b as
V i ðbÞ ¼ s bðsÞαi ðsÞ. Fig. 3, a graph inspired by Kaelbling et al.
(1998) that refers to a simple example of a two-state POMDP,
provides a better understanding of these concepts. Belief is completely described by a scalar value bðs1 Þ, as bðs2 Þ ¼ 1 − bðs1 Þ.
Fig. 3 reports the value for four conditional plans, and the bold line
indicates the optimal value, depending on the belief state.
The optimal value function can be written asX
V  ðbÞ ¼ max
bðsÞαi ðsÞ
ð5Þ
i

s

where i is defined on the domain f1; 2; : : : ; nc g. The proof of
Eq. (5) can be found in the work by Smallwood and Sondik(1973), which shows that the optimal value function for any finite
horizon POMDP is a piecewise-linear and convex function over thedomain B. Eq. (5) cannot be solved explicitly, except for very shorttime horizon, because of the high value of nc . However, as Fig. 3
clearly shows, some conditional plans are completely dominated(e.g., plan 4 in Fig. 3) and can be neglected (Russell and Norvig
2010). Each conditional plan begins with a specific first action, so
Eq. (5) allows defining implicitly the optimal policy π as, for any
belief state b, the optimal action is that to be executed as first one
in the optimal conditional plan.
Kaelbling et al. (1998) have proposed the so-called witness
algorithm for finding the exact solution to POMDPs through value
iteration. However, this algorithm is not practical when the set ofstates, actions, and observations are large. An alternative approachis to discretize the belief space, using either a fixed grid (Lovejoy1991) or a variable grid (Zhou and Hansen 2001). The value of any
belief is then defined by interpolation of the points on the grid.
However, in general, regular grids do not scale well in problems
with high dimensionality, and nonregular grids suffer from expensive interpolation routines. Other point-based value iteration methods restrict the search to the beliefs that can be reached startingfrom the initial belief state (Pineau et al. 2003). In particular, oneof the most effective point-based value iteration methods is successive approximations of the reachable space under optimal policies(SARSOP) (Kurniawati et al. 2008), which identifies the optimallyreachable belief states and approximates the optimal value function

04014076-3

J. Comput. Civ. Eng.

J. Comput. Civ. Eng.

using this set. SARSOP represents the state-of-the-art in solving
POMDPs in terms of efficiency and accuracy. As all algorithms
for POMDP, SARSOP formally solves the finite horizon problem,
but it can be used as an approximation to also solve the infinite
horizon case.

Downloaded from ascelibrary.org by HUNT LIBRARY ACQ DEPT on 07/09/14. Copyright ASCE. For personal use only; all rights reserved.

Bayes-Adaptive Partially Observable Markov Decision
Process
The BA-POMDP framework is a generalization of POMDP, where
the transition and emission probabilities, T and O, are unknown
components of the system and are treated as random variables, witha prior distribution PðT; OÞ. The BA-POMDP model is shown in
Fig. 4. Technically, the BA-POMDP model can be interpreted as a
POMDP with a continuous state space and with an augmented belief state that also includes T and O. The augmented belief state attime t is now defined as b~ t ¼ Pðst ; T; Ojāt−1 ; z̄t Þ. In principle, thebelief at time t can be expressed as a function of that at the previous
step, as in the POMDP formulation reported in Eq. (2). However,as in most cases, any closed-form representation of the posterior
cannot be found; in a BA-POMDP, it is easier to express the belief
at any step by integrating the joint probability:

Fig. 5. PLUS algorithmPðst ; T; Ojāt−1 ; z̄t Þ ∝ Pðz̄t ; st jT; O; āt−1 ÞPðT; OÞX¼ PðT; OÞPðz̄t ; s̄t jT; O; āt−1 Þs̄t−1 ∈StX

¼ PðT; OÞs̄t−1 ∈St



Y

×


Pðs0 Þ

Y
s;a;s 0 ∈½S×A×S
aa½Tðs; a; s 0 ÞN ss 0 ðs̄t ;āt−1 Þ½Oðs; a; zÞN sz ðs̄t ;āt−1 ;z̄t Þð6Þs;a;z∈½S×A×S

Fig. 6. PLUS learning algorithmtwhere S = set of possible sequences of states up to time t;
N ass 0 ðs̄t ; āt−1 Þ = number of times the transition ðs; a; s 0 Þ appears
in the process; and N asz ðs̄t ; āt−1 ; z̄t Þ = number of times the emission
ðs; a; zÞ appears in the process.
Learning is a challenging task in the BA-POMDP framework, as
the posterior is defined by the complicated formula in Eq. (6), and it
is not possible to compute the posterior exactly because the numberof possible sequences of states grows exponentially as the time

Fig. 7. PLUS planning algorithm

Fig. 4. BA-POMDP model© ASCE

horizon grows. It is convenient to adopt a Dirichlet distribution
for the prior distribution PðT; OÞ because that is the conjugate
of a multinomial distribution. In this context, this implies that, after
perfect observation of states s̄t , actions āt−1 , and observations z̄t ,the posterior PðT; Ojs̄t ; āt−1 ; z̄t Þ would still be in the Dirichletfamily.
Formally, the Dirichlet distribution (which is indicated as“Dir”) is the multivariate extension of the beta distribution and04014076-4

J. Comput. Civ. Eng.

J. Comput. Civ. Eng.

defines a probability density over discrete distributions. Supposeq ¼ ½q1 ; : : : ; qk T defines a k-dimensional discrete distribution.
The density can be parameterized by count vector ϕ ¼½ϕ1 ; : : : ; ϕk T , listing only non-negative entries. The Dirichlet
probability density on q is defined as

Downloaded from ascelibrary.org by HUNT LIBRARY ACQ DEPT on 07/09/14. Copyright ASCE. For personal use only; all rights reserved.

QKkΓðϕ Þ YPK iDirðqÞ ¼ i¼1
qϕi −1Γð i¼1 ϕi Þ i¼1 i
The textbook of Murphy (2012) gives an introduction to the Dirichlet distribution. In the model presented in Fig. 4, it is assumed
that s0 , T and O are marginally independent, and parameters η andβ define the count parameters of the prior Dirichlet distributions of
T and O, respectively.
The transition probability T can be represented by a threedimensional matrix (of size ½S × S × A) and the parameters ηof the Dirichlet prior probability PðTÞ by a matrix of the same
size as T. Let ηasik;sj define the counts of transitioning from si to
sj after action ak . After observing s̄t and āt−1 , the parameters of
the posterior Dirichlet distribution PðTjs̄t ; āt−1 Þ are computed asηasik;sj þ N asik;sj ðs̄t ; āt−1 Þ, i.e., by simply updating the counts withthe observed state transitions. A similar analysis can be performed
for the emission probability O. However, in the BA-POMDP
framework, states are not observed directly. The augmented belief
state, as expressed in Eq. (6), can also be derived as productPðT; Ojāt−1 ; z̄t ÞPðst jT; O; āt−1 ; z̄t Þ: the first term is the posterior
distribution of the model parameters given the observations anddoes not belong to the Dirichlet distribution. Rather, this density
is a mixture of Dirichlet distributions, in which each component
corresponds to the counts of one specific possible state sequence,
weighted by the probability of that sequence (Fruhwirth-Schnatter
2006). The next section presents the method to perform approximate updating in that context.
In the BA-POMDP framework, the optimal policy may be suboptimal for any specific value of T and O as long as it maximizes
the value for the entire state of epistemic uncertainty about themodel. Furthermore, the value of a policy also includes the benefitof taking exploratory actions that help the agent reduce uncertainty
in the model parameters themselves. Generally, a policy maps beliefs over ðs; T; OÞ to actions. This suggests that the sequential
decision problem of optimally behaving under state and model
uncertainty can be modeled as a POMDP over the augmented state,including the actual states of the system and model parameters T
and O. However, solving POMDPs over the infinite space of beliefs
over this augmented state, including continuous components, is notcomputationally feasible. The next section proposes the PLUS
algorithm, which is an approximate solution to this computational
complexity.
Jaulmes et al. (2005a, b) have proposed an algorithm called
MEDUSA to find the optimal policy for a POMDP when the model
is not known or poorly specified. The algorithm tries to improve the
POMDP incrementally using selected queries while still optimizing
the total expected reward. The next sections introduce the proposed
PLUS algorithm and compare its performance with MEDUSA andwith the use of a POMDP with fixed parameters.

it represents the augmented belief state b~ t by a set of samples,and it suggests action a . In the algorithm, notation xðkÞ indicatesthe kth sample of variable x.
PLUS Learning Phase
The PLUS algorithm makes use of an approximate method basedon Markov chain Monte Carlo (MCMC) Gibbs sampling (Carter
and Kohn 1994). The present approach is a slight variation of
the beam sampling approach used in the context of infinite hidden
Markov models (Van Gael et al. 2008) and infinite POMDPs(Doshi-Velez 2010). Fig. 6 shows the details of the proposed algorithm for learning: the method samples T, ON instances of T, O,
and belief state bt from the joint posterior distribution. The
procedure starts with sampling T, O from the corresponding prior
Dirichlet distributions, then alternate between sampling state
sequence s̄t , and sampling T and O. For each fixed T and O, a
state sequence is drawn by forward filtering backward sampling(FFBS) (Fruhwirth-Schnatter 2006), as described in the next section. In turn, as mentioned previously, the posterior distribution
given each sample s̄t is still in the Dirichlet family. Parameterset η 0 defines the updated Dirichlet distribution for the transition
probabilities, depending on sampled state sequence s̄t , whereas β 0
defines that of the emission probabilities, depending on s̄t andobservations z̄t . After appropriate burn-in phase, this proposed
method is selecting samples from the true posterior distribution.
In Fig. 6, nb indicates the number of samples in the burn-in phase
to be discarded (Murphy 2012), and the notation x ∼ p indicatesthat sample x is generated from distribution p.
Forward Filtering Backward Sampling
FFBS is a multimove sampling method for discrete systems(Fruhwirth-Schnatter 2006). The steps are as follows: (1) for eachtime step j ranging from 0 to t, derive the posterior probabilityPðsj jT; O; āj−1 ; z̄j Þ, solving the so-called filtering problem;and (2) sample state st0 from the last distribution and sj0 , fromtime step j ¼ t − 1 backward to j ¼ 0, from distribution0Fðsj Þ ∝ Pðsj jT; O; āj−1 ; z̄j ÞPðsjþ1jT; sj ; aj Þ. The outcome of
FFBS algorithm is the sequence of states fs00 ; : : : ; st0 g sampledfrom distribution Pðs̄t jT; O; āt−1 ; z̄t Þ.
PLUS Planning Phase
The planning method is based on two approximations. First,
neglect the exploratory value of learning variables T; O, i.e., thesystem model parameters. The proposed method aims at identifying
the optimal policy as that for transition and emission probabilities
modeled by PðT; Ojāt−1 ; z̄t Þ, neglecting the updating attributableto future observations. To formalize the second approximation, define QT;O ða; bÞ as the quality of a belief-state-action (Q-value) for a
POMDP, i.e., the value of starting from belief b, performing actiona, and following the optimal policy after that for a model definedby T; O. Identify the optimal action a for the overall BA-POMDP
by the following approximate formula:a ≅ argmaxET;O ½QT;O ða; bÞaProposed Methodology
An approximate method is proposed for optimally planningand learning in uncertain dynamic systems (PLUS) within the
BA-POMDP framework. Fig. 5 shows the overall PLUS method,which is organized in two main parts: learning and planning. The
algorithm can be called at any stage of the process. At time t,© ASCE

ð7Þ

where Ex indicates the statistical expectation, according to actual
knowledge of variable x; and the belief state at time t is defined
as in a POMDP as b ¼ Pðst jT; O; āt−1 ; z̄t Þ. Eq. (7) represents
an approximation because it combines quantities related to optimal
policies for different models. However, do not use the approximation to estimate the value of the policy but only to select the current
optimal action. Computationally, the advantage of Eq. (7) is that

04014076-5

J. Comput. Civ. Eng.

J. Comput. Civ. Eng.

QT;O ða; bÞ can be obtained from the results of a POMDP solver,i.e., SARSOP.
The Q-value of a belief-state-action can be related to the
α-vectors presented in the POMDP section. For a model T; O
and belief b, the optimal conditional plan starting with action afor each available action can be identified. Define αa;b;T;O ðsÞ as
the component referring to state s of the corresponding α-vector,
and the Q-value of a belief-state-action can be computed as
X
QT;O ða; bÞ ¼bðsÞαa;b;T;O ðsÞ
ð8Þ
Downloaded from ascelibrary.org by HUNT LIBRARY ACQ DEPT on 07/09/14. Copyright ASCE. For personal use only; all rights reserved.

sFig. 7 presents the scheme of the planning algorithm, which isbased on Eqs. (7) and (8). At time t, after the learning phase, augmented belief state b~ t is represented by N samples. The next step isto solve the corresponding POMDP problem for each sample, using
SARSOP (Kurniawati et al. 2008). The outcome of SARSOP is theset of m nondominated α-vectors. Among them, the algorithm selects one optimal α-vector per each action: this is the pruning routine mentioned in the algorithm. αj refers to the optimal vector forðkÞ
the jth action; Qj to the Q-value of a belief-state-action for the kth
sampled model under the jth action; and Qj to the expected
Q-value of a belief-state-action for the entire model space, whichis computed by sample average. Action a is selected by identifying the maximum of Qj among all possible actions.

Numerical Validation
To validate the proposed methodology, a numerical example ofwind farm management is used. It is assumed that the condition
state of each turbine can be modeled by a Markov process
defined by a few states, and the observations collected can beclassified within a few possible discrete values. Although PLUS
can be applied to much more complicated problems, this simple
setup allows the extensive investigation of the performance of
the algorithm so it can be compared to other existing methods.
The condition state of the turbine degrades owing to fatigue andaging, potentially causing a structural failure and a relevant economical loss to the agent. In turn, the agent can perform repairs
to avoid failures and inspections to refine the knowledge about eachcondition state. In detail, the farm is assumed to consist of 10 turbines of the same type, so that a unique value of transition andemission probabilities can be referred to. The cost in Figs. 8, 9,
and 12 refers to the average per one turbine. Specifically, three condition states are assumed: s ¼ 1 refers to an intact structure; s ¼ 2
to a damaged one; and s ¼ 3 to a collapsed turbine. The threeactions are the following: a ¼ 1 corresponds to do nothing (DN);
a ¼ 2 to repair (RE); and a ¼ 3 to performing a visual inspection(VI). When DN is selected, the condition state evolves according
to the degradation process. RE models a costly intervention, which
is supposed to improve the condition state, whereas VI models aneffort providing only information on the condition state, without
affecting the degradation process. Each time step is assumed tobe six months, and the agent takes one action per turbine at eachtime step.
Observations are classified in four discrete outcomes: z ¼ 1 isintended as a reassuring output, suggesting that the turbine isundamaged; z ¼ 2 and z ¼ 3 indicate two symptoms of damage;after recording z ¼ 4, the agent knows that the turbine is collapsed.
As indicated in “Proposed Methodology,” the agent’s priorknowledge is modeled on transition and emission probabilities
by independent Dirichlet distributions with parameters η and β,
respectively. Parameter η can be represented by three matrices:ηDN , ηRE , and ηVI , referring to the actions listed above© ASCE

2

8

4 2

0

0 1

6
ηDN ¼ ηVI ¼ 4 0

2

3

7
4 25

8 4

0

4 2

0

6
ηRE ¼ 4 4 2

3

7
05

The transitions are assumed to be identical for actions DN
and VI. The zeros in the matrix ηDN indicate that after any of theseactions, the condition state cannot improve; therefore, for example,
the turbine stays in a collapsed state after action DN. Generally,
according to this matrix, the turbine in the intact state has a tendency to stay undamaged, but it can also become damaged or
directly collapse, whereas a turbine in the damaged state has a tendency to stay there, but it can also collapse. After action RE, theturbine cannot be in a collapsed state, but it can still be damaged
because the intervention is not known to be perfect and, even after
a perfect repair, the turbine can transit to the damage state during
the following period, considering the long time step (six months).
As for any feature of the process, the effectiveness of such an intervention can be learnt by the agent during the management history. Knowledge about emissions, depending on the action, aremodeled by the following values:
232
3
8 4 2 04 2 0 067
6
7
β DN ¼ β RE ¼ 4 2 8 4 0 5
β VI ¼ 4 0 2 4 0 5
0

0 0

1

0

0 0

1

As can be deduced from these matrices, the agent thinks that,as a tendency, States 1 and 2 generate Observations 1 and 2, respectively, under actions DN or RE. The visual inspection VI isregarded as possibly imperfect; again, its actual effectiveness canbe discovered during the management process. Independent of theaction, the collapse State 3 is univocally related to Observation 4,
so that the agent is immediately aware of any failure event.
The reward function is the sum of three components: thecosts for repairing, inspecting, and down-time. The agent pays
US$10,000 for any repair, $500 for any visual inspection, and$50,000 for any time step in which a turbine is in the collapsed
state. The discount factor is assumed to be γ ¼ 0.95.
The belief about the initial state is modeled asb0 ¼ ½ 0.8

0.2 0 therefore, the agent believes that the turbines are in the intact statewith 80% probability and in damaged state with 20% probability.
The behaviors of different turbines in the farm are assumed to
be independent, and the agent refers the planning to the infinite
horizon setting.
The method is validated by two sets of numerical experiments.
First, transition and emission were fixed to a value compatible
with the available knowledge, referring to this as the true model.
The true model was assigned to each turbine in the farm, and theperformance of alternative schemes for learning and planning were
simulated. Next, the planning algorithm was tested for the range ofall possible models representing the turbines.
In both experiments, four types of agents were considered: Thetrue model agent has perfect knowledge about the true underlying
transition and emission probabilities and adopts a POMDP modelwith correct value for T and O, making use of the SARSOP algorithm for planning: this represents an lower bound (in terms of costto be minimized) to the performance of any planning strategy under
uncertainty. The expected model agent derives the expected valueof T and O from the prior Dirichlet distribution and again, adopts
POMDP solved by SARSOP: it represents the simplest and mostcommon approach to solve the planning problem under model

04014076-6

J. Comput. Civ. Eng.

J. Comput. Civ. Eng.

where ln = natural logarithm. In computing the KL divergence
between two transition (or emission) models, the results referringto the average over all values of st and at .
Learning and Planning Validation
The first numerical campaign is devoted to the validation of theplanning and learning algorithms. To do so, a model is fixed andassigned to all turbines. This is called the true model, and it is defined by transition T  and emission O , as listed in the following:
2
2
3
3
0.9 0.08 0.02
1
0 0
6
6
7
7
T DN ¼ T VI ¼ 4 0
0.9 0.1 5
T RE ¼ 4 0.9 0.1 0 5
2

0

0

1

3

0.8 0.1 0.1 0
7
6ODN ¼ ORE ¼ 4 0.05 0.9 0.05 0 5
0

0

0

1

0.9 0.1 0
3
2
1 0 0 0
7
6
OVI ¼ 4 0 0 1 0 5
0 0 0 1This specific model describes a turbine that is more reliable thanthat defined by the expected value of the distribution reported in theprevious section. These models were selected by adapting examples from the literature (Byon et al. 2010; Byon and Ding 2010;
Nielsen and Sorensen 2012) after discussion with industry experts
from EverPower Wind Holdings (Pittsburgh, PA). For example, theprobability of a collapse in one 6-month period, for an intact turbine, is only 2%. The emissions related to the visual inspection
models perfect information on the condition state.
For each agent, the management of the wind farm is simulated
20 times, and the average outcome is plotted in Fig. 8. In each simulation, the initial state is sampled according to the distribution b0 .
Fig. 8(a) reports the average immediate cost versus the time step.
The dashed line represents the true model agent, the line represents
the expected model agent, and the dash-dotted line represents the
PLUS agent, whereas other lines refer to the MEDUSA algorithm,
with a learning rate (LR) of 0.1, 0.5 and 1.
Each agent starts with a low cost in the first steps owing to thegood state of the turbines as assumed by the initial belief state. The
true model and the expected model agents adopt a stationary policy,
and the corresponding immediate cost converges to a constantvalue, which is approximately $2,200/6 months for the former,
and $3;500=6 months for the latter agent. Fluctuations are attributable to randomness in the average of the small set of simulations.
Agents adopting the MEDUSA and the PLUS algorithm, on the© ASCE

Average Cost of O&M pet Time Step (K$)

4.5
4
3.5
3
2.5
2
1.5
True Model
POMDP
MEDUSA LR 0.1
MEDUSA LR 0.5
MEDUSA LR 1.0
PLUS

1
0.5
0
0

10

20

30

40

50

60

70

80

90

100

60

70

80

90

100

Time Step(a)350

Cumulative Cost of O&M (K$)

Downloaded from ascelibrary.org by HUNT LIBRARY ACQ DEPT on 07/09/14. Copyright ASCE. For personal use only; all rights reserved.

uncertainty. The MEDUSA agent makes use of the algorithm described in Jaulmes et al. (2005a, b), whereas the PLUS agent adopts
the method that was presented in “Proposed Methodology.”
Two different metrics are used to validate the methods. First,
the immediate and cumulative management cost for assessing
the performance of the planning methods is evaluated because they
are directly related to what each agent is trying to optimize. For
additional validation of the learning process itself, evaluate theKullback-Leibler (KL) divergence (Cover and Thomas 2006) between the transition (or emission) probabilities as modeled by theposterior distribution and in the true model. The KL divergence is anonsymmetric measure of the differences between two probability
distributions. Specifically, the KL divergence of distributions Q
from distribution P (both are distributions defined on n discretevalues), denoted as DKL ðPjjQÞ, is a measure of information lostwhen Q is used to approximate P, and is defined as


n
X
PðiÞ
PðiÞ
ð9Þ
DKL ðPjjQÞ ¼
ln
QðiÞ
i¼1

True Model
POMDP
MEDUSA LR 0.1
MEDUSA LR 0.5
MEDUSA LR 1.0
PLUS

300
250
200
150
100
50
0
0

(b)

10

20

30

40

50

Time Step

Fig. 8. Costs for O&M of a wind farm versus time for six agents:(a) immediate; (b) cumulative

other hand, adopt nonstationary policies because of the learning
process. At each time, the knowledge about the model is affectedby processing the previous observations, and the policy varies accordingly. Ideally, if sufficient information is collected, the policies(and consequently the immediate cost) of these agents should converge to that of the true model agent. As expected, it is apparent
from the figure that the immediate cost grows in the first phase(i.e., the first 10–20 steps) and then is reduced in time becauseof the effect of learning. The PLUS algorithm also performs wellin the first phase because of the robust algorithm for planning. After
30 steps, the immediate cost is approximately $2,600=6 months.
In this simulation, the MEDUSA algorithm achieves a higher cost
for a range of different learning rates. The benefit of the PLUS
algorithm over the expected model approach can be quantified
as approximately $1,000=6 months.
Fig. 8(b) shows the cumulative costs of O&M, computed as theintegral in time of the curves plotted in Fig. 8(a). This representation is useful for assessing the long-term benefit of adopting alternative schemes. In a 100-step period (corresponding to 50 years),
the true model agent expects a cost of approximately $220,000; theexpected model agent a cost of approximately $350,000, whereas
the PLUS agent expects a cost of approximately $250,000. Thus,
the benefit of adopting PLUS is quantifiable to approximately$100,000 for this period. These costs and savings are for a single
turbine, and the costs and savings regarding the entire farm is ten
times higher.
Fig. 9 shows the cumulative costs for O&M of a wind farmfor PLUS, true model, and POMDP agents, including the 95%
confidence intervals.

04014076-7

J. Comput. Civ. Eng.

J. Comput. Civ. Eng.

Transition Probabilities

400
0.22
0.2

300
0.18

250

0.16

KL Divergence

Cumulative Cost of O&M (K$)

350

200
150
100
True Model
POMDP
PLUS

0
0

10

20

30

40

50
60
Time Step

70

80

90

0.12
0.1
0.08
PLUS
MEDUSA LR 0.1
MEDUSA LR 0.5
MEDUSA LR 1.0
Expected Model

0.06

100

0.04

Fig. 9. Cumulative costs for O&M of a wind farms versus time forthree agents, including the 95% confidence intervals

0.02

0

20

40

(a)

60

80

100

80

100

Time Step

Observation Probabilities
0.22

Fig. 10 focuses on the learning process, showing the evolution of the KL divergence between the posterior distribution of
the model, as formulated by each agent, and the true model.
Fig. 10(a) plots the transition probabilities and Fig. 10(b) theemission probabilities. The expected model agent does not learn
during the process; and consequently, the KL divergence is constant. The agents using MEDUSA or PLUS update their knowledge during the management process, and the KL divergence is
expected to go to zero when the information encoded in the collected observations is sufficient to identify the model. For theseagents, the KL divergence is computed as the average from a
set of samples generated according to the posterior distribution(as illustrated in “PLUS Learning Phase,” PLUS algorithm requires the generation of samples, so this further computation isstraightforward). This study used 10 samples in this simulation.
As shown in the figure, the learning is fast in the initial phase,
but it becomes slow as more and more observations have been
already collected. According to this simulation, the MEDUSA
agents learn the transition probabilities well but not the emission
probabilities [Fig. 10(b)]. MEDUSA learns the emission probabilities poorly, perhaps because of the different planning approachcompared with PLUS, and may need more data. However, inthe long run, provided that sufficient exploration is performed,
MEDUSA is conjectured to asymptotically learn the true model.
Generally, MEDUSA and PLUS are different in terms of the
tradeoff between computational cost and accuracy: MEDUSA iscomputationally cheaper and easier to scale; however, it provides
less accurate solutions compared with PLUS.
Fig. 10 shows that initially the KL divergence of the expectedmodel agent is lower than that of the PLUS agent. This is a random effect owing to the selection of the true model in thissimulation. The expected model agent adopts the mean transitionand emission. Depending on the actual model of the turbine, it
may be the case that the KL divergence can be arbitrarily small,and possibly much smaller than that of the PLUS agent. In otherwords, it may be the case that the model assumed by the expectedmodel agent is actually the correct one; and therefore, no learningis needed. Generally, the performance of the alternative methods
depends on the specific actual model. In the next section, a validation of the planning algorithm is performed for all possiblemodels.
Fig. 11 shows the same results in Fig. 10(a), including the
95% confidence intervals for the learning process of PLUS agent.
© ASCE

0.2
0.18
0.16

KL Divergence

Downloaded from ascelibrary.org by HUNT LIBRARY ACQ DEPT on 07/09/14. Copyright ASCE. For personal use only; all rights reserved.

50

0.14

0.14
0.12
0.1
0.08
PLUS
MEDUSA LR 0.1
MEDUSA LR 0.5
MEDUSA LR 1.0
Expected Model

0.06
0.04
0.02

(b)

0

20

40

60

Time Step

Fig. 10. Performance of the proposed learning methodology (PLUS)
compared with MEDUSA (with different learning rates) and POMDP(does not involve learning); the graphs show the KL divergence
between each model and the true model: (a) transition; (b) emission
probabilities

Planning Approach Validation
The outcomes of the previous section highlight that the PLUS
algorithm outperforms the expected model agent. It is possible,
however, that this benefit derives only from the learning process.
The second experiment aims specifically to validate the planningalgorithm only, removing the learning effect. The expected benefitof a method cannot be assessed by referring only to a single model,
so in this campaign a set of samples are drawn from the priordistributions of models, and the performance of the methods areaveraged across these scenarios.
Fig. 12 reports the immediate and cumulative costs of O&M,
for the true model, the expected model, and the PLUS agents.
Again, the true model agent represents a lower bound, leading to an immediate cost of approximately $2,900=6 months,
whereas the expected model agent achieves a cost of approximately $8,300=6 months, and the PLUS agent a cost of approximately $7,700=6 months. The difference between these latter

04014076-8

J. Comput. Civ. Eng.

J. Comput. Civ. Eng.

Transition Probabilities

0.22

experiment highlights the value of uncertainty-aware planningin and of itself. The costs and savings regarding the entire farm
are ten times higher.

PLUS

0.2

POMDP

KL Divergence

0.18
0.16

Conclusion

0.14
0.12
0.1

0.06
0.04

0

10

20

30

40

50

60

70

80

90

100

Time Step

Average Cost of O&M per Time Step (K$)

Fig. 11. Performance of the proposed learning methodology (PLUS)
compared with POMDP (does not involve learning) including the 95%
confidence intervals; the graphs show the KL divergence between eachmodel and the true model parameters

9
8
7
6
5
4
3
2
True Model
POMDP
PLUS

1
0
0

10

20

30

(a)

40

50

60

70

80

90

Acknowledgments

100

Time Step

900
True Model
POMDP
PLUS

800
Cumulative Cost of O&M (K$)

Downloaded from ascelibrary.org by HUNT LIBRARY ACQ DEPT on 07/09/14. Copyright ASCE. For personal use only; all rights reserved.

0.08

A method named PLUS is proposed for learning and planningwithin the BA-POMDP framework and applicable to the context
of wind farm management. The BA-POMDP framework overcomes one of the primary limitations of the POMDP framework
by treating the transition and emission probabilities as random variables, whose distributions can be updated during the learning
process. The PLUS algorithm uses Markov chain Monte Carlosimulations to find an approximate solution for the BA-POMDP
problem. The approach allows for a rational treatment of data collected by sensors and visual inspections, a reliable tracking of thecondition states of turbines, and robust decision-making support.
The PLUS algorithm has been validated with synthetic data andis shown to out-perform state-of-the-art reinforcement learning approaches, such as MEDUSA. MEDUSA was originally proposed
for applications of robot navigation and it scales easier than PLUS,
requiring less computational effort. However, for application towind farms, it is believed that the computational drawback of PLUS
is not a significant concern because the computational cost is low
with respect to the direct costs for operation and maintenance of awind farm. On the contrary, in this context it is necessary to achievea rational and robust selection of the management policy, making
use of the knowledge available at any state of the process. PLUS
allows this; it also allows the agent to learn, during the management, the statistics of the degradation process (transition probabilities) and the performance and reliability of the monitoring system(emission probabilities).

700

The authors would like to thanks Stephane Ross for his valuable
discussion on the BA-POMDP framework and Ada Zhang and
Jingwei Zhang for their valuable discussion and contribution.
The authors would like to also especially thank EverPower Wind
Holdings for invaluable collaboration.

600
500

References

400

Aoki, M. (1965). “Optimal control of partially observable Markov systems.” J. Franklin Inst., 280(5), 367–386.
Astrom, K. J. (1965). “Optimal control of Markov decision processes withincomplete state estimation.” J. Math. Anal. Appl., 10(1), 174–205.
Bagnell, A., Ng, A., and Schneider, J. (2001). “Solving uncertain Markov
decision processes.” Proc., Neural Information Processing Systems,
Carnegie Mellon Univ., Pittsburgh, PA.
Bertsekas, D. P. (1996). Dynamic programming and optimal control,Vol. 1–2, Athena Scientific, Belmont, MA.
Byon, E., and Ding, Y. (2010). “Season-dependent condition-based maintenance for a wind turbine using a partially observed Markov decision
process.” IEEE Trans. Power Syst., 25(4), 1823–1834.
Byon, E., Ntaimo, L., and Ding, Y. (2010). “Optimal maintenance strategies
for wind turbine system under stochastic weather conditions.” IEEE
Trans. Reliab., 59(2), 393–404.
Carter, C. K., and Kohn, R. (1994). “On Gibbs sampling for state spacemodels.” Biometrika, 81(3), 541–553.
Cover, T. M., and Thomas, J. A. (2006). Elements of information theory,
Wiley, New York.

300
200
100
0

0

(b)

10

20

30

40

50
60
Time Step

70

80

90

100

Fig. 12. Planning performance of the PLUS algorithm compared with
POMDP and true model agents; costs for O&M of a wind farm versus
time: (a) immediate; (b) cumulativevalues, i.e., $600=6 months, quantifies the benefit of adopting therobust planning approach presented in “PLUS Planning Phase” forthe specific example. Naturally, adding the learning process as wellwould make PLUS perform much closer to the true model, but this© ASCE

04014076-9

J. Comput. Civ. Eng.

J. Comput. Civ. Eng.

Downloaded from ascelibrary.org by HUNT LIBRARY ACQ DEPT on 07/09/14. Copyright ASCE. For personal use only; all rights reserved.

Doshi-Velez, F. (2010). “The infinite partially observable Markov decision
process.” Proc., Neural Information Processing Systems, Vol. 22,
477–485.
Fruhwirth-Schnatter, S. (2006). Finite mixture and Markov switchingmodels, Springer, New York.
Iyengar, D. (2005). “Robust dynamic programming.” Math. Oper. Res.,30(2), 257–280.
Jaulmes, R., Pineau, J., and Precup, D. (2005a). “Active learning in partiallyobservable Markov decision processes.” European Conf. on Machine
Learning, Porto, Portugal, 601–608.
Jaulmes, R., Pineau, J., and Precup, D. (2005b). “Learning in nonstationary partially observable Markov decision processes.” European
Conf. on Machine Learning Workshop on Reinforcement Learning in
Non-Stationary Environments, Porto, Portugal.
Kaelbling, L. P., Littman, M. L., and Cassnadra, A. R. (1998). “Planningand acting in partially observable stochastic domain.” J. Artif. Intell.,101(1–2), 99–134.
Koller, D., and Friedman, N. (2009). Probabilistic graphical models,MIT, Cambridge, MA.
Kurniawati, H., Hsu, D., and Lee, W. (2008). “SARSOP: Efficient pointbased POMDP planning by approximating optimally reachable belief
spaces.” Robotics: Science and Systems, ETHZ, Zurich, Switzerland.
Li, B., and Si, J. (2007). “Robust dynamic programming for discounted
infinite horizon Markov decision processes with uncertain stationarytransition matrices.” IEEE Symp. on Approximate Dynamic Programming and Reinforcement Learning, Honolulu, HI, 96–102.
Lovejoy, W. S. (1991). “Computationally feasible bounds for partiallyobserved Markov decision process.” Oper. Res., 39(1), 162–175.
Marquez, F. P. G., Tobias, A. M., Perez, J. M. P., and Papaelias, M. (2012).
“Condition monitoring of wind turbines: Techniques and methods.”
Renewable Energy, 46, 169–178.

© ASCE

Murphy, K. P. (2012). Machine learning: A probabilistic perspective,
MIT, Cambridge, MA.
Nielsen, J. S., and Sorensen, J. D. (2012). “Maintenance optimization foroffshore wind turbines using POMDP.” Proc., 16th Conf. of Int. Federation for Information Processing on Reliability and Optimization of
Structural Systems, 175–182.
Nilim, A., and Ghaoui, L. E. (2005). “Robust solutions to Markov
decision problems with uncertain transition matrices.” Oper. Res.,
53(5), 780–798.
Pineau, J., Gordon, G., and Thrun, S. (2003). “Point-based value iteration:
An anytime algorithm for POMDPs.” Proc., Int. Joint Conf. on Artificial Intelligence, Acapulco, Mexico.
Ross, S., Pineau, B., Chaib-draa, B., and Kreitmann, P. (2011). “A Bayesian
approach for learning and planning in partially observable Markov
decision process.” J. Mach. Learn. Res., 12, 1729–1770.
Russell, S. J., and Norvig, P. (2010). Artificial intelligence: A modern
approach, 3rd Ed., Prentice Hall, Upper Saddle River, NJ.
Smallwood, R. D., and Sondik, E. J. (1973). “The optimal control of partially observable Markov processes over a finite horizon.” Oper. Res.,
21(5), 1071–1088.
Sondik, E. J. (1978). “The optimal control of partially observable Markov
processes over the infinite horizon.” Oper. Res., 26(2), 282–304.
Sutton, R. S., and Barto, A. G. (1998). Reinforcement learning: An introduction, MIT, Cambridge, MA.
Van Gael, J., Saatchi, Y., The, Y. W., and Ghahramani, Z. (2008). “Beamsampling for infinite hidden Markov model.” Int. Conf. on Machine
Learning, Helsinki, Finland, 25.
Zhou, R., and Hansen, E. A. (2001). “An improved grid-based approximation algorithm for POMDPs.” Proc. Int. Joint Conf. on Artificial
Intelligence, Seattle, Washington.

04014076-10

J. Comput. Civ. Eng.

J. Comput. Civ. Eng.