# A hybrid approach to solve the maintenance problem of an offshore wind farm formulated as a large-scale POMDP

Matthieu Roux, Yiping Fang, Anne Barros

► To cite this version:

Matthieu Roux, Yiping Fang, Anne Barros. A hybrid approach to solve the maintenance problem of an offshore wind farm formulated as a large-scale POMDP. 24ème congrès annuel de la Société Française de Recherche Opérationnelle et d'Aide à la Décision, Feb 2023, Rennes, France. hal-04013826

HAL Id: hal-04013826

https://hal.science/hal-04013826

Submitted on 3 Mar 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A hybrid approach to solve the maintenance problem of an offshore wind farm formulated as a large-scale POMDP

Matthieu Roux, Yi-Ping Fang, Anne Barros

Univ. Paris-Saclay, CentraleSupélec, LGI EA 2606

Chair Risk and Resilience of Complex Systems

{matthieu.roux,yiping.fang,anne.barros}@centralesupelec.fr

**Keywords** : complex industrial system, Partially Observable Markov Decision Process, maintenance optimization, imperfect monitoring, large-scale POMDP, offshore wind farm

## 1 Introduction

For a couple of decades, we have observed a growing interest in condition-based maintenance (CBM) strategies, a framework in which maintenance decisions are based on the current degradation level of the items [1]. In this work, we seek to optimize the CBM policy of a distributed system where inspections and remote sensors can inform on the current degradation level. The system is composed of many units that function and degrade independently. They are monitored continuously by remote sensors and occasionally inspected. The sensors imperfectly estimate the item's state at a low cost, whereas inspections reveal perfectly the true state at a higher cost. We model the problem as a partially observable Markov decision process (POMDP). For that, time is discretized and each unit is characterized by a discrete degradation state $s_t \in S$. Sensors provide imperfect observations $o_t \in O$, where imperfection in the monitoring process is modeled by the conditional probabilities $P(o_t = o|s_t = s)$. Eventually, at the beginning of each observation epoch (a group of $T$ time steps), one observation $o_t$ is used to update a belief $b_t$ representing the probability that the item is in each state.

We study, in particular, the optimization of maintenance operations in the offshore wind turbine industry and propose an extension of [4] to the case of mixed sources of condition monitoring information. However, although researchers recently made significant progress in POMDP solvers [2], the obtained POMDP remains way too large to be solved efficiently. Such complexity is due to the combinatorial explosion in the state and action spaces when having multiple units in the system. Our main contribution is to propose a novel and efficient hybrid heuristic, mixing dynamic programming (DP) and integer linear programming (ILP), to overcome the curse of dimensionality. We also analyze the impact of imperfect monitoring on optimal maintenance policies, extending a previous work [3].

## 2 Problem description

At the beginning of each observation epoch, imperfect observations $o_t$ from the remote sensors are collected and lead to update the belief $b_t$ using Bayes' formula. For each time step of the observation epoch and each unit of the system, we should select one maintenance action among **NA** (do nothing), **PM** (preventive maintenance), **CM** (corrective maintenance) and **I** (perfect inspection). Each action is associated with a cost and requires a certain amount of resources (e.g., number of technicians). An opportunity cost is also incurred each time a unit remains failed (e.g., electricity not produced). The objective is to minimize the total maintenance cost over a finite time horizon $H$ (with $H \gg T$) while respecting a resource constraint.

Since deploying maintenance crews is an expensive operation, there is a solid incentive to group interventions together (opportunistic maintenance). Such a system-level cost, combined
with a resource limitation at each time step, explains why the problem cannot be easily decomposed and solved item per item. As a result, the maintenance policy should be optimized and coordinated at the scale of the system, which justifies the need for a scalable solving method capable of handling a system composed of 20 to 200 units.

## 3 Hybrid heuristic approach

The maintenance planning is optimized sequentially. At each observation epoch, we solve the following ILP (1) to schedule the interventions that should be conducted within the epoch (i.e., for $t \in \{0, T - 1\}$). We split the set of units $I$ into $I_w$ (working units) and $I_f$ (failed units). Binary variables $z_{i,t}^a$ indicate whether the maintenance action $a$ should be performed on item $i$ at time $t$; $x_t$ is a binary variable indicating whether the maintenance crew should be deployed for at least one intervention at time $t$.

To ensure this framework does not result in a repair-at-failure policy, where PM would never be conducted because of cost minimization, we carefully design the objective function to balance costs from immediate decisions with expected future costs (close to the concept of value function from DP). To do so, we introduce the functions $C^a(b, t)$. They estimate the expected future costs resulting from the schedule of intervention $a \in \{\mathbf{PM}, \mathbf{CM}, \mathbf{I}\}$ at time $t$ for a unit that is, at $t = 0$, in a state described by the belief $b$. In a sense, it is a heuristic estimation of the added value that can be expected from any available action. In addition, the function $C^{NA}(b)$ estimates the future costs resulting from the choice not to schedule any intervention within the current epoch. This gives us the following objective function to minimize:

$$ \min_{x,z \in \{0,1\}} \sum_{t=0}^{T-1} cost_{deploy} \cdot x_t + \sum_{i \in I_w} \left[ \left( \sum_{t=0}^{T-1} C^{PM}(b_i, t) \cdot z_{i,t}^{PM} + C^I(b_i, t) \cdot z_{i,t}^I \right) + C^{NA}(b_i) \cdot z_i^{NA} \right] + \sum_{i \in I_f} \sum_{t=0}^{T-1} C^{CM}(t) \cdot z_{i,t}^{CM} \quad (1) $$

The critical element of our method consists in approximating those expected future costs $C^a(b, t)$ using the Q-function obtained when solving an appropriate and well-chosen single unit POMDP. We numerically validate this approach on a numerical use case, a wind farm containing 50 offshore turbines, and show that, although sub-optimal, our method performs significantly better than two alternative approximations. In addition, the fact that the proposed decomposition method can exploit the value of information coming from imperfect condition monitoring further confirms its relevance. Eventually, this is also encouraging as it computes quite efficiently the policy for a relatively large system, for which using a traditional POMDP solver would have been impossible.

## References

[1] Alaswad, S., & Xiang, Y. (2017). A review on condition-based maintenance optimization models for stochastically deteriorating system. *Reliability Engineering & System Safety*, 157, 54-63.

[2] Papakonstantinou, K. G., & Shinozuka, M. (2014). Planning structural inspection and maintenance policies via dynamic programming and Markov processes. Part I: Theory. *Reliability Engineering & System Safety*, 130, 202-213.

[3] Roux, M., Fang, Y. P., & Barros, A. (2022). Impact of imperfect monitoring on the optimal condition-based maintenance policy of a single-item system. In *ESREL 2022*.

[4] Yildirim, M., Gebraeel, N. Z., & Sun, X. A. (2017). Integrated predictive analytics and optimization for opportunistic maintenance and operations in wind farms. *IEEE Transactions on Power Systems*, 32(6), 4319-4328.