# A 2D VMD video image processing-based transfer learning approach for the detection and estimation of biofouling in tidal stream turbines

Houssem Habbouche, Haroon Rashid, Yassine Amirat, Arindam Banerjee, Mohamed Benbouzid

► To cite this version:

Houssem Habbouche, Haroon Rashid, Yassine Amirat, Arindam Banerjee, Mohamed Benbouzid. A 2D VMD video image processing-based transfer learning approach for the detection and estimation of biofouling in tidal stream turbines. Ocean Engineering, 2024, 312, pp.119283. 10.1016/j.oceaneng.2024.119283. hal-04759261

HAL Id: hal-04759261

https://hal.science/hal-04759261v1

Submitted on 5 Nov 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

Research paper

# A 2D VMD video image processing-based transfer learning approach for the detection and estimation of biofouling in tidal stream turbines

Houssem Habbouche<sup>a</sup>, Haroon Rashid<sup>b</sup>, Yassine Amirat<sup>c</sup>, Arindam Banerjee<sup>d</sup>, Mohamed Benbouzid<sup>b,e,*</sup>

<figure><img src="image_3.png" /></figure>

## ARTICLE INFO

**Keywords:**

Tidal stream turbine

Biofouling

Detection

Estimation

Generative adversarial network

2-dimensional variational mode decomposition

Image processing

Transfer learning

## ABSTRACT

Harnessing the power of tidal streams is a sustainable way of exploiting renewable marine energy resources. It involves installing tidal stream turbines underwater to harness the energy. Nevertheless, these turbines are prone to the accumulation of biofouling, which significantly reduces their energy output and operational efficiency. It is therefore crucial to implement a condition-based monitoring system to detect biofouling promptly and ensure the continuous operation of a tidal stream turbine. In this context, this paper presents a data-centric approach that uses model submerged tidal stream turbine video images to detect and quantify biofouling. The relevance of a two-dimensional variational mode decomposition approach is investigated to extract relevant information from the potentially noisy collected images. While generative adversarial networks are used to address the data imbalance problem, a convolutional neural network is adopted to detect and assess the extent of biofouling. The performance of the proposed approach is assessed and validated using two experimental datasets obtained from the tidal stream turbine platforms of the Shanghai Maritime University and the Lehigh University.

## 1. Introduction

The energy sector is a pillar of the country's economic activity through its various forms of energy in nature, both fossil and renewable (Nafkha-Tayari et al., 2022). Fossil energy comes from sources available in limited quantities on Earth, which are exhausted over time. Renewable energy comes from natural resources that can be replenished and are environmentally friendly. This type of energy is available in nature in various forms, such as solar, wind, ocean, etc. The advantage of the latter is its independence from the climate, as well as its sustainability. There are several ways of harnessing this energy, such as tidal stream, with the advantages of good predictability and minimal visual and environmental impact (Rashid et al., 2023d; Dezhdar et al., 2023).

Harnessing tidal stream energy requires a tidal turbine (TST) embedded in the estuary, rotating under the effect of the tidal stream (Liguo et al., 2022). Unfortunately, these turbines present a favorable environment for biofouling deposits (Titah-Benbouzid et al., 2023).

These plant organisms cause an increase in aerodynamic resistance, a reduction in efficiency, and accelerated corrosion of the turbines (Rashid et al., 2023c,a). At this stage, cleaning is necessary. Systematic preventive maintenance consists of setting dates for turbine cleaning. This strategy has proved costly and has been replaced by condition-based maintenance (CBM), which uses various monitoring techniques such as vibration monitoring caused by turbine blade imbalance, current and voltage monitoring, underwater imaging inspection, etc. (Abbas and Shafiee, 2020; Freeman et al., 2021).

Recent frameworks have exploited image-based methods, which are the easiest to master and the most efficient, despite the disadvantage of noisy and blurry images (Rashid et al., 2023b). Bloomfield et al. (2021) applied deep learning to automate the classification of biofouling in images and compared it to expert assessments. A model trained on over 10,000 annotated images showed accuracy comparable to that of experts, suggesting that automated analysis could effectively replace costly manual inspections. Zhao et al. (2023) introduced the FIDCE

<footnote><sup>a</sup> Ecole Militaire Polytechnique, Mechanical Structures Laboratory, Bordj Elbahri 16046 Algiers, Algeria</footnote>

<footnote><sup>b</sup> University of Brest, UMR CNRS 6027, 29238 Brest, France</footnote>

<footnote><sup>c</sup> ISEN Yncrea Ouest, L@blsen, 29200 Brest, France</footnote>

<footnote><sup>d</sup> Department of Mechanical Engineering & Mechanics, Lehigh University, Bethlehem, PA 18015, USA</footnote>

<footnote><sup>e</sup> Shanghai Maritime University, Logistics Engineering College, Shanghai 201306, China</footnote>

<footnote>* Corresponding author at: University of Brest, UMR CNRS 6027, 29238 Brest, France.
E-mail addresses: houssem.habbouche@emp.mdn.dz (H. Habbouche), haroon.rashid@univ-brest.fr (H. Rashid), yassine.amirat@isen-ouest.yncrea.fr (Y. Amirat), arb612@lehigh.edu (A. Banerjee), mohamed.benbouzid@univ-brest.fr (M. Benbouzid).</footnote>
algorithm to enhance biofouling image segmentation. The MFONet model, incorporating MobileNetV2 and ASPP, achieved precise pixel-level segmentation. Results demonstrated that MFONet enabled accurate and swift identification of biofouling, making it suitable for underwater cleaning robots. Santos et al. (2022) utilized hyperspectral imaging to detect and quantify marine biofouling on coated surfaces, offering greater precision than visual methods. A new imager, incorporating a liquid crystal filter and uniform LED illumination, achieved high accuracy. The model enabled detailed biofouling quantification despite spectral similarities among species. Gormley et al. (2018) tested CoralNet, an automated image analysis software, on images of marine biofouling from UK offshore platforms. The results revealed varying levels of biofouling diversity and showed no significant differences between analysis methods. CoralNet has enabled a more efficient and consistent approach to the assessment of biofouling in offshore structures. Pedersen et al. (2022) used image analysis to evaluate fouling control coatings, in contrast to manual inspections. A pixel classification model, trained with ilastik, quantified biofouling coverage on panels exposed at sea. The results facilitated a standardized assessment of coating resistance. First et al. (2021) developed a method for quantifying biofouling using a low-cost underwater camera and image processing algorithms. In situ images were analyzed with machine learning models to classify and quantify fouling. This approach proved to be quick, simple, and cost-effective for managing biofouling. As is well known, underwater vision is always noisy, which makes object detection more difficult, especially for biofouling, whose irregular shapes and colors vary from species to species. In this context, the above-discussed works on the detection of biofouling through imaging involves a variety of image processing techniques. These techniques include feature extraction by convolutional layers (Bloomfield et al., 2021), combined with a guided filter (Zhao et al., 2023), a physical filter combined with hyperspectral imaging (Santos et al., 2022), software for image processing (Gormley et al., 2018; Pedersen et al., 2022), and color code filtering (First et al., 2021).

Recently, the two-dimensional variational mode decomposition (2D-VMD) has been suggested as an efficient tool for image processing. This technique offers better advantages, such as good multi-scale decomposition and better image denoising. 2D-VMD is used in different areas and has proven its efficiency for image processing. In the field of medicine, Parashar and Agrawal (2021) treat glaucoma disease for severity classification (3 classes). Compact 2D-VMD is used for the decomposition of the retinal image of the eye. For each variational mode, they calculate indicators that are then reduced using LDA. At the end, a classification by SVM and experimental validation. In the field of image processing, Ma et al. (2019) used 2D-VMD for color image reconstruction, beginning with the decomposition of the image using compact 2D-VMD and the removal of artifact pixels. They then adjusted the image lighting and reconstructed the image to achieve better contrast. Pei et al. (2020) used 2D-VMD in geological interests for source edge detection. Using decomposition by 2D-VMD and mathematical morphology, this makes it possible to detect field edges. 2D-VMD is also used for hyperspectral image classification by Zhuo et al. (2023). It begins by decomposing the images into a series of mode components, then collects the temporal characteristics of each mode to be learned using an SVM.

Researchers in recent years have adopted machine learning-based approaches for early detection and diagnosis. Such an approach provides a decision-support tool deployed in real-time and permanently. Chen et al. (2020) transformed vibration signals from bearing defects into RGB images after time-frequency transformation-based CWT. These images are used for the training of a ResNet network and for the diagnosis of bearing defects. Zhong et al. (2022) With the same processing technique, signals are transformed into images. To improve diagnostic quality, a transfer model called SqueezNet is combined with the self-attention mechanism. This methodology is applied to the bearing defect. The same methodology was applied using ResNet34

for bearing fault classification (Zhou et al., 2022). Zhu et al. (2023) converted vibration signals to spectral images using synchrosqueezing wavelet transform. These images are used to train a network combining VGG for feature extraction and LSTM for good memory and better classification. This method is applied to fault diagnosis of hydraulic axial piston pumps.

Despite the significant advancements made by machine learning, the primary challenge remains the lack of sufficient data. Various data augmentation techniques are employed to address this issue, including noise addition, rotation, scaling, translation, and cropping (Anaya-Isaza and Mera-Jiménez, 2022; Habbouche et al., 2021a). An innovative augmentation technique known as generative adversarial networks (GANs) has recently emerged. These networks are designed to generate synthetic images that closely resemble the original input images. Several crossings were based on this technique, Liang et al. (2020) transformed the monitoring vibratory signals into images using CWT, then increased using GAN. The resulting images are then used for fault classification using CNN. Liu et al. (2022) used GAN to generate synthetic vibratory signals for better CNN network training for fault classification, validated on an experimental dataset. For the same purpose of increasing data, Li et al. (2022b) generated spectra of the different mechanical faults based on conditional Wasserstein GAN, and then evaluated the correlation between generated and real signals. Finally, faults classification by GRU. Li et al. (2020) applied GAN with a gradient penalty to balance and complete the dataset. From real surveillance spectra, new signals are generated to improve learning. This approach is applied to gearbox diagnosis.

Based on the above background and current state-of-the-art techniques, this paper proposes the development of a new method for the detection and estimation of biofouling. The method relies on GANs for data enhancement through synthetic image generation, uses 2D-VMD for image processing and filtering, and employs ResNet50 for feature extraction and classification. In this context, the main contributions of the proposal are as follows:

- Providing an intelligent decision support tool for real-time estimation of biofouling by first detecting and then estimating it;

- Data augmentation using GAN for synthetic image generation;

- Underwater image denoising using 2D-VMD for improved biofouling detection;

- Extraction of multi-scale functions using ResNet50 to retrieve the most discriminating features between different classes;

- Double experimental evaluation and validation using TST platform datasets;

- Demonstration of the effectiveness of the proposed methodology in different operating modes (5 different tidal stream speeds).

The paper is organized as follows: Section 2 is devoted to the theoretical presentation of the methodology. Section 3 deals with experimental evaluation and validation. Section 4 presents an analysis of the obtained results, while Section 5 is the conclusion.

## 2. Proposed 2D-VMD image processing-based machine learning approach

The proposed methodology for biofouling detection and estimation is illustrated by the flowchart of Fig. 1. Starting with data augmentation using GAN, then image processing by 2D-VMD feature extraction and classification using ResNet50. Finally, an experimental evaluation of the methodology using different statistical criteria.

### 2.1. Generative adversarial network

GAN is a type of deep learning network proposed by Goodfellow in 2014. Consisting of two internal networks, the first one is called generator (G) and the second one is called discriminator (D). The role of the G is to generate images resembling real images. The role of D is to differentiate between real and fake images (fake images are
<figure><img src="image_4.png" /><figcaption>Fig. 1. Proposed methodology flowchart.</figcaption></figure>

those generated by G), as illustrated in the flowchart of Fig. 2. The two networks continue to improve the images generated so that they are as close as possible to the original, minimizing the difference between the generated images and the real ones (min_G), on the other hand, improving the quality of differentiation between fake and real images (max_D) (Li et al., 2022a).

This minimax competition is expressed by an optimization problem formulated as follows:

$$ \min_G \max_D V(D, G) = E_{X \sim p_X(x)} [\log D(x)] + E_{Z \sim p_Z(z)} [\log(1 - D(G(z)))] \quad (1) $$

z represents the random signal generated with the distribution p(z); x represents the input signal with the distribution p(x); G(z) is the generated sample; D(x) is the obtained result. If D(x) > 0.5 the sample is real, else it is fake (Li et al., 2022b,a).

The training process is done by the descent gradient for updating parameters, and loss function minimization.

## 2.2. 2D variational mode decomposition

Based on multi-scale image decomposition, allow decomposing image to k modes, named intrinsic mode function (IMF). Each one is characterized by a limited bandwidth and a central frequency. 2D-VMD, an extension of VMD, is an adaptive, non-recursive algorithm.

<figure><img src="image_5.png" /><figcaption>Fig. 2. GAN flowchart (Li et al., 2022a).</figcaption></figure>

Based on a multi-scale decomposition of the image, it allows the image to be divided into k modes, known as intrinsic mode functions (IMFs). Each of these is characterized by a limited bandwidth and a central frequency.

The decomposition steps are described in the following steps (Dragomiretskiy and Zosso, 2015; Jiang et al., 2019):

1. The analytic signal ($\hat{u}_{AS,k}(\vec{x})$) to be decomposed is formulated as in Eq. (2). Obtained by the Hilbert transformation of the original signal $f(x)$, which allows to obtain unilateral frequency spectrum:

$$ \hat{u}_{AS,k}(\vec{x}) = \hat{u}_k(\vec{x}) * \left( \delta(\langle\vec{x}, \vec{w}_k\rangle) + \frac{j}{\pi \langle\vec{x}, \vec{w}_k\rangle} \delta(\langle\vec{x}, \vec{w}_{k,\perp}\rangle) \right) \quad (2) $$

where $\delta(t)$ is the unit impulse function; * is the 2D convolution operator.

2. The formulated spectrum consists of a set of modes around central frequencies $e^{-j\langle\vec{x}, \vec{w}_k\rangle}$ and specific bandwidth. The spectral formulation of the multi-modal signal is as follows:

$$ \left[ \hat{u}_k(\vec{x}) * \left( \delta(\langle\vec{x}, \vec{w}_k\rangle) + \frac{j}{\pi \langle\vec{x}, \vec{w}_k\rangle} \delta(\langle\vec{x}, \vec{w}_{k,\perp}\rangle) \right) \right] e^{-j\langle\vec{x}, \vec{w}_k\rangle} \quad (3) $$

3. The frequency bandwidth is estimated by an optimization of the gradient L2-norm, formulated as in Eq. (4).

$$ \min \left\{ \sum_{k=1}^{K} \left\| \nabla \left[ \hat{u}_k(\vec{x}) * \left( \delta(\langle\vec{x}, \vec{w}_k\rangle) + \frac{j}{\pi \langle\vec{x}, \vec{w}_k\rangle} \delta(\langle\vec{x}, \vec{w}_{k,\perp}\rangle) \right) \right] e^{-j\langle\vec{x}, \vec{w}_k\rangle} \right\|_2^2 \right\} \quad (4) $$

$$ \text{s.t.} \quad \sum_{k=1}^{K} \hat{u}_k(\vec{x}) = f(x) $$

where $\hat{u}_k = \{u_1, u_2, \dots, u_k\}$ represents 2D modes after decomposition; $\omega_k = \{\omega_1, \omega_2, \dots, \omega_k\}$ represents center frequencies of each mode; $\nabla$ represents second derivative.

The resolution of the constrained variational problem is transformed into an iterative optimization problem, by the introduction of the Lagrangian matrix function L. The new formulation in Eq. (5).

$$ L(\{u_k\}, \{\omega_k\}, \lambda) = \alpha \sum_{k=1}^{K} \left\| \nabla \left[ \hat{u}_k(\vec{x}) * \left( \delta(\langle\vec{x}, \vec{w}_k\rangle) + \frac{j}{\pi \langle\vec{x}, \vec{w}_k\rangle} \delta(\langle\vec{x}, \vec{w}_{k,\perp}\rangle) \right) \right] e^{-j\langle\vec{x}, \vec{w}_k\rangle} \right\|_2^2 + \left\| f - \sum_{k=1}^{K} u_k(x) \right\|_2^2 + \left\langle \lambda(x), f - \sum_{k=1}^{K} u_k(k) \right\rangle \quad (5) $$

where $\lambda$ represents the Lagrange multiplier; $\alpha$ represents the penalty factor parameter.

This transformation allows to solve unconstrained problem Eq. (6).

$$ \min_{u_k, \omega_k} \max_{\lambda} L(\{u_k\}, \{\omega_k\}, \lambda) \quad (6) $$
<figure><img src="image_6.png" /><figcaption>Fig. 3. Residual learning (He et al., 2016).</figcaption></figure>

At each iteration, the optimization variables are updated by the following formulas:

$$u_k^{n+1}(\vec{\omega}) = \frac{\hat{f}(\vec{\omega}) - \sum_{k=1}^{K} u_k(\omega) + \frac{\lambda(t)}{2}}{1 + 2\alpha(\omega - \omega_k)^2} \quad (7)$$

$$\vec{\omega}_k^{n+1} = \frac{\int_{\Omega_k} \vec{\omega} |\hat{u}(\vec{\omega})|^2 d\vec{\omega}}{\int_{\Omega_k} |\hat{u}(\vec{\omega})|^2 d\vec{\omega}} \quad (8)$$

$$\hat{\lambda}^{n+1}(\omega) = \hat{\lambda}^{n}(\omega) + \tau \left( \hat{f}(\omega) - \sum_{k=1}^{K} u_k^{n+1}(\vec{\omega}) \right) \quad (9)$$

A check of the stop condition at each iteration in Eq. (10):

$$\sum_{k=1}^{K} \frac{\| \hat{u}_k^{n+1} - \hat{u}_k^n \|_2^2}{\| \hat{u}_k^n \|_2^2} < \varepsilon \quad (10)$$

where $\tau$ is the fixed time step and $\varepsilon$ is convergence error.

After convergence of the problem, the mode spectra obtained are transformed into the time domain by the inverse Fourier transform.

## 2.3. Transfer learning

Convolution networks have been a considerable success in recent years, and their introductions in different fields for image classification. The strength of these networks is in relation to their two-stage architecture, the first stage of feature extraction through convolution and pooling operations. The second stage, learning, can be performed by different types of networks, such as MLP, LSTM, etc. (Habbouche et al., 2021b,a). The design of these networks is time-consuming and requires significant resources and a large amount of data. For this task, pre-trained networks are preferable, and this procedure is called transfer learning. This procedure involves using networks trained on complex classification tasks and adapting them to achieve a less complex task (Anaya-Isaza and Mera-Jiménez, 2022). Several convolution networks are used for this purpose, such as ResNet50, VGG16, VGG19, etc. (Rashid et al., 2023b). These networks were trained on a large dataset developed for standard computer vision benchmarks. Deep Residual Network (ResNet) was created by He et al. (2016) and won the ImageNet competition in 2015. ResNet is a network of 50 deep layers (48 convolutive layers, one MaxPool layer, and one medium pool layer). Previously, learning was enhanced by means of deeper and deeper networks. Gradient vanishing is an emerging problem. ResNet can improve accuracy without going deep by adding residual connection skip layers, as shown in Fig. 3.

This residual connection tip formulated as fellow (He et al., 2016):

$$H(x) = F(x) + x \quad (11)$$

where $x$ denote the input layer, $F(x)$ output layer, and $H(x)$ output of residual layer. where $x$ denotes the input layer, $F(x)$ the output layer, and $H(x)$ the residual layer output.

<figure><img src="image_7.png" /><figcaption>Fig. 4. The Shanghai Maritime University tidal stream turbine experimental platform (Saidi et al., 2020).</figcaption></figure>

<table><caption>Table 1
Summary of training parameter.</caption><thead><tr><td>Parameter</td><td>Value</td></tr></thead><tbody><tr><td>Optimizer</td><td>Adam</td></tr><tr><td>Mini batch size</td><td>128</td></tr><tr><td>Learning rate</td><td>0.0002</td></tr><tr><td>Gradient decay factor</td><td>0.5</td></tr><tr><td>Squared gradient decay factor</td><td>0.99</td></tr></tbody></table>

## 2.4. Evaluation and classification

The classification is evaluated using statistical indicators. Among these indicators, accuracy is the most used and gives a global overview of learning and classification quality. The confusion matrix is another representation of the classification and gives more detail on the classification quality for each class (Habbouche et al., 2021b). Other performance metrics classifications are used for global and specific evaluation which are: Recall, precision, and F1-score (Chien et al., 2022).

# 3. Experimental dataset-based evaluation and validation

## 3.1. Dataset description

For the experimental validation of the proposed methodology, a dataset created at the Shanghai Maritime University TST platform laboratory is used (Fig. 4). The test rig consists of a tunnel to simulate aquatic turbulence. In this tunnel is submerged an electric turbine 230W/8 pole pairs direct-drive permanent magnet synchronous generator. On the blades of this turbine, different masses are glued to prevent the imbalance caused by the deposition of algae, as shown in Fig. 5. The test rig made it possible to create a dataset of five classes (healthy, single blade densely attached, single blade sparsely attached, two blades densely attached, and three blades densely attached). Every class contains five images for five speed levels (level 1, level 2, level 3, level 4, and level 5) captured from a recorded video (Saidi et al., 2020).

## 3.2. Methodology description

The implementation procedure for the proposed methodology is as follows: We begin by reading images from the dataset containing 5 images (5 different operating modes) for the 5 classes, resulting in a total of 25 images. This number of images is extremely low for deep network training, which requires a data augmentation process.

For data augmentation, GAN is used to generate synthetic images to expand the training dataset. The description of the used network is illustrated in Figs. 6 and 7.

The training parameters are summarized in Table 1.
<figure><img src="image_8.png" /><figcaption>Fig. 5. Emulated biofouling states: (a) healthy, (b) single blade sparsely attached, (c) single blade densely attached, (d) two blades densely attached, (e) three blades densely attached.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 6. Generator network architecture.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Fig. 7. Discriminator network architecture.</figcaption></figure>

<table><caption>**Table 2**
Performance metrics of proposed methodology.</caption><thead><tr><td>Accuracy</td><td>Recall</td><td>Precision</td><td>F1-score</td></tr></thead><tbody><tr><td>94.1%</td><td>94.1%</td><td>93.7%</td><td>93.9%</td></tr></tbody></table>

<table><caption>**Table 3**
Comparison of transfer learning networks.</caption><thead><tr><td></td><td>Resnet50</td><td>VGG16</td><td>VGG19</td><td>MobileNet</td></tr></thead><tbody><tr><td>Accuracy</td><td><b>94.1%</b></td><td>92.9%</td><td>93.8%</td><td>85.1%</td></tr></tbody></table>

Through this step, the database is increased from 5 to 105 images for each class (5 originals and 100 generated). At this level, the quantity of images is acceptable, with the disadvantage that the original images are noisy as shown in Fig. 5, as well as the images obtained by GAN. To address this issue, an image processing approach known as 2D-VMD, which was thoroughly detailed in the preceding section, is applied. For the configuration of this technique, learning accuracy is chosen as the objective function. The best accuracy is obtained for a random initialization and a decomposition at 5 IMF only. The setting adopted is: bandwidth constraint = 5000, Lagrangian multipliers = 0.25, and tolerance = $5 \times 10^{-6}$.

The next step is the detection and then estimation of biofouling. ResNet50 is used with modifications to adapt to our problem. Replace the classification layer from 1000 classes to 2 classes for detection (healthy and faulty) and to 5 classes for severity estimation.

Learning is initiated by the ResNet50 network and the dataset is obtained after processing with a random selection of 80% for training and 20% for testing. Adopting the following training options obtained by grid-search: optimizer = Adam; mini batch size = 32; learning rate = 0.01.

The final step is the evaluation, using different statistical criteria and comparisons, to show the effectiveness of the proposed methodology, with a discussion of results in the next section.

# 4. Results and evaluation

## 4.1. Experimental validation using the Shanghai Maritime University platform

To successfully complete this part, the evaluation criteria are shown for each step to justify the adopted approach.

The first step is data augmentation, resulting in an accuracy of 22.9%. This step is necessary in this context with the dataset used, and the accuracy obtained shows that the beginning of learning is not sufficient due to the quality of the noisy images used.

To improve accuracy, the image was then processed using 2D-VMD. All images were processed prior to training using the ResNet50 network (Fig. 8). This technique improved accuracy by up to 94.1%, with the confusion matrix shown in Fig. 9. The confusion matrix shows good classification with an average recall of 94.1% and an average precision of 93.7% (Table 2). With a maximum shift of 3% for both indicators, the detection of the different classes is homogeneous.

The results obtained are satisfactory, demonstrating the interest and effectiveness of the 2D-VMD technique for processing underwater images, by retaining only the first five IMFs and eliminating the residue.

As previously mentioned, different networks are used for transfer learning. This study compared different pre-trained networks, including VGG16, VGG19, and MobileNet. Table 3 outlines this comparison and highlights the superiority of ResNet50 in the classification of underwater biofouling images.

For practical reasons, it is preferable to detect the biofouling before its estimation. Detection, as defined in many frameworks, is the binary classification between defective and healthy states (Habbouche et al., 2021a). Using the previous dataset, detection is performed between
<figure><img src="image_11.png" /><figcaption>**Fig. 8.** Turbine image decomposition: (a) original, (b) 1st IMF, (c) 2nd IMF, (d) 3rd IMF, (e) 4th IMF.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>**Fig. 9.** Confusion matrix.</figcaption></figure>

the healthy state and the other collected states. The accuracy obtained was 98.7%. This demonstrates the effectiveness of the proposed methodology for detecting biofouling in the event of data imbalance.

<figure><img src="image_13.png" /><figcaption>**Fig. 10.** The Lehigh University tidal stream turbine experimental platform.</figcaption></figure>

Condition-based monitoring systems are needed to reduce the impact of biofouling on energy production. The proposed methodology addresses two practical challenges of TST monitoring. The first is data imbalance, which has been remedied by using GAN for data augmentation. The second concerns the processing of noisy underwater images, which was solved using the 2D-VMD technique. Deployment of the proposed methodology has provided a decision support tool for tidal stream turbine monitoring, with a detection accuracy of 98.7% and an estimation accuracy of 94.1% in different operating modes.

## 4.2. Experimental validation using the Lehigh University platform

To reinforce the effectiveness of the proposed methodology, additional experiments were carried out at the Tidal Turbulence Test Facility at Lehigh University, Pennsylvania, USA (Vinod et al., 2021). The open-surface, recirculating water tunnel (Engineering Laboratory Design Inc., Model# 505) has a test section that is 0.61 m wide, 0.61 m tall and 1.98 m long. It is equipped with a 25HP single stage axial flow propeller pump and is capable of attaining flow speeds up to 1 m/s in the test section (Kolekar et al., 2019; Kolekar and Banerjee, 2015).

The test set-up used for creating the database is illustrated in Fig. 10. A 1:20 scale, three-bladed tidal turbine model was tested in the current work. The 0.279 m diameter (D) rotor was made of constant chord (c = 0.016 m), no twist, SG6043 profiled blades (Hu et al., 2014; Subhra Mukherji et al., 2011; Kolekar et al., 2011). The turbine has a rotor area of 0.061m² and is operated at a blockage ratio (rotor area/cross-section of the test section) of 16% while operating in the T3F facility. It is operated under a water flow velocity of 0.83 m/s and an image capturing frequency of 70 images per second (Vinod and Banerjee, 2019). To replicate the biofouling in a laboratory, we employed plastic beads and tripwire as a surrogate for biological organisms commonly associated with fouling in aquatic environments. These beads, characterized by specific dimensions and features, were randomly distributed and affixed to the front face of blades listed below and depicted in Fig. 11:

- 200 images in the “Leading Edge Trip Wire” class (Fig. 11(a));

- 200 images in the “One blade Randomly Distributed Seventeen Beads” class (Fig. 11(b));

- 100 images in the “Test With Clean Turbine” class (Fig. 11(c));

- 200 images in the “Three blades Randomly Distributed Eight Beads” class (Fig. 11(d));

- 200 images in the “Three Blades Randomly Distributed Seventeen Beads” class (Fig. 11(e));

- 200 images in the “Trip Wire With Fifty Eight Percent Coverage” class (Fig. 11(f));

- 200 images in the “Two beads along RPS” class (Fig. 11(g));

- 200 images in the “Two Blades Randomly Distributed Seventeen Beads” class (Fig. 11(h)).

Additional details about the default class execution procedure are provided in Table 4.

The class “Test With Clean Turbine” is a minority class with 100 samples compared to other classes, each having 200 images. The issue
<figure><img src="image_14.png" /><figcaption>Fig. 11. Emulated biofouling states.</figcaption></figure>

<table><caption>**Table 4**
Experimental conditions.</caption><thead><tr><td>Parameter</td><td>Value</td></tr></thead><tbody><tr><td>Distance of wire from leading edge</td><td>0.11 cm</td></tr><tr><td>Distance of wire from tip</td><td>0.65 cm</td></tr><tr><td>Length of turbine blade</td><td>10.3 cm</td></tr><tr><td>Width of leading edge trip wire</td><td>0.48 cm</td></tr><tr><td>width of turbine blade</td><td>1.4 cm</td></tr><tr><td>Height of wire</td><td>0.11 cm</td></tr><tr><td>Diameter of wire</td><td>0.11 cm</td></tr><tr><td>Area of turbine blade</td><td>14.42 cm²</td></tr><tr><td>Area of one blade covered by leading edge wire</td><td>4.94 cm²</td></tr><tr><td>percentage of covered area by wire of blade</td><td>29.3%</td></tr></tbody></table>

<caption>Fig. 12. Turbine image decomposition: (a) original, (b) 1st IMF, (c) 2nd IMF, (d) 3rd IMF, (e) 4th IMF.</caption>

of imbalanced data necessitates an increase in data to ensure that all classes have 200 images. To address this, GAN is used to generate synthetic images.

The image set (original and generated by GAN) is decomposed using 2D-VMD under the same parameters as those applied to the first dataset. The original image and those obtained by decomposition are shown in Fig. 12.

The original images and those generated by 2D-VMD, are partitioned into training and testing sets. Following the ResNet50 training, the accuracy reaches 95.7%, and the corresponding confusion matrix is depicted in Fig. 13.

Through this dual validation, we affirm the accuracy of the proposed methodology for TST monitoring through photographic imaging, even with the constraint of lack of data and noisy images.

## 5. Conclusion

This paper proposed a data-centric approach using submerged tidal stream turbine video images for biofouling detection and estimation. The proposed methodology framework incorporated 2D variational mode decomposition (2D-VMD) for denoising underwater images, generative adversarial networks (GAN) to address data imbalance, and ResNet50 for multi-scale feature extraction. The approach demonstrated superior accuracy in detection and estimation tasks, validated through two experimental datasets from the tidal stream turbine platforms of Shanghai Maritime University and Lehigh University.
<figure><img src="image_15.png" /><figcaption>Fig. 13. Confusion matrix.</figcaption></figure>

The proposed method has several advantages, including that of providing an intelligent decision-support tool for real-time estimation of biofouling, guaranteeing accurate detection and subsequent estimation. The use of GAN for synthetic image generation has increased the training data, improving the robustness of the model. The method was validated on the TST platforms dataset, demonstrating effectiveness under various operational conditions, including different speeds, and particularly showing an increase in accuracy in highly turbulent environments.

However, the study did not address the annotation of different types of biofouling, which limited the detailed analysis of biofouling. The focus was solely on classifying the blades images affected by biofouling rather than detecting biofouling as an object in the images. The 2D-VMD technique required significant expertise in image and signal processing, and the lack of a clear criterion for selecting appropriate decomposition levels could potentially affect efficiency. Additionally, while ResNet50 performed well, its effectiveness could be further enhanced by incorporating additional feature extraction tools, as suggested in prior works. Overall, the proposed detection and estimation framework has proven to be an effective decision-support tool for monitoring biofouling on tidal stream turbines.

Future investigations aim to design a digital twin of the TST for the detection and prediction of biofouling. To develop this digital twin, it is necessary to combine a model (mathematical or numerical) with experimental data such as images or signals (e.g., vibrations, currents, etc.). When it comes to modeling the biofouling settling phenomenon, this is probabilistic and depends on environmental parameters, such as temperature, salinity, nutrient levels, flow rate, light availability, etc., as well as operational parameters, such as rotation speed limits, operating hours, etc. The operational parameters are operator-dependent, while the environmental parameters are probabilistic and vary according to the geographical deployment site of the TST. If we manage to model these environmental parameters and establish probabilistic colonization models for each region, this will enable us to predict the behavior of the machine at each level of deposit. The biofouling deposit induces an imbalance that is reflected in vibrations or fluctuations in the TST-generated current. By deploying a data acquisition system (vibration and/or current) and studying the similarity between the obtained signal and that simulated by the deposition model, we can define the severity state and estimate the remaining operational life. This information is crucial for maintenance purposes, as it enables to plan the TST cleaning in terms of human and material resources, therefore avoiding an unscheduled shutdown.

### CRediT authorship contribution statement

**Houssem Habbouche:** Writing – original draft, Visualization, Validation, Software, Methodology, Formal analysis, Conceptualization.

**Haroon Rashid:** Writing – review & editing, Visualization, Validation, Methodology, Formal analysis. **Yassine Amirat:** Writing – review & editing, Visualization, Validation, Methodology, Formal analysis.

**Arindam Banerjee:** Writing – review & editing, Visualization, Validation. **Mohamed Benbouzid:** Writing – review & editing, Validation, Supervision, Methodology, Funding acquisition, Formal analysis, Conceptualization.

### Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

### Data availability

Data will be made available on request.

### Acknowledgment

This work is supported by the PIA 3 CMQ Industries de la Mer Bretagne (IndMer), France.

### References

Abbas, M., Shafiee, M., 2020. An overview of maintenance management strategies for corroded steel structures in extreme marine environments. Mar. Struct. 71, 102718.

Anaya-Isaza, A., Mera-Jiménez, L., 2022. Data augmentation and transfer learning for brain tumor detection in magnetic resonance imaging. IEEE Access 10, 23217–23233.

Bloomfield, N.J., Wei, S., A. Woodham, B., Wilkinson, P., Robinson, A.P., 2021. Automating the assessment of biofouling in images using expert agreement as a gold standard. Sci. Rep. 11 (1), 2739.

Chen, Z., Cen, J., Xiong, J., 2020. Rolling bearing fault diagnosis using time-frequency analysis and deep transfer convolutional neural network. IEEE Access 8, 150248–150261.

Chien, C.-F., Hung, W.-T., Liao, E.T.-Y., 2022. Redefining monitoring rules for intelligent fault detection and classification via CNN transfer learning for smart manufacturing. IEEE Trans. Semicond. Manuf. 35 (2), 158–165.

Dezhdar, A., Assareh, E., Agarwal, N., Keykhah, S., Aghajari, M., Lee, M., et al., 2023. Transient optimization of a new solar-wind multi-generation system for hydrogen production, desalination, clean electricity, heating, cooling, and energy storage using TRNSYS. Renew. Energy 208, 512–537.

Dragomiretskiy, K., Zosso, D., 2015. Two-dimensional variational mode decomposition. In: Lecture Notes in Computer Science. Springer International Publishing, pp. 197–208.

First, M.R., Riley, S.C., Islam, K.A., Hill, V., Li, J., Zimmerman, R.C., Drake, L.A., 2021. Rapid quantification of biofouling with an inexpensive, underwater camera and image analysis. Manage. Biol. Invasions 12 (3), 599–617.

Freeman, B., Tang, Y., Huang, Y., VanZwieten, J., 2021. Rotor blade imbalance fault detection for variable-speed marine current turbines via generator power signal analysis. Ocean Eng. 223, 108666.

Gormley, K., McLellan, F., McCabe, C., Hinton, C., Ferris, J., Kline, D.I., Scott, B.E., 2018. Automated image analysis of offshore infrastructure marine biofouling. J. Mar. Sci. Eng. 6 (1), 2.

Habbouche, H., Amirat, Y., Benkedjouh, T., Benbouzid, M., 2021a. Bearing fault event-triggered diagnosis using a variational mode decomposition-based machine learning approach. IEEE Trans. Energy Convers. 37 (1), 466–474.

Habbouche, H., Benkedjouh, T., Amirat, Y., Benbouzid, M., 2021b. Gearbox failure diagnosis using a multisensor data-fusion machine-learning-based approach. Entropy 23 (6), 697.

He, K., Zhang, X., Ren, S., Sun, J., 2016. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 770–778.

Hu, Z., Du, X., Kolekar, N.S., Banerjee, A., 2014. Robust design with imprecise random variables and its application in hydrokinetic turbine optimization. Eng. Optim. 46 (3), 393–419.

Jiang, L., Zhou, X., Che, L., Rong, S., Wen, H., 2019. Feature extraction and reconstruction by using 2D-VMD based on carrier-free UWB radar application in human motion recognition. Sensors 19 (9), 1962.
Kolekar, N., Banerjee, A., 2015. Performance characterization and placement of a marine hydrokinetic turbine in a tidal channel under boundary proximity and blockage effects. Appl. Energy 148, 121–133.

Kolekar, N., Mukherji, S.S., Banerjee, A., 2011. Numerical modeling and optimization of hydrokinetic turbine. In: Energy Sustainability. Vol. 54686, pp. 1211–1218.

Kolekar, N., Vinod, A., Banerjee, A., 2019. On blockage effects for a tidal turbine in free surface proximity. Energies 12 (17), 3325.

Li, Z., Zheng, T., Wang, Y., Cao, Z., Guo, Z., Fu, H., 2020. A novel method for imbalanced fault diagnosis of rotating machinery based on generative adversarial networks. IEEE Trans. Instrum. Meas. 70, 1–17.

Li, Y., Zou, W., Jiang, L., 2022a. Fault diagnosis of rotating machinery based on combination of wasserstein generative adversarial networks and long short term memory fully convolutional network. Measurement 191, 110826.

Li, M., Zou, D., Luo, S., Zhou, Q., Cao, L., Liu, H., 2022b. A new generative adversarial network based imbalanced fault diagnosis method. Measurement 194, 111045.

Liang, P., Deng, C., Wu, J., Yang, Z., 2020. Intelligent fault diagnosis of rotating machinery via wavelet transform, generative adversarial nets and convolutional neural network. Measurement 159, 107768.

Liguo, X., Ahmad, M., Khattak, S.I., 2022. Impact of innovation in marine energy generation, distribution, or transmission-related technologies on carbon dioxide emissions in the united states. Renew. Sustain. Energy Rev. 159, 112225.

Liu, S., Jiang, H., Wu, Z., Li, X., 2022. Data synthesis using deep feature enhanced generative adversarial networks for rolling bearing imbalanced fault diagnosis. Mech. Syst. Signal Process. 163, 108139.

Ma, F., Chai, J., Wang, H., 2019. Two-dimensional compact variational mode decomposition-based low-light image enhancement. IEEE Access 7, 136299–136309.

Nafkha-Tayari, W., Ben Elghali, S., Heydarian-Forushani, E., Benbouzid, M., 2022. Virtual power plants optimization issue: A comprehensive review on methods, solutions, and prospects. Energies 15 (10), 3607.

Parashar, D., Agrawal, D., 2021. 2-d compact variational mode decomposition-based automatic classification of glaucoma stages from fundus images. IEEE Trans. Instrum. Meas. 70, 1–10.

Pedersen, M.L., Weinell, C.E., Ulusoy, B., Dam-Johansen, K., 2022. Marine biofouling resistance rating using image analysis. J. Coat. Technol. Res. 19 (4), 1127–1138.

Pei, Y., Liu, C., Lou, R., 2020. Multi-scale edge detection method for potential field data based on two-dimensional variation mode decomposition and mathematical morphology. IEEE Access 8, 161138–161156.

Rashid, H., Benbouzid, M., Amirat, Y., Berghout, T., Titah-Benbouzid, H., Mamoue, A., 2023a. Biofouling detection and classification in tidal stream turbines through soft voting ensemble transfer learning of video images. Preprint.

Rashid, H., Benbouzid, M., Amirat, Y., Berghout, T., Titah-Benbouzid, H., Mamoue, A., 2023b. Biofouling detection and extent classification in tidal stream turbines via a soft voting ensemble transfer learning approach. In: IECON 2023- 49th Annual Conference of the IEEE Industrial Electronics Society. pp. 01–06.

Rashid, H., Benbouzid, M., Titah-Benbouzid, H., Amirat, Y., Berghout, T., Mamoue, A., 2023c. Mapping a machine learning path forward for tidal stream turbines biofouling detection and estimation. In: IECON 2023- 49th Annual Conference of the IEEE Industrial Electronics Society. pp. 1–6.

Rashid, H., Benbouzid, M., Titah-Benbouzid, H., Amirat, Y., Mamoue, A., 2023d. Tidal stream turbine biofouling detection and estimation: A review-based roadmap. J. Mar. Sci. Eng. 11 (5), 908.

Saidi, L., Benbouzid, M., Diallo, D., Amirat, Y., Elbouchikhi, E., Wang, T., 2020. Higher-order spectra analysis-based diagnosis method of blades biofouling in a PMSG driven tidal stream turbine. Energies 13 (11), 2888.

Santos, J., Pedersen, M.L., Ulusoy, B., Weinell, C.E., Pedersen, H.C., Petersen, P.M., Dam-Johansen, K., Pedersen, C., 2022. A tunable hyperspectral imager for detection and quantification of marine biofouling on coated surfaces. Sensors 22 (18), 7074.

Subhra Mukherji, S., Kolekar, N., Banerjee, A., Mishra, R., 2011. Numerical investigation and evaluation of optimum hydrodynamic performance of a horizontal axis hydrokinetic turbine. J. Renew. Sustain. Energy 3 (6).

Titah-Benbouzid, H., Rashid, H., Benbouzid, M., 2023. Biofouling issue in tidal stream turbines. In: Design, Control and Monitoring of Tidal Stream Turbine Systems. IET, pp. 181–204.

Vinod, A., Banerjee, A., 2019. Performance and near-wake characterization of a tidal current turbine in elevated levels of free stream turbulence. Appl. Energy 254, 113639.

Vinod, A., Han, C., Banerjee, A., 2021. Tidal turbine performance and near-wake characteristics in a sheared turbulent inflow. Renew. Energy 175, 840–852.

Zhao, W., Han, F., Qiu, X., Peng, X., Zhao, Y., Zhang, J., 2023. Research on the identification and distribution of biofouling using underwater cleaning robot based on deep learning. Ocean Eng. 273, 113909.

Zhong, H., Lv, Y., Yuan, R., Yang, D., 2022. Bearing fault diagnosis using transfer learning and self-attention ensemble lightweight convolutional neural network. Neurocomputing 501, 765–777.

Zhou, J., Yang, X., Li, J., 2022. Deep residual network combined with transfer learning based fault diagnosis for rolling bearing. Appl. Sci. 12 (15), 7810.

Zhu, Y., Su, H., Tang, S., Zhang, S., Zhou, T., Wang, J., 2023. A novel fault diagnosis method based on SWT and VGG-LSTM model for hydraulic axial piston pump. J. Mar. Sci. Eng. 11 (3), 594.

Zhuo, R., Guo, Y., Guo, B., 2023. A hyperspectral image classification method based on two-dimensional compact variational mode decomposition. IEEE Geosci. Remote Sens. Lett.