# LAVA: Leveraging Self-attention to Learn Video Features for Wastewater Pipe Anomaly Detection

Ti-Hon Nguyen, Minh-Thu Tran-Nguyen, Carole Delenne, Salem Benferhat

► To cite this version:

Ti-Hon Nguyen, Minh-Thu Tran-Nguyen, Carole Delenne, Salem Benferhat. LAVA: Leveraging Self-attention to Learn Video Features for Wastewater Pipe Anomaly Detection. Third International Conference, ISDS 2025, Intelligent Systems and Data Science, Oct 2025, Can Tho City, Vietnam. pp.3-18, 10.1007/978-981-95-3358-9_1 . hal-05338748

HAL Id: hal-05338748

https://hal.science/hal-05338748v1

Submitted on 30 Oct 2025

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# LAVA: Leveraging Self-Attention to Learn Video Features for Wastewater Pipe Anomaly Detection

Ti-Hon NGUYEN¹, Minh-Thu TRAN-NGUYEN¹, Carole DELENNE²,³, and
Salem BENFERHAT⁴

¹ College of Information Technology

Can Tho University, 92000-Cantho, Vietnam

² IUSTI, AMU, CNRS, Marseille, France.

³ Previously at HSM, Univ Montpellier, CNRS, IRD, Montpellier, France

⁴ CRIL, CNRS UMR 8188, Université d'Artois, France.

nthon@ctu.edu.vn, tnmthu@ctu.edu.vn, carole.delenne@univ-amu.fr,
benferhat@cril.fr

**Abstract.** Urban network data often originate from multiple sources and are represented in diverse formats, which can make their processing and analysis complex. In this work, we focus specifically on video data, particularly inspection television (ITV) videos of wastewater pipes. These videos play a crucial role in the management and maintenance of urban networks. On one hand, they help identify anomalies that may affect the pipes, such as obstructions or degradations. On the other hand, they provide essential information about the structural properties of the pipes and networks, including their diameter and the direction of wastewater flow. In this paper, we propose a classification algorithm for ITV videos, with a particular focus on detecting diameter changes, internal cracks, chemical attacks, and turbid-colored water within the pipes. This task is essential for predictive maintenance and hydraulic modeling of wastewater networks. We build on Video Vision Transformer (ViViT) and TimeSformer-based methodologies for video classification, which allow for the effective capture of both spatial and temporal relationships between the different frames in the video data. We specifically describe various mechanisms for generating training datasets from a subset of manually annotated images. The experimental study shows promising results on real-world ITV videos of wastewater pipe networks.

**Keywords:** Wastewater pipe Anomalies · Video classification · Transformer · Video feature.

## 1 Introduction

The exploitation of geospatial data is a major challenge in many applications today. The development of artificial intelligence technologies, particularly machine learning, has provided new opportunities for managing such geospatial data. This development is further reinforced by open science policies, which make a large number of datasets on geographic or georeferenced information
publicly available. This work fully aligns with this perspective, with a particular focus on the automatic exploitation of video data related to wastewater network management. The automatic processing of these video data has many potential applications. The most obvious is the detection of anomalies, such as pipe degradation or blockages. Another, less obvious but equally important, application is the enrichment of geographic information system (GIS) data, particularly for attributes or properties related to the pipes in these networks. Indeed, GIS contains vast amounts of data on various components of wastewater networks, which are essential both for mapping these networks and for supporting hydraulic modeling. However, one significant challenge for optimal hydraulic modeling is incomplete or missing data. This work aims to exploit video inspection data of pipes to detect and provide more precise information.

In this study, we focus on four types of changes in wastewater pipes: changes in pipe sizes (BAA), internal cracks (BAB), chemical attacks (BAF), and turbid-colored water (BDD). A properly maintained sewage pipe system is crucial for effective urban water management. As the amount of data—particularly from sensors—continues to grow, professionals in the wastewater sector face increasing challenges in analyzing this large volume of information and detecting network anomalies [?]. To effectively leverage video data for these applications, advanced machine learning models are required, among them, the Transformer architecture has emerged as a powerful tool capable of capturing complex sequential patterns. The Transformer architecture, originally introduced by Vaswani et al. [?], is a sequence-to-sequence model initially developed for machine translation tasks [?]. Since its introduction, the Transformer has demonstrated remarkable success across a wide range of natural language processing (NLP) applications, particularly in the development of large pre-trained language models such as BERT [?]. Building on this success, Dosovitskiy et al. [?] proposed the Vision Transformer (ViT), a pure Transformer architecture applied to sequences of image patches, achieving competitive performance on image classification benchmarks. Extending this approach to the video domain, Arnab et al. [?] introduced ViViT (Video Vision Transformer), a Transformer-based architecture designed specifically for video classification, which directly models spatio-temporal dependencies to extract high-level video representations. To tackle the challenges of anomaly detection in video inspections, we employ video classification models based on ViViT [?] and TimeSformer [?]. These models leverage the self-attention mechanisms to automatically learn rich and discriminative video features, making them particularly suited for complex video analysis tasks. The rest of this paper is organized as follows. Section ?? reviews previous work on anomaly detection in wastewater inspection videos. Section ?? briefly presents the algorithm used to construct an experimental video dataset from wastewater network data for training video classification models, as well as the model for detecting anomalies in wastewater pipes. Section ?? presents the experimental results on real inspection video data, followed by conclusions and directions for future work in Section ??.
## 2 Related work

In 2022, Li et al. [?] conducted a comprehensive review of methodologies, datasets, and challenges related to wastewater network pipe inspection since 2000. They proposed a systematic taxonomy of sewer inspection algorithms, categorizing them into three main approaches: classification, detection, and segmentation. They highlighted the critical need for publicly available datasets and open-source code to facilitate reproducible research. A total of 32 datasets from various countries were summarized, with the United States contributing the largest share—12 datasets. These datasets exhibit significant variability in size, from 32 to 2,202,582 images, and encompass a wide range of resolutions—from 64×64 to 4000×46,000 pixels. Their study also identified key challenges, primarily arising from poor dataset quality, as reported in the existing literature. While many researchers have achieved promising results, most studies focus on a limited subset of damage types, leaving the effectiveness of these models on comprehensive datasets largely unaddressed. To overcome limitations due to insufficient data quality, Jung et al. (2024) [?] proposed a sensor system capable of capturing both images and point clouds in a standardized way, facilitating the creation of datasets suitable for developing deep learning (DL) models. Although point clouds are not yet widely used in current inspection practices, they offer considerable potential for detecting geometric anomalies not easily identifiable in conventional images. Their approach involves a multi-sensor robotic system integrated with a deep learning framework. They also synthesized several datasets related to anomaly detection in sewer pipelines. Besides image datasets, video data is also becoming increasingly available, with new datasets derived from video recordings collected in China since 2022. In this study, we also aim to leverage video-based models to enhance the accuracy and interpretability of anomaly detection and prediction in sewer inspection tasks. Li et al. [?] proposed a novel instance segmentation model, Pipe-SOLO, designed to detect six common types of sewer defects: cracks, faulty joints, open joints, protruding laterals, broken pipes, and surface damage. The model builds upon the original SOLOv2 architecture by integrating a more efficient backbone module (Res2Net-Mish-BN-101) and adopting an enhanced Bi-directional Feature Pyramid Network (EBiFPN) as the neck to improve feature fusion. To address the challenge of blurred CCTV images from sewer inspections, a GAN-based dehazing model was applied during preprocessing. The dataset used in the study was manually collected and annotated from CCTV sewer footage recorded in various locations across Seoul, South Korea, and provided by the Seoul Digital Foundation. Experimental results demonstrated that Pipe-SOLO outperformed existing segmentation methods, achieving a 7.3% improvement in mean Average Precision (mAP) compared to state-of-the-art approaches. Zhang et al. (2023) [?] proposed an enhanced sewer anomaly detection model based on YOLOv4 for identifying four common defect types: cracks, deposits, roots, and stagger. A key improvement of their method is the use of a spatial pyramid pooling (SPP) module to expand the receptive field and enhance the model's ability to fuse contextual features across different receptive fields. The effectiveness of combining the DIoU loss
function with the SPP module was also validated. They utilized 2,700 relabeled images from the publicly available Sewer-ML dataset, formatted in a multi-label structure. Experimental results showed that, compared to the original YOLOv4 model, the improved version achieved a 4.6% increase in mean Average Precision (mAP), reaching 92.3%, with a recall of 89.0%.

Chen et al. [?] proposed a modified deep learning architecture, RegNet+, for the automated detection and classification of sewer defects using CCTV inspection footage. The model builds upon the original RegNet by incorporating dropout layers and replacing the standard ReLU activation with LeakyReLU in the squeeze-and-excitation (SE) block to enhance generalization and reduce overfitting. From 7,733 CCTV sewer videos, 20 distinct defect categories were manually annotated, and 12,000 images were selected through data augmentation, with 600 samples per class. Experimental results demonstrated that RegNet+ consistently outperformed several widely used deep learning models, including VGG16, VGG19, GoogleNet, ResNet50, and the original RegNet, in both single-class and multi-class classification scenarios.

Biswas et al. [?] processed data from 221 German municipalities, including 10,205 video files, and extracted 156,028 images of wastewater pipe anomalies. These were classified using standard German inspection codes, corresponding to the European pipe inspection standard DIN EN 13508-2 [?], consistent with the database used here. They tested six deep learning architectures: HRNet, ResNet-152, EfficientNet, MobileNetV3-Large, InceptionV4, and DenseNet-264. The models achieved balanced accuracy above 85% on 19 categories, including BAF (chemical attacks) and BDD (turbid and colored waters), with remaining categories around 80%. BAF was the most prevalent, with 8,629 instances and 91.5% balanced accuracy. On the Sewer-ML dataset, for BAF and BDD (OB and VA), true positive recall dropped to 2.49% and 2.2%, respectively.

Liu et al. [?] introduced the VideoPipe Challenge to advance video anomaly analysis for urban pipe inspection, crucial for maintaining real-world sewer infrastructure. They launched a challenge website (https://videopipe.github.io) and released two benchmark datasets—QV-Pipe and CCTV-Pipe—on February 28, 2022, supporting Video Anomaly Classification and Temporal Anomaly Localization. The video data, authorized by Shenzhen Bwell Robotics Co., Ltd., is shared under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 License. Challenge participants demonstrated strong performance using deep learning architectures and advanced loss functions. Video-based backbones consistently outperformed image-based models. Transformer-based architectures excelled at modeling long-range dependencies across frames, and the super-image approach, rearranging frames into a single composite image, effectively captures temporal relationships. Focal loss also helps address class imbalance in long-tailed datasets.
## 3 Methodology

In this work, we start with a dataset composed of wastewater pipe inspection videos and corresponding anomaly images annotated by domain experts. Based on this dataset, we propose an approach to construct experimental video datasets tailored for training video classification models such as TimeSformer [?] and ViViT [?], with the objective of detecting anomalies in sewer inspection videos. Algorithm ?? focuses on identifying defective anomalies in inspection videos by employing a pretrained video classification model. To support this task, Algorithm ?? is introduced to generate video datasets by selecting *n* consecutive frames in close proximity to a manually annotated target frame. The constructed datasets serve as inputs for training and evaluating the anomaly detection performance of Algorithm ??.

<caption>**Algorithm 1 Detection of anomalies in inspection video**</caption>

1: Given: inspected video *insVideo*, pre-trained video classification model *classifier*, the set of anomaly code *anomalies_code*, length of sub video by second *sub_length*.
2: Initialize: an empty list of tuple *anomalies_found* ▷ This empty list is used to store detected anomalies; each element will be a sublist containing: the index of the sub-video, the video segment itself, and the predicted class
3: *set_of_subVideo* ← *splitVideo_function*(*insVideo*, *sub_length*) ▷ Usually *sub_length* is 5 seconds
4: **for** *i* ∈ *range*(*len*(*set_of_subVideo*)**) **do**
5:     *class* ← *classifier*predict(*set_of_subVideo*[*i*])
6:     **if** *class* ∈ *anomalies_code* **then**
6:         *anomalies_found*.append([*i*, *set_of_subVideo*[*i*], *class*])
7:         **end if** ▷ We ignore sub-videos that are classified as having no anomalies.
8: **end for**
9: **return** *anomalies_found*

The goal of Algorithm ?? is to identify defective segments in a sewer inspection video using a pre-trained video classification model. The inputs are: a video denoted as *insVideo*, a trained classification model *classifier*, a set of known anomaly codes *anomalies_code*, and a defined segment duration *sub_length* used to divide the video into smaller sub-videos. In the initialization step, an empty list *anomalies_found* is created to store sub-video segments identified as containing anomalies, along with their index and associated predicted class (line 2 of the algorithm). The algorithm proceeds by splitting the input video into a collection of smaller sub-videos, each of length *sub_length* seconds, using the function *splitVideo_function* (line 3). This produces the set *set_of_subVideo* containing all the sub-segments generated from the original video. The main processing loop then iterates over each sub-video (line 4). For each segment at index *i*, the classifier predicts a class label *class* (line 5). If the predicted class matches one of the predefined anomaly codes in *anomalies_code*, the sub-video is flagged as problematic. The algorithm records the index *i*, the
corresponding sub-video, and the predicted class by appending them as a sub-list to *anomalies_found* (line 6). Finally, the algorithm returns the complete list of detected anomalies, enabling targeted analysis of specific segments in the inspection video (line 9).

## 3.1 Video classification models

**TimeSformer** TimeSformer [?] takes as input a video clip $X \in \mathbb{R}^{H \times W \times 3 \times F}$, where $H$ and $W$ denote the height and width of each frame, 3 is the number of RGB channels, and $F$ is the number of frames in the clip. Each frame is decomposed into $N$ non-overlapping patches (similar to ViT [?]), each of size $P \times P$, such that the $N$ patches cover the entire frame:

$$N = \frac{H \cdot W}{P^2}$$

TimeSformer then flattens each patch into a vector

$$x_{(p,t)} \in \mathbb{R}^{3P^2}$$

where $p = 1, \dots, N$ indexes spatial locations within a frame, and $t = 1, \dots, F$ indexes the temporal frames. Each vector contains the RGB values of the corresponding patch. Then TimeSformer linearly map each patch $x_{(p,t)}$ into an embedding vector $\mathbf{z}_{p,t}^{(0)} \in \mathbb{R}^D$ as equation (3):

$$\mathbf{z}_{(p,t)}^{(0)} = E\mathbf{x}_{(p,t)} + \mathbf{e}_{(p,t)}^{\text{pos}} \quad (1)$$

where $\mathbf{e}_{(p,t)}^{\text{pos}} \in \mathbb{R}^D$ is the linear embedding added to encode the spatiotemporal position of each patch. The resulting sequence of embedding vector $\mathbf{z}_{p,t}^{(0)}$ for $p = 1, \dots, N$, and $t = 1, \dots, F$ presents the input to the Transformer and plays a role similar to the sequences of embedded words that are fed to text Transformer in NLP. Like BERT Transformer [?], TimeSformer adds in the first position of the sequence a special learnable vector $\mathbf{z}_{(0,0)}^{(0)} \in \mathbb{R}^D$ representing the embedding of the classification token.

**Divided Space-Time Attention** in TimeSformer is both more efficient and accurate than other types of attention. In this mechanism, within each encoder block, temporal attention is first computed by comparing each patch $(p, t)$ with all patches at the same spatial location $p$ across other frames. The resulting temporal attention is then used to compute spatial attention by comparing each patch with all other patches within the same frame at time $t$.

**The Video Vision Transformer (ViViT)** [?] introduces a straightforward method for converting a video $V \in \mathbb{R}^{T \times H \times W \times C}$ into a sequence of tokens $\hat{z} \in \mathbb{R}^{n_t \times n_h \times n_w \times d}$. This process, known as **Tubelet embedding**, involves extracting non-overlapping spatio-temporal “tubelets” from the input video and
projecting them linearly into $\mathbb{R}^d$. The resulting sequence is then reshaped into $\mathbb{R}^{N \times d}$ and enriched with positional embeddings to produce $z$, which serves as the input to the Transformer. As illustrated in Figure ??, this approach generalizes the standard 2D patch-based embedding in ViT to three dimensions, analogous to applying 3D convolutional operations for spatio-temporal representation learning. Formally, for a tubelet of size $t \times h \times w$, the number of tokens along each axis is defined as $n_t = \frac{T}{t}$, $n_h = \frac{H}{h}$, and $n_w = \frac{W}{w}$, where $T$, $H$, $W$ and $t$, $h$, $w$ are the temporal, height and width dimensions of the video and tubelet, respectively. Thus, reducing the tubelet size leads to a higher number of tokens and increased computational complexity. Unlike models such as TimeSformer, which extract temporal information independently across frames through “patch sampling”, ViViT integrates both spatial and temporal information during tokenization, enabling a more unified spatio-temporal feature representation.

<figure><img src="image_2.png" /><figcaption>Fig. 1: Tubelet embedding. Tubelet is the combination of $k$ patches ($k=3$) that have the same position $p$ of $k$ continuous frame.</figcaption></figure>

## 3.2 Building experiment datasets

The data used in this paper were obtained from “Montpellier Méditerranée Métropole” (3M), a public organization responsible for water management throughout the Montpellier metropolitan area, located in the south of France. Different portions of the network were studied, and for each portion, 897 videos of wastewater pipe inspections were recorded. Based on these videos, PDF and image reports of anomalies were created, containing information such as the details of wastewater pipe anomalies: positions of anomalies, associated codes and names, as well as maps showing the locations of manholes, pipes, and manhole identifiers. In addition to these reports, relationships between images and videos, and the problem codes of images, are stored in TXT and XML metadata files.

The following section outlines the process of constructing the experimental dataset, focusing first on the BAA anomaly (reduction of pipe height) as an example, with the same methodology applied to other anomalies. A total of 3,583 labeled images were manually extracted from 897 inspection videos, including
268 BAA-labeled images and 3,315 images labeled with other defect codes such as BAB (internal crack) and BAF (chemical attack). Among these videos, 173 contain at least one BAA image, while the remaining 724 do not. Some videos include multiple BAA-labeled frames, indicating the presence of the anomaly at multiple locations along the pipeline. The models [?,?] use short-labeled videos in training and testing sets. Therefore, we have to build the experiment dataset with short videos. Specifically, sub-videos centered around the manually annotated BAA frames were extracted and labeled as BAA samples, forming the positive examples in the classification dataset.

However, the metadata does not let us know how long the problem occurred in the video. We can split the entire video into a sub-video with equal window time, such as one or two seconds. However, if the sub-video contains too low-number BAA frames problem, it will be hard for the model to learn the feature to discriminate the BAA problem. We used Algorithm ??, named "Vnframe", for getting a sub-video with all frames close to the manually labeled BAA image. This algorithm tries to find *n* consecutive frames in the full video and meets the condition that these frames are close to the manually labeled BAA image (extracted in that full video). Then, *n* of these frames will be selected as a sub-video with the BAA label (positive class). We have three primary sources to take the sub-video for the none-BAA video (negative class).

- The first source is to get the sub-video in the same full video, which has manually extracted BAA images. Getting a none-BAA sub-video in the same full video with a BAA sub-video helps the model learn the difference between the BAA and none-BAA features in the same pipeline. To select a sub-video in this source, we can use Algorithm ?? to find *n* consecutive frames in the full video that are not close to the manually labeled BAA image, constructing these *n* frames as a sub-video and label it as none-BAA video.

- The second source to get a none-BAA sub-video is in the videos that do not contain any manually extracted BAA images, which may cause a lot of none-BAA sub-video; therefore, we could use random numbers to identify if we should get that considered sub-video. Additionally, we need to control the balance number of BAA and non-BAA in the dataset to help train the video classification model more easily. Getting a none-BAA sub-video in the full video that contains no BAA images can help the classification model to learn rich general features between all the pipelines we have in the data.

- The third source is to get positive sub-videos for all labeled images of other classes (except BAA images) and use them as non-BAA videos.

The sources for creating the none-BAA video are much richer than the source of the BAA video. Therefore, we proposed to create two experiment datasets; the first dataset, BAA-1, gets the none-BAA video from the second source, and the second dataset, BAA-2, gets the none-BAA video from the third source. Both datasets also include none-BAA videos from the first source, which helps the model learn useful features for clarifying the BAA problem instead of learning to classify the pipeline with other pipelines. Both BAA-1 and BAA-2 included three sub-sets: the training, testing and validation sets. We used the training to
checks whether the number of valid frames collected so far is greater than or equal to *n*. If the condition is met, a sub-video is constructed from the collected frame indices. Regardless of whether the sub-video is saved, the list and penalty counter are reset to allow evaluation of the next potential segment. Additionally, if the maximum number of frames *m* is reached, or if the end of the video is encountered and the collected frames meet the minimum requirement (*n*), the current frame list is similarly used to construct and save a *sub-video*. This dynamic frame selection strategy ensures that each sub-video exhibits high semantic alignment with the reference image, while tolerating limited frame inconsistencies. The integration of a penalty mechanism enables robust handling of *frame-level* noise and allows for adaptive segmentation without requiring rigid temporal boundaries. The Algorithm ?? uses the variables *maxPenalty* and *penalty* because the content of the videos in the dataset is affected by many factors such as water height, robot speed, etc. That means we accept a few frames in the sequence of frames that mismatch the condition of being the frame of the sample. The *frame_segment* and *constructVideo* functions are implemented using the OpenCV library.

Figure ?? presents an example illustrating the application of Algorithm ?? for constructing a video dataset corresponding to the BAA anomaly class. In this approach, each sub-video labeled as BAA is generated by selecting *n* consecutive frames that are temporally and visually closest to a manually annotated BAA image. Figure ?? further depicts the process of extracting and labeling such sub-videos. When a BAA anomaly is identified within a video, the algorithm extracts *n* continuous frames centered around the anomaly's timestamp. These frames are then aggregated to form a sub-video, which is subsequently labeled as belonging to the BAA class.

<figure><img src="image_3.png" /><figcaption>Fig. 2: Extracting and labeling sub-video as the sample of the BAA class.</figcaption></figure>
<caption>**Algorithm 2 Constructing sample video from image and corresponding video**</caption>

1: Given: a set of labeled images A and corresponding video V, the min and max numbers of frames in new video n and m, the min and max cosine similarity min\_cos and max\_cos, the max allowed number of frames that do not fit conditions max\_penalty, the pre-trained vision transformer model vit\_model.
2: Initialize:
   an empty list of frame index frame\_index\_list,
   penalty = 0
3: **for** image ∈ A **do**
4:   v ← corresponding video of image in V
5:   video\_frame\_set ← frame\_segment(v)
6:   vt\_image ← vit\_model.get\_feature(image)
7:   **for** i ∈ [0, len/video\_frame\_set)) **do**
8:     vt\_frame\_i ← vit\_model.get\_feature/video\_frame\_set[i])
9:     cosine\_value ← cosine(vt\_image, vt\_frame\_i)
10:    **if** cosine\_value ≥ min\_cos and cosine\_value ≤ max\_cos **then**
11:     frame\_index\_list.append(i)
12:     penalty = 0
13:    **else**
14:     penalty += 1
15:    **end if**
16:    **end for**
17:    **end if**
18:    **if** len(frame\_index\_list) == m
19:      or (i == len/video\_frame\_set)
20:      and len(frame\_index\_list) ≥ n) **then**
21:      construct\_video(frame\_index\_list[0], frame\_index\_list[len(frame\_index\_list) - 1], video\_frame\_set)
22:      frame\_index\_list.empty()
23:      penalty = 0
24:    **end if**
25:    **end for**
26:    **end for**
27:    **return** 0
# 4 Experimentation

## 4.1 Parameters and dataset

We fine-tuned TimeSformer and ViViT on our experimental datasets. ViViT requires substantial RAM; therefore, we used 4 NVIDIA A100-SXM4 GPUs with 40 GB of memory each. The software environment consists of Python 3.12.4, managed via Anaconda along with the necessary libraries. The parameters for TimeSformer and ViViT in our experiments are as follows: the *per-device train batch size* is 2 on 4 GPUs, yielding a *total batch size* of 8; the *max training steps* is calculated as (*number of videos* / *total batch size*) ×10; and the *learning rate* is 5×10⁻⁵. In creating the BAA video (and other anomalies), the parameters of the Vnframe algorithm are as follows: *max\_cos* is 1.0, *min\_cos* is 0.9, *max\_penalty* is 3. The value of *cosine similarity* will never be higher than 1. However, this parameter is needed for the algorithm to work when creating non-BAA videos. In creating the none-BAA video from *the first source*, the parameters of the Vnframe algorithm are as follows: *max\_cos* is 0.8, *min\_cos* is 0.5, *max\_penalty* is 3. We set value for *max\_cos* because we want none-BAA videos to be different from the labeled BAA image in the same full video, while *min\_cos* helps ensure that the none-BAA class doesn't have excessively noisy frames, such as pipe connection frames. Table ?? shows the number of videos in the experiment datasets. It includes training, testing and validation sets of the BAA-1, BAA-2 and T1 datasets (See section ?? for the dataset construction methods). Besides BAA, we also evaluate three other problems, the BAF, BAB and BDD. The T1 for testing the classifier of other problems are also different from each other, therefore, we name them with additional problem codes, T1-BAA, T1-BAF, T1-BAB and T1-BDD.

<table><caption>Table 1: Number of videos in the experiment dataset.</caption><thead><tr><th>Dataset</th><th>Class</th><th>Train</th><th>Test</th><th>Val</th><th>Total</th><th>Dataset</th><th>Class</th><th>Train</th><th>Test</th><th>Val</th><th>Total</th></tr></thead><tbody><tr><td rowspan="2">BAA-1</td><td>BAA</td><td>1475</td><td>185</td><td>184</td><td>1844</td><td rowspan="2">BAB-1</td><td>BAB</td><td>343</td><td>43</td><td>43</td><td>429</td></tr><tr><td>None</td><td>3256</td><td>407</td><td>407</td><td>4070</td><td>None</td><td>926</td><td>116</td><td>116</td><td>1158</td></tr><tr><td rowspan="2">BAA-2</td><td>BAA</td><td>1475</td><td>185</td><td>184</td><td>1844</td><td rowspan="2">BAB-2</td><td>BAB</td><td>343</td><td>43</td><td>43</td><td>429</td></tr><tr><td>None</td><td>10995</td><td>1375</td><td>1374</td><td>13744</td><td>None</td><td>11149</td><td>1394</td><td>1394</td><td>13937</td></tr><tr><td rowspan="2">T1-BAA</td><td>BAA</td><td>-</td><td>268</td><td>-</td><td>268</td><td rowspan="2">T1-BAB</td><td>BAB</td><td>-</td><td>64</td><td>-</td><td>64</td></tr><tr><td>None</td><td>-</td><td>3103</td><td>-</td><td>3103</td><td>None</td><td>-</td><td>3449</td><td>-</td><td>3449</td></tr><tr><td rowspan="2">BAF-1</td><td>BAF</td><td>688</td><td>86</td><td>86</td><td>860</td><td rowspan="2">BDD-1</td><td>BDD</td><td>842</td><td>106</td><td>105</td><td>1053</td></tr><tr><td>None</td><td>1824</td><td>229</td><td>228</td><td>2281</td><td>None</td><td>2121</td><td>266</td><td>265</td><td>2652</td></tr><tr><td rowspan="2">BAF-2</td><td>BAF</td><td>688</td><td>86</td><td>86</td><td>860</td><td rowspan="2">BDD-2</td><td>BDD</td><td>842</td><td>106</td><td>105</td><td>1053</td></tr><tr><td>None</td><td>10639</td><td>1330</td><td>1330</td><td>13299</td><td>None</td><td>11115</td><td>1390</td><td>1389</td><td>13894</td></tr><tr><td rowspan="2">T1-BAF</td><td>BAF</td><td>-</td><td>80</td><td>-</td><td>80</td><td rowspan="2">T1-BDD</td><td>BDD</td><td>-</td><td>309</td><td>-</td><td>309</td></tr><tr><td>None</td><td>-</td><td>3365</td><td>-</td><td>3365</td><td>None</td><td>-</td><td>3082</td><td>-</td><td>3082</td></tr></tbody></table>

## 4.2 Results

Table ?? presents the numerical test results in terms of accuracy, precision, recall and F₁ score for TimeSformer and ViViT models across various datasets. These
datasets were constructed using the Vnframe algorithm (videos composed of *n* frames) and by converting anomalous images into videos (single-frame videos). The results in Table 2 show that both TimeSformer and ViViT achieve very high accuracy, mostly above 93% and even exceeding 99% when trained and tested on the same dataset. Both models demonstrate strong performance on the video datasets (VnFrames), with ViViT slightly outperforming TimeSformer in most cases, and all reported F1 scores are above 86%, ranging from the highest of 98.11% (TimeSformer on BAF-1) to the lowest of 86.46% (TimeSformer on BDD-2). Both models demonstrate high performance when trained and tested on the video datasets (VnFrames), with ViViT slightly outperforming TimeSformer in most cases. All reported F1 scores exceed 86%, with the highest being TimeSformer on the BAF-1 dataset (F1 = 98.11%) and the lowest being TimeSformer on the BDD-2 dataset (F1 = 86.46%). Notably, the BAF and BAB codes consistently demonstrate strong performance across all types of datasets in our experiments.

<table><caption>Table 2: Numerical test result (%).</caption><thead><tr><th>Dataset</th><th>Model</th><th>Accuracy</th><th>Precision</th><th>Recall</th><th>F<sub>1</sub></th></tr></thead><tbody><tr><td>Trained/tested on BAA-1 (standard test)</td><td>TimeSformer ViViT</td><td>97.30<br>96.45</td><td>92.54<br>91.09</td><td>99.47<br>98.40</td><td>95.88<br>94.60</td></tr><tr><td>Trained on BAA-1/ Tested on T1-BAA</td><td>TimeSformer ViViT</td><td>95.25<br>93.44</td><td>76.26<br>56.55</td><td>60.73<br>84.73</td><td>67.61<br>67.83</td></tr><tr><td>Trained/tested on BAA-2 (standard test)</td><td>TimeSformer ViViT</td><td>97.88<br>98.27</td><td>91.30<br>93.41</td><td>90.81<br>91.89</td><td>91.06<br>92.64</td></tr><tr><td>Trained on BAA-2/ Tested on T1-BAA</td><td>TimeSformer ViViT</td><td>94.90<br>96.41</td><td>94.02<br>93.26</td><td>40.00<br>60.36</td><td>56.12<br>73.29</td></tr><tr><td>Trained/tested on BAF-1 (standard test)</td><td>TimeSformer ViViT</td><td>95.24<br>98.10</td><td>93.62<br>94.17</td><td>90.72<br>100.00</td><td>92.15<br>97.00</td></tr><tr><td>Trained on BAF-1/ Tested on T1-BAF</td><td>TimeSformer ViViT</td><td>97.68<br>97.56</td><td>55.00<br>52.24</td><td>61.11<br>77.78</td><td>57.89<br>62.50</td></tr><tr><td>Trained/tested on BAF-2 (standard test)</td><td>TimeSformer ViViT</td><td>98.87<br>99.01</td><td>98.53<br>98.57</td><td>81.71<br>84.15</td><td>89.33<br>90.79</td></tr><tr><td>Trained on BAF-2/ Tested on T1-BAF</td><td>TimeSformer ViViT</td><td>98.03<br>98.35</td><td>86.67<br>92.31</td><td>28.89<br>40.00</td><td>43.33<br>55.81</td></tr><tr><td>Trained/tested on BAB-1 (standard test)</td><td>TimeSformer ViViT</td><td>98.74<br>98.11</td><td>98.11<br>100.00</td><td>98.11<br>94.34</td><td>98.11<br>97.09</td></tr><tr><td>Trained on BAB-1/ Tested on T1-BAB</td><td>TimeSformer ViViT</td><td>97.92<br>93.51</td><td>40.78<br>17.05</td><td>77.78<br>83.33</td><td>53.50<br>28.30</td></tr><tr><td>Trained/tested on BAB-2 (standard test)</td><td>TimeSformer ViViT</td><td>99.86<br>99.79</td><td>100.00<br>96.30</td><td>96.23<br>98.11</td><td>98.08<br>97.20</td></tr><tr><td>Trained on BAB-2/ Tested on T1-BAB</td><td>TimeSformer ViViT</td><td>99.12<br>99.43</td><td>96.00<br>90.48</td><td>44.44<br>70.37</td><td>60.76<br>79.17</td></tr><tr><td>Trained/tested on BDD-1 (standard test)</td><td>TimeSformer ViViT</td><td>98.66<br>96.77</td><td>97.96<br>96.77</td><td>96.97<br>90.91</td><td>97.46<br>93.75</td></tr><tr><td>Trained on BDD-1/ Tested on T1-BDD</td><td>TimeSformer ViViT</td><td>94.60<br>93.78</td><td>94.38<br>95.24</td><td>46.46<br>36.92</td><td>62.27<br>53.22</td></tr><tr><td>Trained/tested on BDD-2 (standard test)</td><td>TimeSformer ViViT</td><td>98.26<br>99.67</td><td>96.51<br>99.03</td><td>78.30<br>96.23</td><td>86.46<br>97.61</td></tr><tr><td>Trained on BDD-2/ Tested on T1-BDD</td><td>TimeSformer ViViT</td><td>93.51<br>93.81</td><td>96.46<br>100.00</td><td>33.54<br>35.38</td><td>49.77<br>52.27</td></tr></tbody></table>

The effectiveness of using a single frame versus a short video segment composed of *n* frames for detecting anomalies such as BAA, BAB, BAF, and BDD
on the T1 testing dataset was compared. The evaluation demonstrates that using video segments (n frames) yields significantly better performance. As shown in Figure ?? and Table ??, both accuracy and $F_1$ score are notably lower when the input consists of only a single frame. These results confirm that incorporating temporal context enhances the model's ability to identify defects in sewer inspection videos.

<figure><img src="image_4.png" /><figcaption>Fig. 3: $F_1$ of tested on T1</figcaption></figure>

## 5 Conclusion and Future Works

In this paper, we build upon Video Vision Transformer (ViViT) and TimeSformer-based methodologies for video classification, which enable the effective capture of both spatial and temporal relationships across frames in video data. Our work specifically targets the BAA, BAF, BAB and BDD anomaly categories. Furthermore, we propose the Vnframe algorithm for constructing experimental video datasets from expert-labeled images, leveraging a pre-trained large vision model to facilitate this process. Numerical experiments conducted on the Prades-le-Lez dataset demonstrate that our approach achieves highly promising detection performance. In future work, we aim to extend our methodology to address additional types of defects within the pipeline infrastructure. We will also investigate further model optimization strategies and explore multi-label classification frameworks, as certain inspection videos may contain multiple co-occurring anomalies. Ultimately, our long-term objective includes developing models capable of supporting automatic video annotation for large-scale sewer inspection tasks

**Acknowledgements:** This research has received support from the European Union's Horizon research and innovation programme under the MSCA-SE (Marie
Skłodowska-Curie Actions Staff Exchange) grant agreement 101086252; Call: HORIZON-MSCA-2021-SE-01; Project title: STARWARS (STormwAteR and WastewAteR networkS heterogeneous data AI-driven management). We thank "Montpellier Méditerranée Métropole" for providing essential data. This research also received support from the French national project ANR (Agence Nationale de la Recherche, ANR-21-CE23-0004) CROQUIS (Collecte, représentation, complétion, fusion et interrogation de données hétérogènes et incertaines de réseaux d'eaux urbains).