# When the Small-Loss Trick is Not Enough: Multi-Label Image Classification with Noisy Labels Applied to CCTV Sewer Inspections

Keryan Chelouche, Marie Lachaize, Marine Bernard, Louise Olgiati, Remi Cuingnet

► To cite this version:

Keryan Chelouche, Marie Lachaize, Marine Bernard, Louise Olgiati, Remi Cuingnet. When the Small-Loss Trick is Not Enough: Multi-Label Image Classification with Noisy Labels Applied to CCTV Sewer Inspections. 2024. [https://arxiv.org/abs/2407.09449](https://arxiv.org/abs/2407.09449)

HAL Id: hal-04716249

https://hal.science/hal-04716249v1

Preprint submitted on 9 Oct 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
When the Small-Loss Trick is Not Enough:
Multi-Label Image Classification with Noisy
Labels Applied to CCTV Sewer Inspections.

Keryan Chelouche, Marie Lachaize, Marine Bernard, Louise Olgiati, and
Remi Cuingnet

Veolia Research & Innovation
keryan.chelouche@veolia.com
remi.cuingnet@veolia.com

**Abstract.** The maintenance of sewerage networks, with their millions of kilometers of pipe, heavily relies on efficient Closed-Circuit Television (CCTV) inspections. Many promising approaches based on multi-label image classification have leveraged databases of historical inspection reports to automate these inspections. However, the significant presence of label noise in these databases, although known, has not been addressed. While extensive research has explored the issue of label noise in single-label classification (SLC), little attention has been paid to label noise in multi-label classification (MLC). To address this, we first adapted three sample selection SLC methods (Co-teaching, CoSELFIE, and DISC) that have proven robust to label noise. Our findings revealed that sample selection based solely on the small-loss trick can handle complex label noise, but it is sub-optimal. Adapting hybrid sample selection methods to noisy MLC appeared to be a more promising approach. In light of this, we developed a novel method named MHSS (Multi-label Hybrid Sample Selection) based on CoSELFIE. Through an in-depth comparative study, we demonstrated the superior performance of our approach in dealing with both synthetic complex noise and real noise, thus contributing to the ongoing efforts towards effective automation of CCTV sewer pipe inspections.

**Keywords:** Multi label classification · Label noise · Sample selection · Sewers · Defect detection

# 1 Introduction

The sanitary sewer system is one of the main public infrastructures playing a vital role in public health and environmental protection. To illustrate the scale of these systems, in the United States, more than 1.2 million kilometers of public sewers serve approximately 240 million Americans [7] and in China, about 444 million people are served by 511,200 kilometers of sewer pipelines in urban areas [21]. The maintenance of these infrastructures depends on routine inspections of sewer pipelines using various methods and technologies to detect defects. Closed-Circuit Television (CCTV) is the most widely utilized method [32].
During this procedure, an operator controls a camera-equipped crawler that navigates through the sewer pipeline. The operator visually identifies defects, captures images of the damaged sections, and categorizes each defect using a standardized coding system.

As the volume of collected CCTV inspection data has surged in recent decades, there is a growing interest in leveraging this data to automate the inspection process. Object recognition [10, 25, 45, 48] and multi-label classification [19, 20, 46, 55] approaches offer promising solutions. However, these technologies hinge on the availability of high-quality, accurately labeled training data.

In practice, the reliance on manual inspection processes introduces significant variability in defect recognition [12]. This often manifests as label noise within historical inspection databases. Another significant source of label noise arises from the inherent limitations of the inspection camera. As it cannot capture a complete 360-degree view of the pipe interior, it is not guaranteed that all reported defects are fully visible in their associated images [19]. As label noise can severely impede the performance of supervised learning algorithms [2] due to the memorization effect [3], addressing this issue is critical.

Given the complexity of sewer pipe defects, where multiple types of damage can occur simultaneously in a single section of pipe, the problem naturally lends itself to a multi-label classification (MLC) framework [19]. Unlike single-label classification (SLC), which restricts each image to one label, MLC allows each image to be tagged with multiple labels corresponding to different defect types. This is essential to accurately reflect real-world scenarios where multiple defects often co-exist. However, while significant research has been conducted on label noise in SLC [14, 44], the field of multi-label noise remains relatively underexplored and lacks standardization such as unified evaluation procedure and noise injection strategies [4, 44]. Furthermore, naive adaptation of robust SLC methods for noisy MLC proves mostly ineffective, particularly when dealing with realistic and complex noise [4, 15, 44, 51].

This paper addresses the challenge of multi-label classification with noisy labels, with a focus on CCTV sewer pipe inspections. The main contributions of this paper are as follows:

- A review of existing noise injection strategies in MLC, along with the development of a standardized method for reporting noise levels in corrupted datasets, facilitating more reliable comparisons across studies.

- A comprehensive study of the impact of the type of noise on the performances of sample selection methods for MLC.

- A thorough adaptation of three sample selection methods from SLC to MLC, specifically Co-teaching [17] based on the small-loss trick and two hybrid methods, CoSELFIE [43], and DISC [28], along with an analysis of their performances, providing insights into their effectiveness and limitations.

- A new method called MHSS (Multi-label Hybrid Sample Selection), built upon the adapted CoSELFIE and specifically tailored for multi-label improving performances.
To the best of our knowledge, this is the first paper to demonstrate a substantial improvement in performance using a sample selection method for MLC on severely corrupted datasets with complex, mixed noise.

# 2 Related Work

## 2.1 Classification with Noisy Labels

**Single Label Classification** Song *et al.* [44] reported that real-world SLC databases can contain a significant amount of noisy labels, ranging from 8% to 38.5%. This noise can hinder model generalization, and traditional regularization techniques like data augmentation, dropout, and batch normalization may not always mitigate its impact. Consequently, there is a growing body of research focused on developing methods robust to label noise. These methods can be grouped into five categories [44]: robust architecture, robust regularization, robust loss, loss adjustment and sample selection.

Robust architecture methods modify the neural network architecture, often by adding a noise adaptation layer [16]. This aims at estimating the noise distribution in the data, to better handle noisy labels. In addition to these methods, advanced regularization techniques, such as the early-learning regularization (ELR) [30], offer alternatives for improving robustness. Other approaches either adjust the loss function, like self-adaptive training (SAT) [22] that computes the loss with exponentially-moving averaged predictions as ground-truth labels, or use intrinsically robust loss functions such as the generalized cross entropy (GCE) [58]. A different line of research tackled label noise by selecting samples to train on. This is the case of Co-teaching [17] and JoCoR [50]. More recent works proposed hybrid methods combining sample selection with other approaches. For instance, in SELFIE, Song *et al.* [43] combined it with a loss adjustment by *label refurbishing*. Similarly, in DISC, Li *et al.* [28] merged sample selection with loss adjustment, robust regularization and robust loss.

**Multi-label classification** Unlike SLC, few MLC methods have been proposed to address label noise [14, 43, 49, 52]. Xie *et al.* [53] established a comprehensive framework for addressing Class-Conditional Multi-label Noise (CCMN), distinguishing between additive noise (where negative labels become positive) and subtractive noise (where positive labels become negative) types. However, their proposed approach was exclusively tested within the context of partial multi-label learning [52], so with additive noise only. Ghiassi *et al.* [15] introduced TLCM, a promising method outperforming ASL [39] on MS-COCO [29] with up to 60% symmetric label noise. However, their approach requires both a subset of clean data and an additional set of single-label clean data, which are rarely available. To overcome these limitations, Xia *et al.* [51] proposed the holistic label correction (HLC) method. It handles multi-label noise by leveraging label dependence to correct noisy labels. Yet, HLC is not model agnostic. It is built upon the GCN-based method ADDGCN [57], while GCN based methods have
been outperformed by TResNet [19, 40]. Instead, to handle multi-label noise, Burgert et al. [4] adapted existing robust SLC methods: SAT [22], ELR [30], and JoCoR [50]. Their adaptations of SAT and ELR managed to improve classification performances in scenarios with additive, subtractive and low to moderate mixed noise. Their adaption of the JoCoR sample selection method did not consistently yield better classification performances for subtractive and mixed noise. Notably, they reported that additive noise alone had little impact on performances while mixed noise, combining both additive and subtractive noise, presented the greatest challenge.

## 2.2 Sewers

In order to automate sewer inspection, some research teams proposed to use state-of-the-art object detection algorithms to detect and identify potential defects in the pipe [25, 31, 36, 37]. To take into account the specificity of sewer pipes, Tan et al. [45] and Dang et al. [10] proposed dedicated approaches with promising results. Other research groups have suggested more advanced defect localization using instance segmentation [27, 33] or semantic segmentation [9, 26, 47, 60]. Although localizing a defect is useful, it's not essential for codification and it necessitates the creation of a dedicated dataset.

An alternative approach is to use historical inspection reports to extract labeled images of sewers and treat automation as an MLC problem. Haurum and Moeslund [19] compared various approaches with methods specifically designed for sewer inspection tasks [5, 18, 24, 34, 35, 54] alongside general MLC models such as TResNet [40], ML-GCN [6] and KSSNet [49]. As a result, for images with defects, the best performances were obtained with a TResNetL [40].

However, CCTV sewer inspection datasets are prone to label noise due to inter-operator variability [12] and limited camera view [19]. Different inspectors may indeed label the same defect differently, leading to inconsistencies in the data. Furthermore, inspection cameras cannot capture a complete 360-degree view of the pipe interior, potentially causing reported defects not to be visible in captured images. To the best of our knowledge, this issue of label noise in CCTV sewer inspection datasets, while known, has not been addressed in the literature.

# 3 Methods

## 3.1 Adapted methods

We adapted three sample selection methods from the robust SLC literature, namely Co-teaching [17], CoSELFIE [43] and DISC [28], as follows. First, softmax functions were replaced by sigmoids and the cross-entropy loss by the asymmetric loss (ASL) [39]. Then, similarly to Burgert et al. [4], instance-level selections and corrections used in SLC were substituted by their label-level equivalents, thereby allowing partial selection and correction more suited to MLC. The following describes in more details Co-teaching [17] and CoSELFIE [43], on which our method (Sec. 3.2) is based.
**Co-teaching** Co-teaching is a sample selection strategy proposed by Han et al. [17], designed to effectively handle noise in single class label datasets. This approach involves simultaneously training two networks. In each mini-batch, each network selects samples for its peer network to train on. This sample selection is carried out using the small-loss trick. It consists in selecting a progressively decreasing proportion of instances with the smallest loss.

The small-loss trick is based on the assumption that noisy labels are harder to learn and on the empirical observation that deep neural networks memorize easy instances first, and gradually adapt to hard instances as training epochs become large. This is called the memorization effect [3]. Thus, it assumes that noisy labels yield larger losses during the first training epochs.

The progressively decreasing proportion of selected instances $(1 - \tau)$ is controlled by a single hyper-parameter $\tau$ called the forget rate. This rate relates to the dataset's noise rate $\varepsilon$ via a proportionality constant $\alpha$, such that:

$$ \tau = \alpha \cdot \varepsilon \qquad (1) $$

Since $\tau$ depends on the dataset, we rather considered $\alpha$ as the hyper-parameter to tune. Notably, in the original Co-teaching paper [17], the optimal results were found with $\alpha \in [1.25, 1.5]$ and setting $\alpha$ to one often yielded satisfactory results.

**CoSELFIE** CoSELFIE [43] is a hybrid method that incorporates label refurbishment into Co-teaching. Hence, this approach also involves the concurrent maintenance of two networks.

After the selection process where each network categorizes instance labels as either clean or noisy, instance labels are also divided between refurbishable and non-refurbishable labels. At epoch $t$, a network deems an instance label $y_i$ as refurbishable if its past predictions $[\hat{y}_i^{(t-Q+1)}, \hat{y}_i^{(t-Q+2)}, \dots, \hat{y}_i^{(t-1)}, \hat{y}_i^t]$ have been consistent across the last $Q$ training epochs. The consistency is measured using entropy $H_i^{(t)}$ scaled between 0 and 1:

$$ H_i^{(t)} = \log \left( \frac{1}{K} \right) \sum_{k=1}^{K} p_{i,k}^{(t)} \log p_{i,k}^{(t)}, \quad \text{with} \quad p_{i,k}^{(t)} = \frac{1}{Q} \sum_{q=0}^{Q-1} \mathbb{1}_{\hat{y}_i^{(t-q)} = k} \qquad (2) $$

where $K$ is the number of classes in the SLC settings. Note that, in MLC, $K = 2$ for each label. An instance label $y_i$ with a scale entropy $H_i^{(t)}$ lower than a threshold $\theta$ is regarded as refurbishable. In this case, the most frequently predicted label in past $Q$ epochs becomes the new corrected ground-truth. Finally its peer network is trained on all but non-refurbishable noisy instance labels.

## 3.2 Our method

This section describes our method MHSS (Multi-label Sample Selection), which is built upon the MLC adaptation of CoSELFIE (Sec. 3.1). It is based on two key, yet simple, modifications: one concerning the selection and the other related to the correction process.
**Class-dependent Noise Rate (CDNR)** Previous approaches based on the small-loss trick used a single noise rate $\epsilon$ for all classes $c \in C$. However, in the MLC setting, assuming a constant noise rate across classes is often unrealistic [51]. Therefore, we modified the forget rate $\tau$ to be class dependent. With $\epsilon_c$ the noise rate of class c, it is then defined as:

$$ \forall c \in C, \tau_c := \alpha \cdot \epsilon_c \qquad (3) $$

**Joint Correction Criterion (JCC)** The rationale behind CoSELFIE is that classification performance can be improved by (i) removing noisy labels through the sample selection step and (ii) augmenting the count of clean labels via the label correction phase (label refurbishment). By raising the entropy selection threshold $\theta$, more corrected labels can be included, allowing for the acceptance of instance labels that show less consistency across prior training epochs. However, this raises the risk of including wrongly corrected labels.

We claim that a more effective method to relax this acceptance criterion is to utilize a joint correction criterion leveraging the dual network architecture by considering an instance as refurbishable if at least one of the two networks deems it refurbishable. When a label instance is considered as refurbishable by both networks, each network corrects the instance labels for its peer. On the other hand, when only one considers the instance label as refurbishable, the corresponding label correction is applied for both networks.

## 3.3 Noise Injection

The standard procedure for assessing the robustness of methods to label noise consists in injecting a controlled synthetic noise into the training datasets. In Single Label Classification (SLC), methods are systematically tested on datasets with symmetric noise and with pair noise [44]. However there is no consensus in MLC with label noise [4]. While some studies only employed uniform noise injection [59], more recent research proposed more advanced techniques tailored to the specificities of MLC [4, 15, 51]. However, these techniques are not standardized and are often unique to each study. This variability presents a challenge when comparing results. Notably, the reported level of label corruption is not equivalent across different studies, making direct comparisons unfeasible.

For the purposes of this paper, we categorized noise injection methods as naive or complex.

**Naive Noise Injection** Some studies introduced noise uniformly by randomly flipping a proportion of the labels according to a fixed noise rate [59]. However, due to the scarcity of positive labels in MLC, this method predominantly generates additive noise (where negative labels become positive) and minimally produces subtractive noise (where positive labels become negative). This imbalance significantly alters the class distribution and the number of labels per instance. Moreover, it has been demonstrated that additive noise tends to have a less adverse impact on performance compared to subtractive noise [4]
**Complex Noise Injection** More recent studies introduced MLC specific noise that maintained certain properties of the clean dataset [4,51]. More specifically, Burgert et al. [4] injected *mixed noise*: Given a noise rate, for each class, the corresponding proportion of positive labels were flipped and then the same number of negative labels were selected and flipped to positive. This approach ensures an equal proportion of additive and subtractive noise, while preserving the class distribution of the original dataset. However, due to the inherent sparsity of positive labels in most multi-label datasets, the actual number of labels that are flipped will be significantly lower than the specified noise rate parameter.

As for Xia et al. [51], they employed a *class-dependent transition matrix* to swap labels between classes within the same instance as opposed to flipping labels from positive to negative or vice versa. Their method ensures an equal proportion of additive and subtractive noise, while preserving the original dataset's number of labels per instance. However, given the sparsity of positive labels in most multi-label datasets, the majority of these swaps occurs between two negative labels. Consequently, similarly to *mixed noise* [4], the actual number of flipped labels is much lower than the noise rate parameter reported.

In this study, we defined $\varepsilon$, the level of noise in a dataset, as the percentage of labels that have been genuinely inverted, regardless of the noise injection technique employed. Having a relevantly defined noise level is necessary to ensure consistent reporting of results. Furthermore, $\varepsilon$ is used to define the forget rate parameter $\tau$.

# 4 Experiments on synthetic noise

## 4.1 Experimental Setup

To assess the robustness to label noise of the methods described in Section 3, we evaluated their performances on two publicly available datasets, UcMerced [56] and TreeSatAI [1], with both *naive* and *complex* noise injections.

**Datasets** The UcMerced Dataset, a land use image collection derived from the USGS National Map Urban Area Imagery, consists of 2100 RGB images. These images are divided into 17 diverse classes, encompassing a variety of subjects from vehicles to natural and urban elements. The dataset is partitioned into 1700 training images, alongside 184 images for validation and 216 for testing. On average, each image in this dataset is annotated with 3.3 labels, corresponding to a positive label prevalence of approximately 20%.

The TreeSatAI Dataset is a larger image dataset for forest applications with highly reliable labels, collected in German forest management services. For our study, we utilize the *aerial 3-band* version of the dataset, which encompasses a total of 50,381 images distributed across 15 multi-label *genus* classes. The dataset is divided into 36,269 training images, 9,068 validation images, and 5,044 test images. On average, each image in this dataset is annotated with 1.87 labels, which translates to a positive label prevalence of approximately 12.5%. Visual examples of both datasets are available in Appendix A.
**Noise Injection** In this study, we used both naive and complex noise injections. For the naive noise injection, we simply flipped a proportion of labels of $\epsilon$. As for the complex noise, the mixed noise strategy was adopted (Sec. 3.3). It allowed us to preserve the class distributions of the original datasets. To be able to compare our results with those obtained by Burgert et al. [4], a noise rate $\epsilon$ of 20% for the UcMerced Dataset and 12.5% for the TreeSatAI Dataset was chosen, which corresponded to a 50% mixed noise reported in their study.

**Experimental setup** The same training set-up was used for every experiment. We used a TResNetM [40] pretrained on ImageNet 21k [38] as a backbone for every model. The loss function was the asymmetric loss (ASL) with its default hyper-parameters [39]. Models were trained on input resolution 256 x 256, for 30 epochs, using an Adam optimizer [23] with default parameters and a 1-cycle policy [42] with a maximum learning rate of $10^{-3}$. The batch size was set to 64 for all experiments, except for DISC, where it was reduced to 32 due to memory constraints. For data augmentation, we used RandAugment [8]. On the UcMerced dataset, each experiment was conducted three times to ensure the robustness and reliability of our findings. However, due to computational costs, experiments on the TreeSatAI dataset were conducted only once.

For evaluation, we followed conventional settings and reported the mean average precision (mAP). Additionally, to evaluate the label selection process, we monitored the *label precision* (ratio of selected clean labels to selected labels) and the *label recall* (ratio of selected clean labels to clean labels) during training:

$$label\ precision := \frac{\#\ clean\ &\ selected}{\#\ selected}, \quad label\ recall := \frac{\#\ clean\ &\ selected}{\#\ clean} \quad (4)$$

All models were compared to a baseline model: TResNetM [40] with ASL [39].

## 4.2 Sample selection with the small-loss trick

**Sample selection on different types of noise** The performances of Co-teaching were assessed on the UcMerced dataset with naive and complex noise. Additionally to the baseline model, Co-teaching was compared to a *cheater model* which simulated an ideal sample selection scenario where the Co-teaching selection would have perfectly excluded all noisy samples and trained on clean data only. We tested different forget rate factors $\alpha$ (Sec. 3) on the validation dataset: 0.25, 0.5, 1 and 1.5. We selected the value that gave the best mAP.

Results are presented in Table 1. We observed that introducing any type of noise adversely affected performances, with a more pronounced decrease with complex noise. Moreover, while Co-teaching considerably improved performances with naive noise, it fell short in effectively handling complex noise, showing only slight improvements over the baseline. This underscores the necessity of evaluating methods beyond the scope of naive noise injection. On the contrary, the *cheater model*'s performances remained closely aligned with performances on
the clean dataset for both types of noise. This shows that, while sample selection approaches could indeed be beneficial for MLC with noisy labels, the *small-loss trick* alone is not enough to handle complex label noise in MLC.

<table><caption>**Table 1:** Comparison of Co-teaching performances on the UcMerced Dataset ($\epsilon$ = 0.2).</caption><thead><tr><th rowspan="2">Method</th><th rowspan="2">Clean</th><th colspan="2">Naive noise</th><th colspan="2">Complex noise</th></tr><tr><th>(α = 1.5)</th><th>(α = 0.25)</th><th>(α = 1.5)</th><th>(α = 0.25)</th></tr></thead><tbody><tr><td>Baseline</td><td>80.94 ± 0.97</td><td>48.97 ± 0.57</td><td>40.13 ± 0.38</td><td></td><td></td></tr><tr><td>Co-teaching [17]</td><td>79.88 ± 0.20</td><td>72.77 ± 0.77</td><td>41.90 ± 1.24</td><td></td><td></td></tr><tr><td>Cheater</td><td>-</td><td>79.88 ± 0.54</td><td>78.47 ± 0.86</td><td></td><td></td></tr></tbody></table>

**Forget Rate Analysis** To comprehend the failure of the *small-loss trick* in dealing with *complex* noise, we compared the performances of Co-teaching with different forget rate factors $\alpha$ in the case of *naive* and *complex* noise injection (Tab. 2). We note a contrasting trend in the performance. Under naive noise injection, the mAP increased with the value of $\alpha$. On the contrary, under *complex* noise injection, performance decreased as $\alpha$ increased past 0.25.

<table><caption>**Table 2:** Co-teaching performances with varying $\alpha$ on UcMerced dataset ($\epsilon$ = 0.2) with *naive* and *complex* noise.</caption><thead><tr><th>α</th><th>0</th><th>0.25</th><th>0.5</th><th>1</th><th>1.5</th></tr></thead><tbody><tr><td>naive</td><td>47.51 ± 0.69</td><td>48.62 ± 0.51</td><td>52.47 ± 0.94</td><td>64.64 ± 1.75</td><td><strong>72.77 ± 0.77</strong></td></tr><tr><td>complex</td><td>39.36 ± 0.81</td><td><strong>41.90 ± 1.24</strong></td><td>40.35 ± 0.11</td><td>36.96 ± 0.86</td><td>35.05 ± 0.37</td></tr></tbody></table>

Intuitively, an $\alpha$ factor of less than one wouldn't fully eliminate noisy labels, even with an optimal selection strategy. Conversely, assuming an efficient selection strategy, a greater $\alpha$ would discard a larger proportion of noisy labels. However, a significantly higher $\alpha$ (larger than 1) would eventually exhaust the pool of noisy samples, leading to the unintended discard of a substantial number of clean samples. Han et al. [17] demonstrated in their research that optimal results are typically achieved with $\alpha$ ranging from 1.25 to 1.5, although a value of 1 consistently yields satisfactory outcomes. In subsequent studies, whether applied to multi-class scenarios or multi-label adaptations, researchers often default to an $\alpha$ value of 1 or do not specify the $\alpha$ value used. The fact that our best results were obtained with an $\alpha$ value considerably less than 1 is then unexpected and warrants further investigation.

To further analyse the *small-loss trick* selection, we graphed the label recall and precision (as defined in 4.1) averaged over the final 10 epochs of training in
relation to the forget rate factor $\alpha$. In Figure 1a, we note a linear increase in label precision from 0 to 1 in the case of naive noise. This increase continues, albeit at a slower pace, from 1 to 1.5, eventually reaching a plateau at a label precision of 0.98. This result aligns with the optimal range of the forget rate factor $\alpha$, previously established in the context of SLC [17]. However, the situation is markedly different with complex noise. In this case, label precision hardly increased past $\alpha = 0.25$, and hit a plateau at $\alpha = 1$ just short of 0.85. This stark contrast highlights the limitations of the small-loss trick when dealing with complex noise. As shown by the label recall curve (Fig. 1b), increasing $\alpha$ beyond 0.25 only resulted in training on a reduced dataset without reducing the amount of noise remaining in the dataset.

<figure><img src="image_2.png" /><figcaption>Fig. 1: Co-teaching average (a) label precision and (b) label recall over the last 10 epochs versus the forget rate factor $\alpha$ for naive uniform and complex mixed noise</figcaption></figure>

## 4.3 Methods Comparison

As described in Section 3.2, our method is an extension of CoSELFIE, which in turn is based on Co-teaching (Sec. 3.1). For a more comprehensive view, not only did we compare our method to both, we also included an adaptation of DISC [28]. DISC is a state of the art hybrid sample selection not based on the small-loss trick. We tested different values for the forget rate factor $\alpha$ (Sec. 3.1) on the validation dataset: 0.25, 0.5 and 1 for both datasets. We selected the parameter values that gave the best mAP. We also tested different values of the entropy threshold ($\theta \in \{0.05, 0.1, 0.15\}$). We selected $\theta$ equal to 0.05 as it gave the best performances, regardless of the method. mAP achieved on UcMerced and TreeSatAI were reported in Table 3. All the methods showed improvement over baseline. Our proposed method achieved the best performances on both datasets. It reached a mAP of 50.03% and 43.71% on UcMerced and TreeSatAI respectively. DISC obtained the second best performance on UcMerced with a
48.28% mAP although it underperformed compared to CoSELFIE on TreeSatAI, scoring 31.07% versus 35.99%. Additional results with varying ε are available in Appendix B.

<table><caption>**Table 3:** Results (mAP in %) on datasets injected with mixed noise.</caption><thead><tr><th rowspan="2">Method</th><th>UcMerced</th><th>TreeSatAI</th></tr><tr><th>ε = 0.2</th><th>ε = 0.125</th></tr></thead><tbody><tr><td>ASL [39]</td><td>40.13 ± 0.38</td><td>24.29</td></tr><tr><td>Co-teaching [17]</td><td>41.90 ± 1.24</td><td>28.48</td></tr><tr><td>CoSELFIE [43]</td><td>43.17 ± 1.19</td><td><u>35.99</u></td></tr><tr><td>DISC [28]</td><td>48.28 ± 1.14</td><td>31.07</td></tr><tr><td>MHSS (Ours)</td><td><u>50.03 ± 2.01</u></td><td><u>43.71</u></td></tr></tbody></table>

Table 4 further presents comparative analysis of mAP improvements upon the baseline for various methods from the literature applied to the UcMerced and TreeSatAI datasets. Our proposed method demonstrates consistent superior performance across both datasets. Interestingly, our implementation of Co-teaching surpasses JoCoR [50], which is surprising as JoCoR usually outperforms Co-teaching in SLC [50]. This might be due to the critical role of the α parameter tuning in adapting the small-loss-based sample selection for MLC.

<table><caption>**Table 4:** mAP improvements upon baseline on the UcMerced and TreeSatAI datasets. Results of the methods denoted with an asterisk (*) are taken from [4].</caption><thead><tr><th>Method</th><th>UcMerced</th><th>TreeSatAI</th></tr></thead><tbody><tr><td>SAT* [22]</td><td>+0</td><td>+4</td></tr><tr><td>ELR* [30]</td><td>+1</td><td>+5</td></tr><tr><td>JoCoR* [50]</td><td>+0</td><td>+3</td></tr><tr><td>Co-teaching [17]</td><td>+1.8</td><td>+4.2</td></tr><tr><td>CoSELFIE [43]</td><td>+3</td><td>+11.7</td></tr><tr><td>DISC [28]</td><td><u>+8.1</u></td><td>+6.8</td></tr><tr><td>MHSS (Ours)</td><td><u>+9.9</u></td><td><u>+19.4</u></td></tr></tbody></table>

**Insights into performance improvements** To better understand why our proposed method achieved superior performances, we evaluated the performance of CoSELFIE and our method for different values of α. As observed in Table 5, our method and CoSELFIE exhibited similar performances on UcMerced and TreeSatAI for an α of 0.25. However, as α increased, our method began to substantially outperform CoSELFIE.
<table><caption>**Table 5:** Performances of CoSELFIE and our method for varying value of α.</caption><thead><tr><th>Dataset</th><th>Method</th><th>α = 0.25</th><th>α = 0.5</th><th>α = 1</th></tr></thead><tbody><tr><td rowspan="2">UcMerced<br>(ε = 0.2)</td><td>CoSELFIE</td><td><b>43.12 ± 1.21</b></td><td>41.96 ± 1.92</td><td>41.47 ± 1.33</td></tr><tr><td>MHSS (Ours)</td><td>43.17 ± 1.19</td><td><b>50.03 ± 2.01</b></td><td>47.58 ± 1.52</td></tr><tr><td rowspan="2">TreeSatAI<br>(ε = 0.125)</td><td>CoSELFIE</td><td>33.21</td><td>35.70</td><td>35.99</td></tr><tr><td>Ours</td><td>34.31</td><td>38.38</td><td><b>43.71</b></td></tr></tbody></table>

To gain a deeper understanding of these results, we graphed in Fig. 2a the average label precision (Sec. 4.1) over the final 10 training epochs on UcMerced in relation to the forget rate factor α for Co-teaching, CoSELFIE and our method. We also similarly plotted the proportion of selected labels on which training is actually performed in Fig. 2b. We can observe that, for our method, α ≥ 0.5 led to a better label precision than for Co-teaching and CoSELFIE. In addition, both our method and CoSELFIE had a very high proportion of selected labels (over 92%) for all value of α. As shown by these experiments, performance improvement of our method is related to its capability to better reject or correct noisy labels with a higher α.

<figure><img src="image_3.png" /><figcaption>**Fig. 2:** (a) Average label precision and (b) average proportion of selected labels over the final 10 training epochs on UcMerced in relation to the forget rate factor α for Co-teaching, CoSELFIE and our method.</figcaption></figure>

## 4.4 Ablation study

We performed an ablation study of our method MHSS. Results are presented in Table 6. We started from the baseline and progressively added the different components of our method. Selection alone is equivalent to Co-teaching. Selection and Correction corresponds to CoSELFIE. CDNR and JCC are described
in details in Section 3.2. Every component individually improved performances on UcMerced, with the biggest impact being the inclusion of either CDNR or JCC to CoSELFIE. However, on TreeSatAI, the addition of CDNR alone to Co-teaching actually decreased performance, same thing with JCC to CoSELFIE. It's only when combined together that those components had a positive impact.

Our method outperformed every combination of components in both datasets. The addition of *Correction* also always improved performances, highlighting the pertinence of hybrid sample selection methods for MLC.

<table><caption>**Table 6:** Ablation study of MHSS</caption><thead><tr><th>Selection</th><th>CDNR</th><th>Correction</th><th>JCC</th><th>UcMerced<br>(α = 0.5)</th><th>TreeSatAI<br>(α = 1)</th></tr></thead><tbody><tr><td>✓</td><td></td><td></td><td></td><td>40.13</td><td>24.29</td></tr><tr><td>✓</td><td>✓</td><td></td><td></td><td>40.35 +0.22</td><td>28.48 +4.19</td></tr><tr><td>✓</td><td></td><td>✓</td><td></td><td>42.12 +1.99</td><td>24.60 +0.31</td></tr><tr><td>✓</td><td></td><td></td><td>✓</td><td>41.96 +1.82</td><td>35.99 +11.70</td></tr><tr><td>✓</td><td></td><td>✓</td><td>✓</td><td><u>48.09</u> +7.96</td><td>32.10 +7.81</td></tr><tr><td>✓</td><td>✓</td><td>✓</td><td></td><td>47.21 +7.08</td><td><u>42.62</u> +18.33</td></tr><tr><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td><b>50.03</b> +9.90</td><td><b>43.71</b> +19.42</td></tr></tbody></table>

# 5 Experiments on real noise

**Dataset** Our database is composed of 249,093 images from 175,011 sewer inspection reports complying with European standard EN 13508-2 A1 [13]. We used 70% for training, 10% for validation and 20% for testing. We ensured that no image from the same inspection report was present in more than one set.

Such dataset are known to have label noise due to (i) inconsistencies between operators in on defect reporting [12] and (ii) the fact that it is not guaranteed that all reported defects are fully visible in their associated image [19]. To build a clean test set, eight experts annotated a subset of 6,131 images from the test set. Each image was annotated by three of them and the ground truth labels were estimated from the multiple annotations using the Dawid-Skene approach [11]. To simplify the procedure, we considered a subset of 12 defects<fnref n="1" /> (cf. Appendix A). The resulting cleansed test dataset was the one used for evaluation.

**Experimental setup** The method that achieved the highest performances on the SewerML benchmark [19] is a two-stage network. The first stage distinguishes between images with and without defects with the binary classifier from Xie et al. [54], while the second stage classifies the type of defect with a TResNetL [40]. Our dataset is only composed of images from inspection reports, so we ignored

<footnote n="1">¹ Defect list: BAB, BAC, BAG, BAH, BAI, BAJ, BBA, BBB, BBC, BBD, BCA, BCC.</footnote>
the first stage and focused solely on the second stage. As a result, we used a TResNetL model [40] pre-trained on Imagenet [41]. We compared MHSS with the baseline (ASL), Co-teaching and CoSELFIE. We used a forget rate factor $\alpha$ of one. We discarded DISC since it is based on data augmentation transformations that do not preserve the pipe geometry. Each model was trained for 30 epochs with a batch size of 32. We used the asymmetric loss (ASL) [39] with the default hyperparameters and an Adam optimizer [23] with a learning rate of $10^{-4}$.

**Results** Results are presented on Tab. 7. The Co-teaching approach failed to improve the classification results with an mAP of 55.7% while the baseline method reached 56.0% mAP. CoSELFIE and our method (MHSS) achieved a 0.2% and 0.6% improvement in mAP respectively. Coupled with our results in Section 4.2, this finding suggests that real-world noise more closely resembles mixed noise than uniform noise. Consequently, it highlights the importance of developing methods capable of effectively handling mixed noise.

<table><caption>**Table 7:** Results on the sewer pipe inspection image dataset.</caption><thead><tr><td>Method</td><td>mAP (%)</td></tr></thead><tbody><tr><td>ASL</td><td>56.0</td></tr><tr><td>Co-teaching</td><td>55.7</td></tr><tr><td>CoSELFIE</td><td>56.2</td></tr><tr><td>MHSS (Ours)</td><td><strong>56.6</strong></td></tr></tbody></table>

# 6 Conclusion

This paper addressed label noise in multi-label classification (MLC). We adapted three sample selection single-label classification (SLC) methods - Co-teaching, CoSELFIE, and DISC - to MLC. These methods were evaluated on two datasets, UcMerced and TreeSatAI, with synthetically introduced *complex* label noise. Unlike previous works, we reported noise levels as the true proportions of flipped labels to allow for better comparisons across studies. This evaluation revealed three key findings. First, it stressed out the importance of using *complex* realistic noise over *naive* uniform noise. Secondly, it revealed that the small-loss trick could manage noisy label MLC, provided a suitable forget rate factor $\alpha$ and a properly defined noise level $\epsilon$. However, these methods were found to be less efficient and were outperformed by hybrid methods.

Based on these insights, we proposed a new method built on the CoSELFIE algorithm tailored for MLC. The proposed approach yielded better sample selections and outperformed others in evaluations using both synthetic *complex* noise and real-world scenarios, showing its potential for the automation of CCTV sewer inspection or other industrial applications.
References

1. Ahlswede, S., Schulz, C., Gava, C., Helber, P., Bischke, B., Förster, M., Arias, F., Hees, J., Demir, B., Kleinschmit, B.: TreeSatAI Benchmark Archive: a multi-sensor, multi-label dataset for tree species classification in remote sensing. Earth System Science Data 15(2), 681–695 (2023). https://doi.org/10.5194/essd-15-681-2023, https://essd.copernicus.org/articles/15/681/2023/

2. Algan, G., Ulusoy, I.: Label noise types and their effects on deep learning. arXiv preprint arXiv:2003.10471 (2020)

3. Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M.S., Marharaj, T., Fischer, A., Courville, A., Bengio, Y., et al.: A closer look at memorization in deep networks. In: International conference on machine learning. pp. 233–242. PMLR (2017)

4. Burgert, T., Ravanbakhsh, M., Demir, B.: On the effects of different types of label noise in multi-label remote sensing image classification. IEEE Transactions on Geoscience and Remote Sensing PP, 1–1 (01 2022). https://doi.org/10.1109/TGRS.2022.3226371

5. Chen, K., Hu, H., Chen, C., Chen, L., He, C.: An intelligent sewer defect detection method based on convolutional neural network. In: 2018 IEEE International conference on information and automation (ICIA). pp. 1301–1306. IEEE (2018)

6. Chen, Z.M., Wei, X.S., Wang, P., Guo, Y.: Multi-label image recognition with graph convolutional networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5177–5186 (2019)

7. of Civil Engineers), A.A.S.: 2021 infrastructure report card: wastewater (2021), https://www.infrastructurereportcard.org/cat-item/wastewater/

8. Cubuk, E.D., Zoph, B., Shlens, J., Le, Q.V.: Randaugment: Practical automated data augmentation with a reduced search space. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. pp. 702–703 (2020)

9. Dang, L.M., Wang, H., Li, Y., Nguyen, L.Q., Nguyen, T.N., Song, H.K., Moon, H.: Lightweight pixel-level semantic segmentation and analysis for sewer defects using deep learning. Construction and Building Materials 371, 130792 (2023)

10. Dang, L.M., Wang, H., Li, Y., Nguyen, T.N., Moon, H.: DefectTR: End-to-end defect detection for sewage networks using a transformer. Construction and Building Materials 325, 126584 (2022)

11. Dawid, A.P., Skene, A.M.: Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics) 28(1), 20–28 (1979)

12. Dirksen, J., Clemens, F., Korving, H., Cherqui, F., Le Gauffre, P., Ertl, T., Plihal, H., Müller, K., Snaterse, C.: The consistency of visual sewer inspection data. Structure and infrastructure engineering 9(3), 214–228 (2013)

13. European Committee For Standardization: NF EN 13508-2+A1 – Investigation and assessment of drain and sewer systems outside buildings - Part 2 : visual inspection coding system

14. Frénay, B., Verleysen, M.: Classification in the presence of label noise: a survey. IEEE transactions on neural networks and learning systems 25(5), 845–869 (2013)

15. Ghiassi, A., Pene, C.O., Birke, R., Chen, L.: Trusted loss correction for noisy multi-label learning. In: Khan, E., Gonen, M. (eds.) Proceedings of The 14th Asian Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 189, pp. 343–358. PMLR (12–14 Dec 2023), https://proceedings.mlr.press/v189/ghiassi23a.html
16. Goldberger, J., Ben-Reuven, E.: Training deep neural-networks using a noise adaptation layer. In: International conference on learning representations (2017)

17. Han, B., Yao, Q., Yu, X., Niu, G., Xu, M., Hu, W., Tsang, I., Sugiyama, M.: Co-teaching: Robust training of deep neural networks with extremely noisy labels. In: NeurIPS. pp. 8535–8545 (2018)

18. Hassan, S.I., Dang, L.M., Mehmood, I., Im, S., Choi, C., Kang, J., Park, Y.S., Moon, H.: Underground sewer pipe condition assessment based on convolutional neural networks. Automation in Construction **106**, 102849 (2019)

19. Haurum, J.B., Moeslund, T.B.: Sewer-ml: A multi-label sewer defect classification dataset and benchmark. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 13456–13467 (June 2021)

20. Hu, C., Dong, B., Shao, H., Zhang, J., Wang, Y.: Toward purifying defect feature for multilabel sewer defect classification. IEEE Transactions on Instrumentation and Measurement **72**, 1–11 (2023)

21. Huang, D., Liu, X., Jiang, S., Wang, H., Wang, J., Zhang, Y.: Current state and future perspectives of sewer networks in urban china. Frontiers of environmental science & engineering **12**, 1–16 (2018)

22. Huang, L., Zhang, C., Zhang, H.: Self-adaptive training: Bridging supervised and self-supervised learning. IEEE transactions on pattern analysis and machine intelligence **46**(3), 1362–1377 (2022)

23. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

24. Kumar, S.S., Abraham, D.M., Jahanshani, M.R., Iseley, T., Starr, J.: Automated defect classification in sewer closed circuit television inspections using deep convolutional neural networks. Automation in Construction **91**, 273–283 (2018)

25. Kumar, S.S., Wang, M., Abraham, D.M., Jahanshani, M.R., Iseley, T., Cheng, J.C.: Deep learning-based automated detection of sewer defects in CCTV videos. Journal of Computing in Civil Engineering **34**(1), 04019047 (2020)

26. Li, M., Li, M., Ren, Q., Li, H., Xiao, L., Fang, X.: Pipetransunet: Cnn and transformer fusion network for semantic segmentation and severity quantification of multiple sewer pipe defects. Applied Soft Computing **159**, 111673 (2024)

27. Li, Y., Wang, H., Dang, L.M., Piran, M.J., Moon, H.: A robust instance segmentation framework for underground sewer defect detection. Measurement **190**, 110727 (2022)

28. Li, Y., Han, H., Shan, S., Chen, X.: DISC: Learning from Noisy Labels via Dynamic Instance-Specific Selection and Correction. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 24070–24079 (2023). https://doi.org/10.1109/CVPR52729.2023.02305

29. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pp. 740–755. Springer (2014)

30. Liu, S., Niles-Weed, J., Razavian, N., Fernandez-Granda, C.: Early-learning regularization prevents memorization of noisy labels. Advances in Neural Information Processing Systems **33** (2020)

31. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.: Ssd: Single shot multibox detector. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14. pp. 21–37. Springer (2016)

32. Liu, Z., Kleiner, Y.: State of the art review of inspection technologies for condition assessment of water pipes. Measurement **46**(1), 1–15 (2013)
33. Ma, D., Fang, H., Wang, N., Zheng, H., Dong, J., Hu, H.: Automatic defogging, de-blurring, and real-time segmentation system for sewer pipeline defects. Automation in Construction **144**, 104595 (2022)

34. Meijer, D., Scholten, L., Clemens, F., Knobbe, A.: A defect classification methodology for sewer image sets with convolutional neural networks. Automation in Construction **104**, 281–298 (2019)

35. Myrans, J., Everson, R., Kapelan, Z.: Automated detection of fault types in CCTV sewer surveys. Journal of Hydroinformatics **21**(1), 153–163 (2019)

36. Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 779–788 (2016)

37. Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence **39**(6), 1137–1149 (2016)

38. Ridnik, T., Ben-Baruch, E., Noy, A., Zelnik-Manor, L.: Imagenet-21k pretraining for the masses. In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) (2021), https://openreview.net/forum?id=Zkj_VcZ6ol

39. Ridnik, T., Ben-Baruch, E., Zamir, N., Noy, A., Friedman, I., Protter, M., Zelnik-Manor, L.: Asymmetric loss for multi-label classification. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 82–91 (2021)

40. Ridnik, T., Lawen, H., Noy, A., Ben Baruch, E., Sharir, G., Friedman, I.: TResNet: High Performance GPU-Dedicated Architecture, booktitle = Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). pp. 1400–1409 (January 2021)

41. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. International journal of computer vision **115**, 211–252 (2015)

42. Smith, L.N.: A disciplined approach to neural network hyper-parameters: Part 1 - learning rate, batch size, momentum, and weight decay. CoRR abs/1803.09820 (2018), http://arxiv.org/abs/1803.09820

43. Song, H., Kim, M., Lee, J.G.: SELFIE: Refurbishing unclean samples for robust deep learning. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 97, pp. 5907–5915. PMLR (09–15 Jun 2019), https://proceedings.mlr.press/v97/song19b.html

44. Song, H., Kim, M., Park, D., Shin, Y., Lee, J.G.: Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems (2022)

45. Tan, Y., Cai, R., Li, J., Chen, P., Wang, M.: Automatic detection of sewer defects based on improved you only look once algorithm. Automation in Construction **131**, 103912 (2021)

46. Tao, M., Wan, L., Wang, H., Su, T.: Cafen: A correlation-aware feature enhancement network for sewer defect identification. In: 2022 21st International Symposium on Communications and Information Technologies (ISCIT). pp. 204–209. IEEE (2022)

47. Wang, M., Cheng, J.C.: A unified convolutional neural network integrated with conditional random field for pipe defect segmentation. Computer-Aided Civil and Infrastructure Engineering **35**(2), 162–177 (2020)
48. Wang, M., Kumar, S.S., Cheng, J.C.: Automated sewer pipe defect tracking in CCTV videos based on defect detection and metric learning. Automation in Construction 121, 103438 (2021)

49. Wang, Y., He, D., Li, F., Long, X., Zhou, Z., Ma, J., Wen, S.: Multi-label classification with label graph superimposing. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 12265–12272 (2020)

50. Wei, H., Feng, L., Chen, X., An, B.: Combating noisy labels by agreement: A joint training method with co-regularization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2020)

51. Xia, X., Deng, J., Bao, W., Du, Y., Han, B., Shan, S., Liu, T.: Holistic label correction for noisy multi-label classification. In: 2023 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 1483–1493 (2023). https://doi.org/10.1109/ICCV51070.2023.00143

52. Xie, M.K., Huang, S.J.: Partial multi-label learning. In: Proceedings of the AAAI conference on artificial intelligence. vol. 32 (2018)

53. Xie, M.K., Huang, S.J.: CCMN: A General Framework for Learning With Class-Conditional Multi-Label Noise. IEEE Transactions on Pattern Analysis and Machine Intelligence 45(1), 154–166 (2023). https://doi.org/10.1109/TPAMI.2022.3141240

54. Xie, Q., Li, D., Xu, J., Yu, Z., Wang, J.: Automatic detection and classification of sewer defects via hierarchical deep learning. IEEE Transactions on Automation Science and Engineering 16(4), 1836–1847 (2019)

55. Xu, S., Gong, Y., Zhu, X., Chang, S., Wang, Y., Yan, L.: Sewer-MoE: A tuned Mixture of Experts Model for Sewer Defect Classification. In: Proceedings of the International Conference on Computer Vision and Deep Learning. pp. 1–5 (2024)

56. Yang, Y., Newsam, S.: Bag-of-visual-words and spatial extensions for land-use classification. In: ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (ACM GIS) (2010)

57. Ye, J., He, J., Peng, X., Wu, W., Qiao, Y.: Attention-driven dynamic graph convolutional network for multi-label image recognition. In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16. pp. 649–665. Springer (2020)

58. Zhang, Z., Sabuncu, M.: Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems 31 (2018)

59. Zhao, W., Gomes, C.: Evaluating multi-label classifiers with noisy labels. arXiv preprint arXiv:2102.08427 (2021)

60. Zhou, Q., Situ, Z., Teng, S., Liu, H., Chen, W., Chen, G.: Automatic sewer defect detection and severity quantification based on pixel-level semantic segmentation. Tunnelling and Underground Space Technology 123, 104403 (2022)