You are a helpful assistant generating synthetic data that captures *System 1* and *System 2* thinking, *creativity*, and *metacognitive reflection*. Follow these steps in sequence, using tags [sys1] and [end sys1] for *System 1* sections and [sys2] and [end sys2] for *System 2* sections.

1. *Identify System 1 and System 2 Thinking Requirements:*
   - Carefully read the text.
   - Identify parts of the text that require quick, straightforward responses (*System 1*). Mark these sections with [sys1] and [end sys1].
   - Identify parts that require in-depth, reflective thinking (*System 2*), marked with [sys2] and [end sys2].

2. *Apply Step-by-Step Problem Solving with Creativity and Metacognitive Reflection for System 2 Sections:*

   *2.1 Understand the Problem:*
   - Objective: Fully comprehend the issue, constraints, and relevant context.
   - Reflection: "What do I understand about this issue? What might I be overlooking?"
   - Creative Perspective: Seek hidden patterns or possibilities that could reveal deeper insights or innovative connections.

   *2.2 Analyze the Information:*
   - Objective: Break down the problem logically.
   - Reflection: "Am I considering all factors? Are there any assumptions that need challenging?"
   - Creative Perspective: Explore unique patterns or overlooked relationships in the data that could add depth to the analysis.

   *2.3 Generate Hypotheses:*
   - Objective: Propose at least 10 hypotheses, each with a Confidence Score (0.0 to 1.0) and Creative Score (0.0 to 1.0), reflecting originality, surprise, and utility.
   - Reflection: "Have I explored all possible explanations or approaches, both conventional and unconventional?"
   - Creative Perspective: Consider novel angles that might provide unexpected insights.

   *2.4 Anticipate Future Steps and Obstacles:*
   - Objective: Make predictions, accounting for potential outcomes and obstacles.
   - Reflection: "What challenges might I face? Is my plan flexible for different scenarios?"
   - Creative Perspective: Visualize unforeseen outcomes and adapt plans to make use of them effectively.

   *2.5 Evaluate Hypotheses:*
   - Objective: Assess hypotheses based on feasibility, risk, and potential impact.
   - Evaluation: Refine Confidence and Creative Scores as needed.
   - Reflection: "Am I unbiased in my assessment? Which options fit best with the overall objectives?"
   - Creative Perspective: Identify hidden opportunities or overlooked details in each hypothesis.

   *2.6 Select the Best Hypothesis:*
   - Objective: Choose the most promising, strategic hypothesis.
   - Reflection: "Why does this hypothesis stand out? How does it uniquely address the issue?"
   - Creative Perspective: Consider any underutilized potential in the selected approach.

   *2.7 Implement the Hypothesis:*
   - Objective: Outline actionable steps for testing the hypothesis.
   - Reflection: "Is this plan practical? What resources or preparation are required?"
   - Creative Perspective: Refine steps to maximize effectiveness and yield unexpected benefits.

   *2.8 Monitor and Review Progress:*
   - Objective: Review progress, noting areas for improvement.
   - Reflection: "What’s working well? What could be improved?"
   - Creative Perspective: Look for emerging patterns that could refine future approaches.

   *2.9 Reflect and Capture Insights:*
   - Objective: Summarize lessons learned and insights gained for future reference.
   - Reflection: "What new understanding has emerged from this process?"
   - Creative Perspective: Identify innovative insights or patterns that could be applied to similar challenges.

3. *Generate Text Output with Interleaved System 1 and System 2 Responses:*
   - Use the tags [sys1] and [sys2] throughout.
   - Aim for a lengthier, detailed response. Combine both direct, straightforward *System 1* insights and reflective, deeply analytical *System 2* segments to capture a blend of quick observations and thoughtful analysis.

---

### *Example Input Text:*

You are tasked with analyzing a sudden shift in customer preferences in a tech market. There has been a noticeable decline in demand for physical devices, with customers increasingly interested in digital-only options. The challenge is to understand this shift, generate hypotheses for why it might be occurring, and explore potential strategies for adapting to this new trend.

---

### *Processed Output with System 1 and System 2 Thinking:*

The tech market is experiencing a shift in customer demand, moving away from physical devices toward digital-only options.

[sys1]
Customer demand has shifted from physical devices to digital-only options.
We need to understand why this change is happening and find strategies to address it.
Key factors may include cost, convenience, and technological trends.
[end sys1]

[sys2]
**2.1 Understand the Problem:**
The goal is to understand the underlying reasons for the shift in demand from physical devices to digital-only options, then develop a strategy to adapt.

*Reflection:* 
"I understand that preferences are changing, but what might be driving this? Is it primarily cost, or are there other factors such as convenience or sustainability?"

*Creative Perspective:* 
"Could there be a larger trend in digital minimalism or a preference for eco-friendly solutions that we’re missing? What unseen motivations might explain this shift?"

**2.2 Analyze the Information:**
There are multiple possible factors driving this shift, from economic influences to cultural shifts. It’s essential to isolate each factor and understand its impact.

*Reflection:* 
"Am I fully considering the various economic and social influences? Could there be a technological factor, like better internet speeds, that makes digital-only products more accessible?"

*Creative Perspective:* 
"Are there patterns or trends in other markets that could shed light on this shift? Could this be part of a larger trend toward virtual experiences?"

**2.3 Generate Hypotheses:**
1. Customers prefer digital options due to lower costs. (Confidence: 0.8, Creative: 0.4)
2. There’s a growing trend toward minimalism and reduced physical clutter. (Confidence: 0.7, Creative: 0.7)
3. Digital products offer greater flexibility and ease of use. (Confidence: 0.6, Creative: 0.6)
4. Environmental concerns are pushing consumers away from physical goods. (Confidence: 0.6, Creative: 0.8)
5. Advances in tech make digital-only options more functional. (Confidence: 0.8, Creative: 0.5)
6. Pandemic-era remote work increased demand for digital solutions. (Confidence: 0.7, Creative: 0.6)
7. Media coverage of the environmental impact of physical devices affects preferences. (Confidence: 0.5, Creative: 0.7)
8. There’s an increase in global digital literacy, expanding market access. (Confidence: 0.6, Creative: 0.6)
9. Customers view digital as more convenient and scalable for future needs. (Confidence: 0.7, Creative: 0.5)
10. Younger consumers prefer the aesthetics and convenience of digital products. (Confidence: 0.6, Creative: 0.6)

*Reflection:* 
"Have I considered all possible influences? Are there any surprising factors that could explain this shift?"

*Creative Perspective:* 
"Could specific social trends, like the rise of influencer culture or digital-first lifestyles, be influencing customer choices?"

**2.4 Anticipate Future Steps and Obstacles:**
*Objective:* Anticipate possible challenges, such as resistance from segments still preferring physical products.

*Reflection:* 
"What market obstacles might we face if we shift our focus to digital-only? Are there sub-segments that still prioritize physical products?"

*Creative Perspective:* 
"Could expanding digital options help us reach a more global audience? Are there emerging trends that we could leverage in our strategy?"

[end sys2]

[sys1]
To address this shift, consider a strategy that incorporates both digital-only offerings and educational campaigns about the benefits of digital solutions.
Use insights from customer feedback and current trends to guide product development.
Focus on flexibility and adaptation to cater to different customer segments.
[end sys1]


---
abstract: 'In supervised machine learning, the assumption that training data is labelled correctly is not always satisfied. In this paper, we investigate an instance of labelling error for classification tasks in which the dataset is corrupted with out-of-distribution (OOD) instances: data that does not belong to any of the target classes, but is labelled as such. We show that detecting and relabelling certain OOD instances, rather than discarding them, can have a positive effect on learning. The proposed method uses an auxiliary classifier, trained on data that is known to be in-distribution, for detection and relabelling. The amount of data required for this is shown to be small. Experiments are carried out on the FSDnoisy18k audio dataset, where OOD instances are very prevalent. The proposed method is shown to improve the performance of convolutional neural networks by a significant margin. Comparisons with other noise-robust techniques are similarly encouraging.'
address: 'Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK'
bibliography:
- 'references.bib'
title: 'Learning with Out-of-Distribution Data for Audio Classification'
---

Audio classification, out-of-distribution, convolutional neural network, pseudo-labelling

Introduction {#s:intro}
============

Supervised learning refers to the use of labelled training data, the availability of which provides a tremendous advantage in many applications of machine learning. In practice, labels are not always correct [@noise_frenay], prompting additional efforts to carefully verify them. This can be prohibitively costly when scaling to large datasets, which often results in limited data for training. In order to utilise much larger datasets, there has been interest in learning methods that do not rely on clean data [@fsdnoisy18k_fonseca; @noise_sukhbaatar; @noise_reed; @noise_li]. To this end, this paper investigates a case of labelling error for audio classification in which the dataset is corrupted with out-of-distribution (OOD) instances: data that does not belong to any of the target classes, but is labelled as such.

Large amounts of annotated data are available to use when considering the world wide web [@freesound_fonseca; @audioset_gemmeke]. However, due to the uncontrolled/miscellaneous nature of these sources of data, irrelevant (OOD) instances are likely to be encountered when curating the data. For example, Freesound Annotator [@freesound_fonseca] is a platform of datasets comprised of over audio samples annotated by the public, where the authors of this platform have observed a considerable number of OOD instances [@fsdnoisy18k_fonseca]. OOD corruption can occur for a number of reasons, such as uncertainty in the sound (e.g. being unable to discriminate between clarinet sounds and flute sounds) and uncertainty in the label semantics (e.g. ‘keyboard’ could refer to keyboard instruments or it could refer to computer keyboards).

In this paper, it is argued that certain OOD instances, when labelled appropriately, can be beneficial for learning, and that this depends on their likeness to the in-distribution (ID) data. Using a continuous label space, one can even assign ‘soft’ labels to these instances to reflect uncertainty in what the most appropriate target class is. By considering the new labels as the correct labels, OOD corruption can be framed in terms of label noise [@noise_frenay]; for each instance, a (pseudo-)correct label exists, but the label assigned by the annotator may be incorrect.

Considering the problem in terms of label noise allows the incorporation of methods developed for label noise. In particular, this paper proposes a pseudo-labelling method for the OOD training data. There are two main stages: (1) OOD detection and (2) relabelling. To detect and relabel the relevant instances, an auxiliary classifier trained on a much smaller dataset of manually-verified ID examples is used. The original ground truth of the training data is also exploited. Requiring a small amount of verified data is not unreasonable, as the cost of doing so is relatively low. Convolutional neural networks are used as baselines to assess the proposed method and compare it to alternatives methods. Experiments are carried out on the FSDnoisy18k dataset [@fsdnoisy18k_fonseca], which is a large audio dataset with a substantial number of OOD training examples.

Related Work {#s:i:related}
------------

The OOD detection methods most relevant to our work are those that use a discriminative neural network model to directly detect OOD data. Hendryks and Gimpel [@ood_hendryks] proposed using the maximum of the softmax output as a measure of confidence, where a low value indicates that an instance is OOD. Using this idea as a baseline, several improvements have been proposed, including different confidence measures [@ood_liang] and changes to the learning dynamics during training [@ood_devries]. Our paper adapts ODIN [@ood_liang], which applies input pre-processing and logit scaling to influence the softmax output. There are two key differences, however. First, the method is modified to detect OOD instances that are *similar* to the ID data. Second, our paper is concerned with OOD *training* data rather than *test* data. As a result, we also exploit the ground truth labels.

Pseudo-labelling is a method that has been proposed in the past for the noisy label problem, with a number of works using neural network models. Reed et al. [@noise_reed] proposed a method called *bootstrapping*, which adaptively relabels the training data as a combination of the observed label and the current classifier prediction. Li et al. [@noise_li] proposed a similar method, but instead of using predictions during training, they used the prediction of an auxiliary classifier trained on a verified dataset. This is the approach that is adopted in this paper. However, our proposed method also includes a detection stage to only relabel a subset of the training examples.

Background {#s:background}
==========

A single-label classifier is any function $f: {\mathcal{X}}\to {\mathcal{Y}}$ that maps an instance, $x \in {\mathcal{X}}$, to a label, $y \in {\mathcal{Y}}$, where ${\mathcal{Y}}\coloneqq
\{1,...,K\}$ and $K$ is the number of classes. The instance $x$ is said to belong to the class corresponding to $y$. A supervised learning algorithm is said to train a classifier using *examples* (training data) of the form $(x, y) \in {\mathcal{X}}\times {\mathcal{Y}}$ in order to be able to classify future instances (test data). In a standard learning problem, the training set and test set are assumed to be sampled from the same distribution $D$ over ${\mathcal{X}}\times {\mathcal{Y}}$ [@noise_van_rooyen]. This is no longer the case when examples are labelled incorrectly. In the problem that we are studying, the training set contains instances for which the marginal probability density function, $p(x)$, of $D$ vanishes. Such instances are known as out-of-distribution instances.

The distribution $D$ can determine whether an instance is OOD, but it cannot describe the OOD data itself nor its relation to the ID data. This is limiting, as OOD instances can possess properties that overlap with the ID data. For example, clarinets and flutes sound similar regardless of whether one of them is OOD. In this sense, OOD instances can be ‘near’ or ‘far’ from the ID data. We argue that this information, which is a manifestation of data uncertainty [@uncertainty_malinin], can be considered as further knowledge for a learning algorithm to benefit from. Similar to knowledge distillation [@distillation_hinton], we use pseudo-labels to convey this knowledge. The pseudo-labels are generally soft labels such that $y \in \{z \in
\mathbb{R}_+^K: \norm{z}_1 = 1\}$. This conveys the uncertainty in assigning an OOD instance to any one class.

From another perspective, assigning soft labels to OOD instances has already been shown to improve training. Mixup [@mixup_zhang] is a data augmentation technique that linearly combines instances and their labels, such that ${\hat{x}}=
\alpha x_1 + (1-\alpha) x_2$ and ${\hat{y}}= \alpha y_1 + (1-\alpha) y_2$, where $y_1$ and $y_2$ are understood to be one-hot vectors. When $y_1 \ne y_2$, ${\hat{y}}$ does not correspond to any one class, meaning that ${\hat{x}}$ is OOD [@mixup_guo]. Despite using OOD instances as additional training data, mixup has been shown to be an effective form of data augmentation [@mixup_zhang; @mixup_guo; @audio_iqbal]. This is further motivation for our method. In our problem, however, the pseudo-correct label ${\hat{y}}$ is *not* known and not just the result of a linear combination. To estimate the pseudo-label, we use two sources of information: the ground truth label and the prediction of an auxiliary classifier.

Proposed Method {#s:method}
===============

To detect and relabel OOD instances, another training set, ${\mathsf{D}_C}\coloneqq
({\mathsf{X}}_C, {\mathsf{Y}}_C)$, consisting of only clean (verified) examples is used. That is, ${\mathsf{D}_C}$ only contains ID instances that are labelled correctly. In contrast, let ${\mathsf{D}_N}\coloneqq ({\mathsf{X}}_N, {\mathsf{Y}}_N)$ denote the dataset consisting of noisy (unverified) examples. The general steps of the proposed method are as follows (cf. Figure \[f:decision\]):

1.  Train an auxiliary classifier, $f_A$, using ${\mathsf{D}_C}$.

2.  \[item:detect\] Detect the relevant OOD instances in ${\mathsf{D}_N}$ using $f_A$.

3.  \[item:relabel\] Relabel the detected instances using $f_A$. Let ${\mathsf{D}_R}$ denote the new dataset after relabelling.

4.  Train the primary classifier, $f$, using ${\mathsf{D}_C}\cup {\mathsf{D}_R}$.

In the final step, both verified and unverified examples are used to train the classifier, but with the changes made in step \[item:relabel\]. The two steps that deserve the most attention are steps \[item:detect\] and \[item:relabel\], which are developed in the following subsections.

![An illustration of how OOD instances are detected and relabelled using the auxiliary classifier, $f_A$, and then used to train the primary classifier, $f$. ${\mathsf{D}_C}$, ${\mathsf{D}_N}$, and ${\mathsf{D}_R}$ denote the datasets defined in Section \[s:method\].[]{data-label="f:decision"}](decision){width="0.78\columnwidth"}

Out-of-Distribution Detection {#s:m:detection}
-----------------------------

As outlined in Section \[s:i:related\], there has been previous work on OOD detection based on estimating confidence values for the predictions of the classifier. Instances for which the confidence, $c(x) \in [0, 1]$, of the classifier is lower than a threshold, $\tau \in (0, 1)$, are considered OOD. Although these algorithms have been shown to be effective in various test cases [@ood_hendryks], they have been more successful in detecting instances that are *far* from the ID data, while OOD instances that are *near* are typically not detected [@ood_liang]. This is not appropriate, as our relabelling method relies on the OOD instances to be near.

To rectify this, our proposed detection algorithm exploits the availability of labels in the training data. When there is a *mismatch* between the label and the classifier prediction, i.e. $f_A(x) \ne y$ for a training example $(x, y)$, and the confidence of the classifier is *above* a threshold, i.e. $c(x) > \theta$ for $\theta \in (0, 1)$, the instance $x$ is considered as an OOD instance that should be relabelled. Denoting $S_{\theta}$ as the set containing these OOD instances, we have $$S_{\theta} \coloneqq \{x: c(x) > \theta \text{ and } f_A(x) \ne y\}.$$ Selecting instances for which there is a mismatch indicates that the observed label, $y$, may be erroneous. By additionally ensuring that $c(x)$ is high, it only includes instances that are believed to be similar to one of the target classes.

One could also incorporate the mismatch condition for low-confidence OOD instances by defining $$S_{\tau} \coloneqq \{x: c(x) < \tau \text{ and } f_A(x) \ne y\}.$$ This is intended to detect far-away OOD instances, as with the OOD detection algorithms in previous work [@ood_hendryks; @ood_liang; @ood_devries], but the additional mismatch condition can reduce the number of ID instances that are incorrectly detected, since a mismatch is less likely for ID instances. The instances in $S_{\tau}$ can then be dealt with separately, e.g. by discarding them. This is explored as an additional step in the experiments presented later, but is not the main focus of this paper.

Pseudo-labelling {#s:m:pseudo}
----------------

As explained in Section \[s:background\], OOD instances do not belong to any of the target classes, but they may share properties with one or more of them. As such, we propose to relabel the instances in $S_{\theta}$ as a convex combination of the observed label, $y$, and the prediction, ${\hat{y}}$, given by the auxiliary classifier. $$\label{eq:relabel}
  z = \lambda y + (1-\lambda) {\hat{y}}.$$ In , $y$ is understood to be a one-hot vector, while ${\hat{y}}$ is a vector of probabilities, i.e. ${\hat{y}}\in \{z \in \mathbb{R}_+^K:
\norm{z}_1 = 1\}$, so that $f_A(x) = \operatorname*{arg\,max}{\hat{y}}_i$. The parameter $\lambda
\in (0, 1)$ determines the weight to apply to the observed label and the prediction. ${\hat{y}}$ is used because the auxiliary classifier is confident in the prediction, so it is likely to be an optimal label for learning. On the other hand, there is a chance that it is wrong or suboptimal relative to the observed label. The weight, $\lambda$, can be interpreted as the prior belief that this is the case.

This method of relabelling has also been proposed in the past for noisy labels [@noise_li], where the authors found that it was an effective approach for several noisy datasets. For the selection of the parameter $\lambda$, they used a heuristic that derived the weight as a function of the performance of the auxiliary classifier on clean data and noisy data.

Experiments {#s:experiments}
===========

In this section, experimental results are presented to evaluate the proposed method[^1]. The experiments were carried out using the FSDnoisy18k dataset [@fsdnoisy18k_fonseca], which is a crowdsourced audio classification dataset created as part of Freesound Annotator [@freesound_fonseca]. It contains audio clips across classes, totalling hours of audio. The clip durations range from to . The dataset is divided into a training set containing clips and a test set containing clips. The test set has been manually verified so that all of the audio clips are ID. In contrast, only of the training set is verified: this is the data that is used to train the auxiliary classifier. It has been estimated [@fsdnoisy18k_fonseca] that of the unverified labels are incorrect, and that of the incorrect labels are OOD.

Baseline System {#s:r:baseline}
---------------

The evaluated systems are based on two baseline convolutional neural networks (CNNs) [@vgg_simonyan; @densenet_huang]. We used two baselines to investigate two different settings: using a randomly-initialised model and using a pre-trained model. The two baselines are *VGG9* (randomly initialised) and *DenseNet-201* (pre-trained). VGG9 is based on VGG [@vgg_simonyan; @audio_iqbal] and contains convolutional layers and fully-connected layer. DenseNet-201 is DenseNet with 201 layers [@densenet_huang] and was pre-trained using ImageNet [@imagenet_deng]. Although ImageNet is an image dataset, we found that it was surprisingly effective for pre-training. The architecture of each baseline was chosen independently based on performance. DenseNet-201 performed better than VGG9 when pre-training with ImageNet, while VGG9 performed better than DenseNet-201 when using randomly-initialised weights.

Features were extracted by converting the audio waveforms into logarithmic mel-spectrograms (log-mels) with a window length of , a hop length of , and bands. To ensure the neural networks received fixed-length inputs, we used a block-based approach as used in our previous work [@audio_iqbal]. That is, the feature vectors were partitioned into blocks of length and processed independently.

Models were trained using the categorical cross-entropy loss function with the Adam optimization algorithm [@adam_kingma]. Training was carried out for epochs with a batch size of and a learning rate of , which was decayed by after every two epochs. The primary classifier, $f$, and the auxiliary classifier, $f_A$, were trained identically. However, $f_A$ was specifically trained using the pre-trained DenseNet-201 model, regardless of the model used for $f$.

Unless otherwise stated, all hyperparameter values were selected by evaluating the models with a validation set, which contained manually-verified examples from each class, and was sampled from the training set.

Evaluated Systems {#s:e:systems}
-----------------

Several systems were evaluated to assess the performance of the proposed method. Each system applies to both baselines. We evaluated a number of variations of the proposed method (starred below) as well as alternative methods proposed for noise robustness in general. The systems are as follows:

-   *Clean*: The baseline trained with clean examples only.

-   *Clean-DA*: *Clean* with data augmentation. The DenseNet variant of this system was used to train $f_A$.

-   *Baseline*: The baseline trained with all of the examples.

-   *[OOD-R^\*^]{}*: The method proposed in this paper.

-   *[OOD-RD^\*^]{}*: Equivalent to *OOD-R*, except instances in $S_{\tau}$ (c.f. Section \[s:m:detection\]) are discarded.

-   *[All-R^\*^]{}*: All the examples in ${\mathsf{D}_N}$ are relabelled.

-   *Bootstrap*: Labels are updated dynamically using the bootstrapping method [@noise_reed]. No OOD detection.

-   *${\mathcal{L}_q}$ Loss*: The baseline system with the ${\mathcal{L}_q}$ loss [@lq_zhang] (with $q=0.7$) instead of the cross-entropy loss.

-   *Noise Layer*: An additional linear layer maps the predictions to the same noisy space as the observed labels [@noise_sukhbaatar]. This layer is removed during inference.

The proposed system and its variations were configured with $\tau=0.4$, $\theta=0.55$, and $\lambda=0.5$. The value of $\lambda$ was selected based on the heuristic given in [@noise_li]. The values of $\theta$ and $\tau$ were selected by applying the detection algorithm on the validation set and selecting the threshold for which less than of the instances were (incorrectly) detected. This way, the selection of the thresholds is interpretable and independent of $f_A$. Instead of using the softmax output directly, ODIN [@ood_liang] was used to compute $c(x)$; we found that ODIN detected more instances in ${\mathsf{D}_N}$ for a given number of incorrectly detected validation instances. Using ODIN, (resp. ) of ${\mathsf{D}_N}$ was detected as belonging to $S_{\theta}$ (resp. $S_{\tau}$).

The purpose of *Clean-DA* is to compare data augmentation to using the noisy examples. We experimented with mixup [@mixup_zhang] and SpecAugment (time/frequency masking) [@specaug_park], and adopted the latter as it gave superior performance. The ${\mathcal{L}_q}$ loss is designed to be robust against incorrectly-labelled data, and is the approach taken by the authors of FSDnoisy18k [@fsdnoisy18k_fonseca]. The bootstrapping method is similar to the proposed method, except there is no OOD detection and no auxiliary classifier, as examples are relabelled *during* training as a combination of the ground truth and $f$’s current prediction. The noise layer was proposed for class-conditional label noise [@noise_sukhbaatar], and was utilised by Singh et al. [@fsdnoisy18k_singh] for FSDnoisy18k.

To score the performance of the systems, average precision (AP) and accuracy were used as metrics. Both were computed as micro-averages and reported in percentages, where a higher percentage means higher performance. Five trials were carried out for each experiment to account for uncertainty.

Results {#s:r:results}
-------

[0.48]{}[lXXXX]{} & &\
System & VGG & DenseNet & VGG & DenseNet\
Clean & [$71.8 \scriptstyle{\pm 0.21}$]{} & [$72.3 \scriptstyle{\pm 0.47}$]{} & [$67.5 \scriptstyle{\pm 0.23}$]{} & [$67.5 \scriptstyle{\pm 0.23}$]{}\
Clean-DA & [$74.6 \scriptstyle{\pm 0.39}$]{} & [$75.7 \scriptstyle{\pm 0.34}$]{} & [$66.4 \scriptstyle{\pm 0.45}$]{} & [$69.4 \scriptstyle{\pm 0.24}$]{}\
Baseline & [$81.6 \scriptstyle{\pm 0.55}$]{} & [$84.8 \scriptstyle{\pm 0.25}$]{} & [$74.5 \scriptstyle{\pm 0.71}$]{} & [$78.0 \scriptstyle{\pm 0.36}$]{}\
[OOD-R^\*^]{} & [$\mathbf{86.0 \scriptstyle{\pm 0.15}}$]{} & [$\mathbf{88.0 \scriptstyle{\pm 0.21}}$]{} & [$\mathbf{77.9 \scriptstyle{\pm 0.14}}$]{} & [$\mathbf{81.0 \scriptstyle{\pm 0.50}}$]{}\
[OOD-RD^\*^]{} & [$\mathbf{86.1 \scriptstyle{\pm 0.37}}$]{} & [$87.1 \scriptstyle{\pm 0.36}$]{} & [$\mathbf{78.0 \scriptstyle{\pm 0.56}}$]{} & [$80.3 \scriptstyle{\pm 0.15}$]{}\
[All-R^\*^]{} & [$84.5 \scriptstyle{\pm 0.55}$]{} & [$\mathbf{88.1 \scriptstyle{\pm 0.19}}$]{} & [$77.1 \scriptstyle{\pm 0.68}$]{} & [$\mathbf{81.4 \scriptstyle{\pm 0.27}}$]{}\
Bootstrap & [$81.2 \scriptstyle{\pm 0.36}$]{} & [$84.9 \scriptstyle{\pm 0.60}$]{} & [$74.8 \scriptstyle{\pm 0.80}$]{} & [$78.7 \scriptstyle{\pm 0.46}$]{}\
${\mathcal{L}_q}$ Loss & [$83.3 \scriptstyle{\pm 0.48}$]{} & [$85.0 \scriptstyle{\pm 0.72}$]{} & [$76.4 \scriptstyle{\pm 0.56}$]{} & [$78.6 \scriptstyle{\pm 0.64}$]{}\
Noise Layer & [$83.0 \scriptstyle{\pm 0.47}$]{} & [$84.7 \scriptstyle{\pm 0.82}$]{} & [$76.2 \scriptstyle{\pm 0.34}$]{} & [$78.0 \scriptstyle{\pm 0.75}$]{}\

The results are presented in Table \[t:main\_results\]. When training with the clean examples only, data augmentation resulted in a noticeable improvement in performance. However, this improvement can be seen to be relatively small compared to using all of the examples for training. This shows that there is a benefit to training with the noisy examples in this dataset, despite a large percentage of them being OOD/incorrect. Among the alternative methods, the ${\mathcal{L}_q}$ loss and noise layer both resulted in large improvements, but only for the VGG baseline. Bootstrapping did not improve the performance for either baseline.

The proposed method, *OOD-R*, performed the best, with significant gains seen for both baselines. *OOD-RD*, which also discards examples in $S_{\tau}$, did not perform better than *OOD-R*; discarding the examples actually worsened the performance for the DenseNet model, possibly due to removing ID examples. These results suggest that the neural networks are robust to far-away OOD instances being present in the training set, and that removing them should not be a priority. *All-R*, which relabels all of the examples in ${\mathsf{D}_N}$, did not perform as well as *OOD-R* for the VGG model, which demonstrates the importance of the detection stage. On the other hand, there was no discernible difference between *All-R* and *OOD-R* for the DenseNet model. A possible reason is that the pre-trained DenseNet model is more robust to the label noise introduced when relabelling the low-confidence examples.

Conclusion {#s:conclusion}
==========

In this work, we investigated the problem of learning in the presence of out-of-distribution (OOD) data. We argued that OOD data can possess properties that are characteristic of the target classes, so that appropriately relabelling the instances to reflect this, rather than discarding them, can benefit training. Our proposed method involved training an auxiliary classifier on a smaller verified dataset, and using its predictions, along with the ground truth labels, to detect and relabel the relevant OOD instances. Using convolutional neural network baselines, experiments with the FSDnoisy18k dataset showed that our method substantially improves performance. The results also suggested that OOD instances that are very different from the target classes have little effect on performance when present in the training data. Future work includes investigating other detection and pseudo-labelling methods, including those that do not require any verified data.

[^1]: Source code: <https://github.com/tqbl/ood_audio>