Title: Resilient Multiple Choice Learning: A learned scoring scheme with application to audio scene analysis
Venue: NeurIPS 2023
Accepted: yes
Keywords: Multiple Choice Learning

Audio processing.
Abstract: We introduce Resilient Multiple Choice Learning (rMCL), an extension of the MCL approach for conditional distribution estimation in regression settings where multiple targets may be sampled for each training input.
Multiple Choice Learning is a simple framework to tackle multimodal density estimation, using the Winner-Takes-All (WTA) loss for a set of hypotheses. In regression settings, the existing MCL variants focus on merging the hypotheses, thereby eventually sacrificing the diversity of the predictions. In contrast, our method relies on a novel learned scoring scheme underpinned by a mathematical framework based on Voronoi tessellations of the output space, from which we can derive a probabilistic interpretation.
After empirically validating rMCL with experiments on synthetic data, we further assess its merits on the sound source localization problem, demonstrating its practical usefulness and the relevance of its interpretation.

Resilient Multiple Choice Learning: A learned scoring scheme with application to audio scene analysis

Abstract

We introduce Resilient Multiple Choice Learning (rMCL), an extension of the MCL approach for conditional distribution estimation in regression settings where multiple targets may be sampled for each training input. Multiple Choice Learning is a simple framework to tackle multimodal density estimation, using the WinnerTakes-All (WTA) loss for a set of hypotheses. In regression settings, the existing MCL variants focus on merging the hypotheses, thereby eventually sacrificing the diversity of the predictions. In contrast, our method relies on a novel learned scoring scheme underpinned by a mathematical framework based on Voronoi tessellations of the output space, from which we can derive a probabilistic interpretation. After empirically validating rMCL with experiments on synthetic data, we further assess its merits on the sound source localization task, demonstrating its practical usefulness and the relevance of its interpretation.

1 Introduction

Machine learning models are commonly trained to produce, for any given input, a single prediction. In most cases, for instance, when minimizing the empirical risk with quadratic loss, this prediction can be interpreted as the conditional output expectation given the input. However, there are many tasks for which the conditional output distribution can be multimodal, either by the nature of the task or due to various sources of uncertainty. Temporal tracking and forecasting, for instance, are problems of this type [5, 32, 44]. In such cases, the mean might fall in a low-density region of the conditional probability function, and it would be beneficial to predict multiple hypotheses instead [34, 20]. In this context, Multiple Choice Learning (MCL) has emerged as a simple pragmatic solution [17, 29]. Thanks to a network architecture with multiple heads, MCL models can produce multiple hypotheses, one per head. During supervised training, the gradients are only computed for the head that provides the best prediction given the current input sample. This Winner-Takes-All (WTA) training scheme allows each head to specialize in a region of the output space. Accordingly, Rupprecht et al. [34] have proposed a probabilistic interpretation of MCL based on Voronoi tessellations.

On that account, MCL suffers from two significant issues: hypotheses collapse and overconfidence . Hypotheses collapse occurs during training when a head takes the lead, the others being almost never selected under WTA and thus not updated. This leads to a situation where most heads are not trained effectively and produce meaningless outputs during inference. Overconfidence can also be observed at inference time when looking at MCL under the lens of its probabilistic interpretation. Hypotheses that correspond to rare events tend to be over-represented, thus not reflecting the true distribution of

outputs well. Multiple variants of MCL have been developed to alleviate these issues, in particular for classification or segmentation tasks [29, 27, 42, 13].

In this paper, we are interested in MCL in the context of regression settings, which has been mostly overlooked. More specifically, we propose rMCL, for resilient Multiple Choice Learning , a MCL variant with a learned scoring scheme to mitigate overconfidence. Unlike prior work, rMCL is designed to handle more general settings where multiple targets may be sampled for each input during training. Moreover, we propose a probabilistic interpretation of rMCL, and show that rMCL's scoring fixes overconfidence by casting multi-choice regression as conditional distribution estimation. While introduced in the context of the original WTA loss, the proposed scoring approach is also compatible with other variants of WTA. We validate our claims with experiments on synthetic data, and we further evaluate the performance of rMCL on the sound source localization problem. The accompanying code with a rMCL implementation is made available. 1

2 Related Work

Uncertainty estimation. The estimation of machine learning model's uncertainty can be studied from different perspectives depending on the type of ambiguity being considered. The uncertainty can concern either the model ( epistemic ) or the data ( aleatoric ) [7, 12]. Kendall and Gal [22] showed that those types of uncertainty can be combined in deep learning settings, placing both a distribution on the weights and the outputs of the model. Epistemic uncertainty estimation may, for instance, be achieved by variational approximation of the weights posterior [31, 14]. Independent ensembles (IE) of neural networks [18, 11] is another widespread technique for epistemic uncertainty approximation. Several neural networks, trained from different random initializations, provide samples for approximating the weights posterior distribution [26].

Multiple Choice learning. Originally introduced by Guzman-Rivera et al. [17] and adapted to deep learning settings by Lee et al. [28, 29], MCL is a framework in which multiple output heads propose different possible predictions. Since many datasets contain only a single output realization for each input, the heads are trained with a Winner-Takes-All scheme where only the head that made the best prediction is updated. It can be understood as a specific case of dependent ensembles [8, 46, 4, 47] where the members interact with each other during training. Alternatively, [34] have shown that MCL can be adapted successfully for multi-label classification [23].

The Winner-Takes-All training scheme, however, can cause two main issues known as hypotheses collapse and overconfidence [34, 6, 10, 19, 10, 27, 42]. Most related previous works tackle either or both of those questions. Hypotheses collapse occurs during training when some heads receive little or no updates due to bad initial values under the WTA loss and other hypotheses taking the lead. To fix this issue, Rupprecht et al. [34] proposed a relaxed Winner-Takes-All (RWTA, or ε -WTA) loss that allows the update of non-winner heads, albeit with gradients scaled by a small constant ε . While this approach ensures that every head receives gradients updates, it also tends to level the different heads, which counters the initial goal of WTA and thus needs careful tuning. Finally, one can leverage the evolving Winner-Takes-All loss [32], with the topn heads getting updated (topn -WTA) instead of only the best one. The authors validate that their approach, with a scheduled decreasing value for n , achieves improved conditioning of the outputs by reducing inconsistent hypotheses in the context of future frames prediction.

On the other hand, overconfidence is an issue that can be observed in inference when evaluating a model as a density estimator. In this case, one would want the different outputs to be distributed across the different modes of the true conditional probability of the outputs. Empirical observations [27, 42] show that this is not usually the case, and rare events tend to be overly represented, rendering such a model inadequate for integration within real-world decision-making systems. In this context, Lee et al. [27] proposed Confident Multiple Choice Learning (CMCL) for solving overconfidence in a classification setup. Additionally to the WTA loss, CMCL is trained by maximizing the entropy of the class distributions predicted by the non-selected hypotheses. The classifier's final prediction is based on a simple average of the discrete class distributions predicted by each of the heads. Although designed for tackling the overconfidence problem, CMCL reduces the diversity of the hypotheses. On that account, Versatile Multiple Choice Learning [42] (vMCL) proposed to address the issue by leveraging a choice network aiming at predicting a score for each hypothesis head. This may be seen

as a variant of Mixture-of-Experts approaches [33], in which the choice network is supervised with explicit targets. The final prediction in vMCL is derived by weighting the class distributions from each hypothesis by their respective scores. These works focus on classification tasks and aggregate the hypotheses, thereby losing individual information from each head. In contrast, we address regression tasks and demonstrate how to benefit from the diversity of predictions without aggregation.

We propose to revisit the vMCL approach for regression, single-target or multi-target, and to extend accordingly the mathematical interpretation of the WTA proposed by Rupprecht et al. [34].

3 rMCL regression and its probabilistic interpretation

After reviewing MCL, this section introduces resilient Multiple Choice Learning, dubbed rMCL. This proposed variant handles multi-target settings thanks to its scoring scheme. We then provide a distribution learning interpretation of rMCL that is relevant to regression tasks, thus completing the MCL probabilistic interpretations of prior works [9, 34].

3.1 Fixing the overconfidence issue in Multiple Choice Learning

Let X ⊆ R d and Y ⊆ R q be the input and output spaces in a supervised learning setting, such that the training set D is composed of samples ( x s , y s ) from an underlying joint distribution p ( x , y ) over X × Y .

Multiple choice learning was proposed to address tasks with ambiguous outputs, i.e. , for which the ground-truth distribution p ( y | x ) is multimodal [17, 29, 10]. Adapted by Lee et al. [29] for deep-learning settings in Stochastic Multiple Choice Learning (sMCL), it leverages several models f θ /defines ( f 1 θ , . . . , f K θ ) ∈ F ( X , Y K ) , referred to as K hypotheses , trained using the Winner-takes-all (or Oracle) loss. It consists, given an underlying loss function /lscript and for each sample ( x s , y s ) in the current batch, of first the computation of

after each forward pass, followed by backpropagation on the winner hypothesis, that is, the minimizing one. Assuming now that a set Y s of targets is available in the training set for input x s , Firman et al. [10] have shown that (1) can be generalized by updating the best hypothesis per target using the meta-loss

where

/negationslash

This loss can, however, lead to poor conditioning of the output set of predictions ( hypotheses collapse ), with only one or a few hypotheses being exclusively selected for backpropagation. Furthermore, sMCL is subject to the overconfidence problem. Following Theorem 1 of [34], a necessary condition for minimizing the risk

is that each prediction f k θ ( x ) coincides with the centroid of Y k ( x ) with respect to p ( y | x ) . We say that the components {Y k ( x ) } k form a centroidal Voronoi tessellation [9]. If /lscript is for instance the /lscript 2 -loss, this condition means that for each non-zero probability cell k , f k θ ( x ) amounts to the conditional mean of p ( y | x ) within the Voronoi cell Y k ( x ) . However, this theorem tells us nothing about the predictions in very low probability zones; in such regions, the inference-time predictions f k θ ( x ) will be meaningless. During inference, the sMCL indeed faces a limitation: it cannot solely rely on the predicted hypotheses to identify Voronoi cells in the output space with low probability mass (see Figure 1). This observation leads us to propose hypothesis-scoring heads γ 1 θ , . . . , γ K θ ∈ F ( X , [0 , 1]) . Those aim to predict, for unseen input x , the probability P ( Y x ∈ Y k ( x )) where Y x ∼ p ( y | x ) with the aim of mitigating this overconfidence issue.

While MCL variants have often been proposed for classification tasks, we are, to the best of our knowledge, the first to propose to solve this issue for multi-target regression, interpreting the problem as a multimodal conditional distribution estimation.

3.2 Resilient Multiple Choice Learning

We consider hereafter a problem of estimation of a multimodal conditional distribution denoted p ( y | x ) for each input x ∈ X . In a real-world setting, only one or several samples (the targets ) drawn from p ( y | x ) are usually accessible. Sound source localization is a concrete instance of such multimodal prediction for whose p ( y | x s ) represents the sound source position for an input audio clip x s ∈ X at a given time t .

In this last example, each target sample in Y s may represent the location of a mode of the ground-truth multimodal distribution. For such multi-output regression tasks, the MCL training of a randomly initialized multi-hypotheses model with scoring functions, ( f 1 θ , . . . , f K θ , γ 1 θ , . . . , γ K θ ) can be adapted as follows.

For each training sample ( x s , Y s ) , let

and K -( x s ) /defines [ [1 , K ] ] -K + ( x s ) be the set of positive (or winner ) and negative hypotheses respectively. It is then possible to combine the multi-target WTA loss L in (2) with a hypothesis scoring loss

in a compound loss L + β L scoring. This novel approach differs from previous MCL variants [42] in its ability to predict multimodal distributions in regression settings and by the introduction of separated scoring branches that are updated based on loss (6), such that the target for the scoring branch k is the probability that hypothesis k is among the winners for that sample.

A resource-efficient implementation of rMCL is achieved by deriving the hypotheses and score heads from a shared representation, with distinct parameters at the final stages of the architecture ( e.g. , through independent fully connected layers). Designed as such, it is also possible to reduce memory cost by updating only a fraction of score heads associated with the negative hypotheses at each training step, typically with distinct samples k -∼ U ( K -( x s )) . This trick could also alleviate the imbalanced binary classification task that the scoring heads face with a large number of hypotheses |K -( x s ) | /greatermuch |K + ( x s ) | (typically, the number of target samples at disposal is small relative to the number of hypotheses K ). The variants of the WTA ( e.g. , topn -WTA, ε -WTA, see Sec. 2) are also compatible with rMCL. Inference with the proposed rMCL model is outlined in Algorithm 1.

As highlighted above, the hypothesis output can be interpreted, in the context of risk minimization, as the conditional mean of the Voronoi cell Y k ( x ) it defines, providing that the cell has non-zero probability. Furthermore, given (6), the output of the score head γ k θ ( x ) can be interpreted as an approximation of the probability of a sample from p ( y | x ) to belong to this cell.

Algorithm 1 Inference in the rMCL model

Input: Unlabelled input x ∈ X . Trained hypotheses and score heads ( f 1 θ , . . . , f K θ , γ 1 θ , . . . , γ K θ ) ∈ F ( X , Y ) K × F ( X , [0 , 1]) K .

1: Perform a forward pass by computing f 1 θ ( x ) , . . . , f K θ ( x ) , γ 1 θ ( x ) , . . . , γ K θ ( x ) .

Output: Prediction of the output conditional distribution p ( y | x ) .

2: Construct the associated Voronoi components Y k ( x ) (3) with Y = ∪ K k =1 Y k ( x ) .

4: If Y x ∼ p ( y | x ) , then for k such that γ k θ ( x ) > 0 , interpret the predictions as estimations of

3: Normalize the predicted scores γ k θ ( x ) ← γ k θ ( x ) / ∑ K k =1 γ k θ ( x ) .

3.3 Probabilistic interpretation at inference time

Let us consider a trained rMCL model, such as the one described in the previous section. Following the theoretical interpretation of [34] and the motivation of solving the overconfidence issue, as explained in Section 3.1, the output objectives of the model can be summarized in (7). Although it is possible, assuming that those properties are verified, to deduce the following law of total expectation

the above quantity may not be informative enough, especially if the law of Y x ∼ p ( y | x ) is multimodal. Indeed, to be able to derive a complete probabilistic interpretation, we still need to characterize how rMCL approximates the full conditional density of Y x . For that purpose, it is sufficient to fix a law π k for Y x within each Voronoi cell k to accurately represent the predicted distribution in each region of the output space as per the rMCL model, as shown in Proposition 1.

Proposition 1 (Probabilistic interpretation of rMCL) . With the above notations, let us consider a multi-hypothesis model with properties of (7) . Let us furthermore assume that

i.e., the law Y k x of Y x conditioned by the event { Y x ∈ Y k ( x ) } is described by π k for each input x ∈ X with constraint E [ Y k x ] = f k θ ( x ) . Then, for each measurable set A ⊆ Y ,

Proof . We have

Following the principle of maximum entropy [38], the uniform distribution inside each cell Y k x ∼ U ( Y k ( x )) is the least-informative distribution, assuming that the output space Y is with finite volume and the hypotheses lie in the geometric center of their associated cell. With this prior, the predicted distribution can be interpreted as

where V ( Y k ( x )) /defines ∫ y ∈ Y k ( x ) d y is the volume of Y k ( x ) . Similarly, if we model the output distribution as a mixture of Dirac deltas such that in each cell k , γ k θ ( x ) > 0 ⇒ Y k x ∼ δ f k θ ( x ) , then

3.4 Toy example

We build upon the toy example formalized in [34, §4.1] to validate the proposed algorithm and its interpretation. In particular, we seek to illustrate how rMCL handles the overconfidence problem in a regression task. The toy problem involves the prediction of a 2D distribution p ( y | x ) based on scalar input x ∈ X , where X = [0 , 1] . The training dataset is constructed by selecting for each random input x ∼ U ([0 , 1]) one of the four sections S 1 = [ -1 , 0) × [ -1 , 0) , S 2 = [ -1 , 0) × [0 , 1] , S 3 = [0 , 1] × [ -1 , 0) , S 4 = [0 , 1] × [0 , 1] , with probabilities p ( S 1 ) = p ( S 4 ) = 1 -x 2 , p ( S 2 ) = p ( S 3 ) = x 2 . Whenever a region is selected, a point is then sampled uniformly in this region. To slightly increase the complexity of the dataset, it has been chosen to enable the sampling of two (instead of one) target training samples with probability q ( x ) for each input x . Here, q is a piece-wise affine function with q (0) = 1 , q ( 1 2 ) = 0 , and q (1) = 1 .

Wecompared the behavior of our proposed approach rMCL against sMCL and independent ensembles (IE) using a 3-layer perceptron backbone with 256 hidden units and 20 output hypotheses, and /lscript 2 as the underlying distance. For rMCL, we used scoring weight β = 1 . In addition to the full-fledged rMCL, we assess a variant denoted 'rMCL ∗ ' where a single negative hypothesis is uniformly selected during training. For independent ensembles, we trained with independent random initialization 20 single-hypothesis models, the predictions of which being combined to be comparable to multihypothesis models. The training processes were executed until the convergence of the training loss. We report the checkpoints for which the validation loss was the lowest. All networks were optimized using Adam [24]. We show in Figure 1 (top right) test predictions of the classical sMCL compared to those of the rMCL model. In visualizing the predictions where the ground-truth distribution is known, we observe empirically that the centroidal tessellation property, E [ Y k x ] = f k θ ( x ) , is verified with good approximation (see red points Figure 1, bottom right). We otherwise notice the overconfidence problem of the sMCL model in low-density zones when the output distribution is multimodal by looking at the output prediction, e.g. , for x = 0 . 1 and x = 0 . 9 in Figure 1 where the sMCL predictions fail in low-density zones. In contrast, rMCL solves this issue, assigning to each Voronoi cell a score that approximates the probability mass of the ground-truth distribution in this zone (see Figure 1, bottom). Furthermore, the independent ensembles (triangles) suffer from a collapse issue around the conditional mean of the ground-truth distribution. This behavior is expected as the minimizer of the continuous formulation of the risk (4) on the whole output space is the conditional expectation of the output given the input.

Predictions

EMD vS.C

900

0.24

0.22

O

rMCL*

0.20

●rMCL

C=

0.1

=0.6

=0.9

0.04

IE

。

Voronoi tessellation

0.18

C

0.16

EMD

其

200

→0.0

0.2

0.4

0.6

0.8

1.0

From a quantitative perspective, as demonstrated in Fig. 1, we can evaluate the effectiveness of rMCL in addressing the overconfidence issue compared to the classical sMCL. To ensure a fair comparison with sMCL settings, we have opted for using the Earth Mover's Distance (EMD) metric, which measures the discrepancy between the ground truth and predicted distributions. Predicted distributions are considered mixtures of Dirac deltas as in (12) for the metric computation, with sMCL predictions assigning uniform mode weights. As depicted in Fig. 1, rMCL outperforms sMCL when the target distribution is multimodal, i.e. , for extreme values of x . However, in the unimodal scenario, the performances of rMCL and sMCL are fairly similar. Furthermore, the 'rMCL ∗ ' variant (depicted by crosses) that is lighter regarding memory consumption exhibits performance comparable to the standard rMCL. Additional assessments, which are detailed in Appendix B.2, have demonstrated the robustness of the rMCL when corrupting the training dataset with outliers in the output space.

4 Experiments with audio data

4.1 Sound source localization as a conditional distribution estimation problem

Sound source localization (SSL) is the task of predicting the successive angular positions of audio sources from a multichannel input audio clip x ∈ X [15]. This problem can be cast either as a regression problem (estimating the continuous positions of the sources, e.g. , [37]) or a classification one (segmenting the output space into zones in which we wish to detect a source, e.g. , [2]). While not requiring the number of sources to be known in advance, the classification approach suffers from the lack of balanced data and limited spatial resolution. On the other hand, the regression approach enables off-grid localization but typically suffers from the source permutation problem [40]. Assuming that the maximum number of overlapping sources M is known, a solution for handling this multi-target regression problem is to predict, at each time frame t depending on the chosen output resolution, a vector accounting for the sources' activity a t ∈ { 0 , 1 } M , as well as azimuth and elevation angles φ t ∈ R M and ϑ t ∈ R M . A model can be trained with a permutation invariant training (PIT) approach [1, 37, 45] using an optimization criterion of the form

where /lscript CE and /lscript g correspond respectively to a cross-entropy term and a geometrical loss, the latter being computed only for active sources indexes. S M is the set of permutations of M elements and the notation σ ( z ) stands for the M -dim vector z with its components permuted according to σ . In the following, we will denote Y t the set of source positions at time t .

With the distribution learning mindset, this task can be seen as an attempt to estimate, at each time step, the sound source position distribution p ( y | x ) , which can be viewed as a Dirac mixture, p ( y | x ) ∝ ∑ y i ∈ Y t δ y i ( y ) , if we suppose that the targets are point-wise, with another Dirac mixture representing the predicted active modes at predicted positions. Therefore, a natural way to evaluate such SSL regression models is to solve the linear assignment problem, e.g. , using Hungarian matching with spherical distance as an underlying metric. To handle more general distributions, we propose to generalize the metric used to the Earth Mover's Distance (see Sec. 4.2).

The rMCL framework is well suited to SSL as it allows one to benefit from both the advantages of the regression and classification viewpoints in the same spirit as [41]. There is no need for prior knowledge of the number of sources, and it avoids challenges related to imbalanced spatial positions and the source permutation problem of (13). This method comes at a low computational cost regarding added parameters when opting for a low-level representation shared by the hypotheses and scores heads (see Sec. 3.2). Furthermore, it allows for producing a heat map of the sound sources' positions with a probabilistic prediction, which could otherwise account for their spatial dispersion depending on the chosen law π k selected in the Voronoi cells. Modeling the sound sources as point sources, a delta law will be selected following the formulation of Proposition 1.

4.2 Experimental setup

Datasets. We conducted our experiments on increasingly complex SSL datasets originally introduced by Adavanne et al. [1]: i) ANSYN, derived from DCASE 2016 Sound Event Detection challenge with spatially localized events in anechoic conditions and ii) RESYN, a variant of ANSYN in reverberant conditions. Each dataset is divided into three sub-datasets depending on the maximum number of overlapping events (1, 2, or 3, denoted as D1, D2, and D3). These sub-datasets are further divided into three splits (including one for testing). Each training split contains 300 recordings (30 s), of which 60 are reserved for validation. Preprocessing is detailed in Appendix A.2. For each experiment, we used all training splits from D1, D2, and D3. Evaluation was then conducted for each test set based on the overlapping levels. Moreover, we present results from additional SSL datasets, including REAL [1] and DCASE19 [3], in Appendix B.3.

Metrics. To assess the performance of the different methods, we employed the following metrics:

The 'Oracle' error ( ↓ ): O ( x n , Y n ) = 1 | Y n | ∑ y m ∈ Y n min k ∈ [ [1 ,K ] ] d ( f k θ ( x n ) , y m ) .

The Earth Mover's Distance (EMD ↓ ): also known as the 1st Wasserstein distance, it is a distance measure between two probability distributions. In this context, these

are the predicted distribution, ˆ p ( y | x n ) = ∑ K k =1 γ k θ ( x n ) δ f k θ ( x n ) ( y ) , and the groundtruth one, p ( y | x n ) = 1 | Y n | ∑ y m ∈ Y n δ y m ( y ) . The EMD solves the optimization problem W 1 ( p ( ·| x n ) , ˆ p ( ·| x n )) = min ψ ∈ Ψ ∑ K k =1 ∑ y m ∈ Y n ψ k,m d ( f k θ ( x n ) , y m ) , where ψ = ( ψ ( f k θ ( x n ) , y m ) ) k,m is a transport plan and Ψ is the set of all valid transport plans [21].

While the oracle error assesses the quality of the best hypothesis predicted for each target, the EMD metric looks at all hypotheses. It provides insight into the overall consistency of the predicted distribution. The EMD also penalizes the overconfidence issue of the WTA with sMCL as described in Sec. 3.4. In order to fit the directional nature of the sound source localization task, these metrics are computed in a spherical space equipped with distance d (ˆ y , y ) = arccos[sin( ˆ ϑ ) sin( ϑ ) + cos( ˆ ϑ ) cos( ϑ ) cos( φ -ˆ φ )] , where ( ˆ φ , ˆ ϑ ) and ( φ , ϑ ) are source positions ˆ y and y , respectively, expressed as azimuth and elevation angles.

Neural network backbone. We employed the CRNN architecture of SeldNet [1] as a backbone, which we modified to enable multi-hypothesis predictions. This adaptation involves adjusting the output format and duplicating the last fully-connected layers for hypothesis and scoring heads. As only the prediction heads are adjusted, the number of parameters added to the architecture is negligible. Refer to Appendix A.2 for additional architecture and training details.

Baselines. We compared the proposed rMCL with several baselines, each utilizing the same featureextractor backbone as previously described. The baselines include a Permutation Invariant Training variant ('PIT variant') proposed in [37] for SSL, the conventional WTA setup ('WTA, 5 hyp.') and its single hypothesis variant ('WTA, 1 hyp.'), its ε -relaxed version with ε = 0 . 5 (' ε -WTA, 5 hyp.') and its topn variant with n = 3 ('topn -WTA, 5 hyp.'), as well as independent ensembles ('IE'). To ensure a fair comparison, we used the same number of 5 hypotheses for the WTAs, sufficient to cover the maximum of 3 sound sources in the dataset (refer to Sec. 4.5 and the sensitivity study in Table 4). For the single hypothesis WTA, the head is updated by only considering the best target, as this fares better than using one update per target. IE was constructed from five such single hypothesis WTA models trained independently with random initialization.

4.3 Comparative evaluation of rMCL in SSL

We present in Tables 1 and 2 the results of rMCL and baselines in the anechoic (ANSYN) and reverberant (RESYN) conditions, respectively. The consistency of the results should be noted. For each model, the metrics tend to improve as the number of sources decreases and, under similar conditions, results are marginally lower for more challenging datasets. Unsurprisingly, the single hypothesis approach exhibits strong performances in the unimodal cases (D1), but performs significantly worse in the multimodal cases (D2 and D3). Ensembling (IE) improves its performance but still displays the lack of diversity already exposed in Section 3.4. As for the 5-hypothesis WTA, in its original form, it is ill-suited for multi-target regression due to the overconfidence issue: despite a strong oracle metric, the EMD results are very poor. On the other hand, rMCL surpasses its competitors and shows the best EMD in every multimodal setting. It also consistently obtains the second-best oracle error, only slightly above WTA, while not suffering from overconfidence.

We also present results for REAL and DCASE19 in Appendix B.3, that display similar trends. We did not observe the collapse issue (see Section 3.1) in our settings, neither with WTA nor with the proposed rMCL model. We believe it is solved in practice by the variability of the data samples in the stochastic optimization during training [19]; refer to Appendix B.1 for further discussions.

4.4 Combining rMCL with other WTA variants

As noted in Section 3.2, rMCL's improvements are, in theory, orthogonal to that of other WTA variants. In this section, we combine the rMCL with topn -WTA and ε -WTA approaches into respectively topn -rMCL and ε -rMCL. The results are shown in Tables 3. We note that while topn rMCL does not provide significant improvement to rMCL, ε -rMCL, on the other hand improves EMD scores at the expense of increased oracle error. We observe similar effects when this method is applied to WTA, indicating that there are indeed some additive effects of ε -WTA and rMCL. Also note that with regards to Tables 1 and 2, all the proposed variants still get better EMD than all competitors under multimodal conditions, showing the robustness of our method.

4.5 Effect of the number of hypotheses

The impact of varying the number of hypotheses on the performance of the rMCL model is presented in Table 4 and Figure B.4. First and foremost, we notice the anticipated trend of the oracle metric improving as the number of hypotheses increases. Concerning the EMD metric, the single hypothesis model, which avoids errors from negative hypotheses, is most effective in handling unimodal

distributions. In the case of the EMD metric applied to multimodal distributions, we observe that multiple hypothesis models improve the results, but excessively increasing the number of hypotheses may marginally degrade performances. A further study could examine this phenomenon in detail, which may be related to the expressive power of the hypothesis heads or the accumulation of errors in score heads predictions.

5 Discussion and limitations

In the audio experiments, the performance of rMCL was found to be affected by the number of hypotheses to tune depending on the complexity of the dataset. Moreover, as with the other variants of WTA, fixing the overconfidence issue with rMCL slightly degrades the performance of the best hypothesis (oracle error) for a reason that is yet to be determined. Otherwise, while ε -WTA behaves as expected with rMCL, trading off the quality of the best hypotheses for overall performance, topn -WTA does not exhibit the same behavior. This discrepancy warrants further investigation.

The probabilistic interpretation of rMCL, as presented in Algorithm 1 and in Sec. 3.3, states that the different hypotheses would be organized in an optimal partition of the output space, forming a Voronoi tessellation. Ideally, each hypothesis would capture a region of the distribution, and the scores representing how likely this zone would activate in a given context. This interpretation remains theoretically valid whenever train and test examples are sampled from the same joint distribution p ( x , y ) . However, in realistic settings, the hypotheses might not adhere to meaningful regions, but this could be controlled and evaluated, provided that an external validation dataset is available. In future work, we intend to use calibration techniques [16, 39] to identify and alleviate these issues.

To sum up, this paper proposes a new Multiple Choice Learning variant, suitable for tackling the MCL overconfidence problem in regression settings. Our method is based on a learned scoring scheme that handles situations where a set of targets is available for each input. Furthermore, we propose a probabilistic interpretation of the model, and we illustrate its relevance with an evaluation on synthetic data. Its practical usefulness is also demonstrated in the context of point-wise sound source localization. Further work could include a specific study about the specialization of the predictors, and the validation of the proposed algorithm in increasingly complex real-world datasets.

Acknowledgments

This work was funded by the French Association for Technological Research (ANRT CIFRE contract 2022-1854). We would like to thank the reviewers for their valuable feedback.

References

Sharath Adavanne, Archontis Politis, Joonas Nikunen, and Tuomas Virtanen. Sound event localization and detection of overlapping sources using convolutional recurrent neural networks. IEEE Journal of Selected Topics in Signal Processing , 13(1):34-48, 2018. 7, 8, 14, 15, 16, 17

Sharath Adavanne, Archontis Politis, and Tuomas Virtanen. Direction of arrival estimation for multiple sound sources using convolutional recurrent neural network. In 2018 26th European Signal Processing Conference (EUSIPCO) , pages 1462-1466. IEEE, 2018. 7

Sharath Adavanne, Archontis Politis, and Tuomas Virtanen. A multi-room reverberant dataset for sound event localization and detection. arXiv preprint arXiv:1905.08546 , 2019. 7, 17

Kazi Md Rokibul Alam, Nazmul Siddique, and Hojjat Adeli. A dynamic ensemble learning algorithm for neural networks. Neural Computing and Applications , 32:8675-8690, 2020. 2

Samuel S Blackman. Multiple hypothesis tracking for multiple target tracking. IEEE Aerospace and Electronic Systems Magazine , 19(1):5-18, 2004. 1

Mike Brodie, Chris Tensmeyer, Wes Ackerman, and Tony Martinez. Alpha model domination in multiple choice learning. In 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA) , pages 879-884. IEEE, 2018. 2

Armen Der Kiureghian and Ove Ditlevsen. Aleatory or epistemic? does it matter? Structural safety , 31(2):105-112, 2009. 2

Thomas G Dietterich. Ensemble methods in machine learning. In International workshop on multiple classifier systems , pages 1-15. Springer, 2000. 2

Qiang Du, Vance Faber, and Max Gunzburger. Centroidal voronoi tessellations: Applications and algorithms. SIAM review , 41(4):637-676, 1999. 3

Michael Firman, Neill DF Campbell, Lourdes Agapito, and Gabriel J Brostow. Diversenet: When one right answer is not enough. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5598-5607, 2018. 2, 3

Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspective. arXiv preprint arXiv:1912.02757 , 2019. 2

Yarin Gal et al. Uncertainty in deep learning. 2016. 2

Nuno Cruz Garcia, Sarah Adel Bargal, Vitaly Ablavsky, Pietro Morerio, Vittorio Murino, and Stan Sclaroff. Distillation multiple choice learning for multimodal action recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 2755-2764, 2021. 2

Alex Graves. Practical variational inference for neural networks. Advances in neural information processing systems , 24, 2011. 2

Pierre-Amaury Grumiaux, Sr ¯ dan Kiti´ c, Laurent Girin, and Alexandre Guérin. A survey of sound source localization with deep learning methods. The Journal of the Acoustical Society of America , 152(1):107-151, 2022. 7

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning , pages 1321-1330. PMLR, 2017. 10

Abner Guzman-Rivera, Dhruv Batra, and Pushmeet Kohli. Multiple choice learning: Learning to produce multiple structured outputs. Advances in neural information processing systems , 25, 2012. 1, 2, 3

Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE transactions on pattern analysis and machine intelligence , 12(10):993-1001, 1990. 2

Eddy Ilg, Ozgun Cicek, Silvio Galesso, Aaron Klein, Osama Makansi, Frank Hutter, and Thomas Brox. Uncertainty estimates and multi-hypotheses networks for optical flow. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 652-667, 2018. 2, 9, 16

Ehsan Imani and Martha White. Improving regression performance with distributional losses. In International conference on machine learning , pages 2157-2166. PMLR, 2018. 1

Leonid V Kantorovich. On the translocation of masses. In Dokl. Akad. Nauk. USSR (NS) , volume 37, pages 199-201, 1942. 8

Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems , 30, 2017. 2

Youngwook Kim, Jae Myung Kim, Zeynep Akata, and Jungwoo Lee. Large loss matters in weakly supervised multi-label classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14156-14165, 2022. 2

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. 6, 14

Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly , 2(1-2):83-97, 1955. 15

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems , 30, 2017. 2

Kimin Lee, Changho Hwang, KyoungSoo Park, and Jinwoo Shin. Confident multiple choice learning. In International Conference on Machine Learning , pages 2014-2023. PMLR, 2017. 2

Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why m heads are better than one: Training a diverse ensemble of deep networks. arXiv preprint arXiv:1511.06314 , 2015. 2

Stefan Lee, Senthil Purushwalkam Shiva Prakash, Michael Cogswell, Viresh Ranjan, David Crandall, and Dhruv Batra. Stochastic multiple choice learning for training diverse deep ensembles. Advances in Neural Information Processing Systems , 29, 2016. 1, 2, 3

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2018. 15

David JC MacKay. A practical bayesian framework for backpropagation networks. Neural computation , 4(3):448-472, 1992. 2

Osama Makansi, Eddy Ilg, Ozgun Cicek, and Thomas Brox. Overcoming limitations of mixture density networks: A sampling and fitting framework for multimodal future prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7144-7153, 2019. 1, 2, 16

Saeed Masoudnia and Reza Ebrahimpour. Mixture of experts: a literature survey. The Artificial Intelligence Review , 42(2):275, 2014. 3

Christian Rupprecht, Iro Laina, Robert DiPietro, Maximilian Baust, Federico Tombari, Nassir Navab, and Gregory D Hager. Learning in an uncertain world: Representing ambiguity through multiple hypotheses. In Proceedings of the IEEE international conference on computer vision , pages 3591-3600, 2017. 1, 2, 3, 5, 14

Justin Salamon, Christopher Jacoby, and Juan Pablo Bello. A dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM international conference on Multimedia , pages 1041-1044, 2014. 17

Christopher Schymura, Benedikt Bönninghoff, Tsubasa Ochiai, Marc Delcroix, Keisuke Kinoshita, Tomohiro Nakatani, Shoko Araki, and Dorothea Kolossa. PILOT: Introducing transformers for probabilistic sound event localization. In Interspeech 2021 , pages 2117-2121. ISCA, 2021. 14, 15, 16

Christopher Schymura, Tsubasa Ochiai, Marc Delcroix, Keisuke Kinoshita, Tomohiro Nakatani, Shoko Araki, and Dorothea Kolossa. Exploiting attention-based sequence-to-sequence architectures for sound event localization. In 2020 28th European Signal Processing Conference (EUSIPCO) , pages 231-235. IEEE, 2021. 7, 8, 14, 15, 16

John Shore and Rodney Johnson. Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy. IEEE Transactions on information theory , 26(1):26-37, 1980. 5

Hao Song, Tom Diethe, Meelis Kull, and Peter Flach. Distribution calibration for regression. In International Conference on Machine Learning , pages 5897-5906. PMLR, 2019. 10

Aswin Shanmugam Subramanian, Chao Weng, Shinji Watanabe, Meng Yu, and Dong Yu. Deep learning based multi-source localization with source splitting and its effectiveness in multi-talker speech recognition. Computer Speech & Language , 75:101360, 2022. 7

Harshavardhan Sundar, Weiran Wang, Ming Sun, and Chao Wang. Raw waveform based end-to-end deep convolutional network for spatial localization of multiple acoustic sources. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 4642-4646. IEEE, 2020. 7

Kai Tian, Yi Xu, Shuigeng Zhou, and Jihong Guan. Versatile multiple choice learning and its application to vision computing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6349-6357, 2019. 2, 4

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017. 15

Zehan Wang, Sihong Zhou, Yuyao Huang, and Wei Tian. Dsmcl: Dual-level stochastic multiple choice learning for multi-modal trajectory prediction. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC) , pages 1-6. IEEE, 2020. 1

Dong Yu, Morten Kolbæk, Zheng-Hua Tan, and Jesper Jensen. Permutation invariant training of deep models for speaker-independent multi-talker speech separation. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 241-245. IEEE, 2017. 7

Zhi-Hua Zhou, Jianxin Wu, and Wei Tang. Ensembling neural networks: many could be better than all. Artificial intelligence , 137(1-2):239-263, 2002. 2

Xiaoyan Zhu, Jiaxuan Li, Jingtao Ren, Jiayin Wang, and Guangtao Wang. Dynamic ensemble learning for multi-label classification. Information Sciences , 623:94-111, 2023. 2

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the problem of multiple-choice learning (MCL) in the regression setting with specific focus on the overconfidence problem and hypothesis collapse problem found in previous approaches for MCL where predictions from the heads corresponding to rare events are overestimated. The proposed rMCL model frames the problem as a multimodal conditional distribution estimation problem. The authors propose a new loss function that adds a new hypothesis scoring loss to the existing multi-target winner takes all (WTA) loss. \n\nThe proposed algorithm is compared against other baseline models on both a toy dataset, as well as a sound source localization (SSL) task. The experiments on the toy datasets visually show the effectiveness of the rMCL approach in tackling the overconfidence problem as compared to a method that does not use the additional loss term. The results on the SSL dataset show that the rMCL algorithm is practically applicable to problems with multimodal outputs with consistently better performance in the case of multiple sound sources as compared to the other algorithms.\n\nSTRENGTHS:\n- The presented approach builds on top of existing literature and solves the overconfidence problem using an additional term to the existing loss function. \n- The authors visually demonstrate the effectiveness of the approach using a simple toy dataset, and follow up with application to a real-world regression task (SSL).\n\nWEAKNESSES:\n- The experiment on SSL is not very clear to me and might benefit from some additional details. Specifically, I am not clear on the interpretation of the EMD and oracle error. My understanding is that for each input sound snippet, the model predicts the angles of each source from each of the output heads. Does the oracle only count the best prediction out of all the source angles predicted, while the EMD accounts for the overall error?\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nThe authors have addressed limitations with the presented work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose Resilient Multiple Choice Learning (rMCL), a modification of the Multiple Choice Learning (MCL) approach, for conditional distribution estimation in regression contexts where each input can have multiple target samples. While MCL is a straightforward strategy for multimodal density estimation, it uses the Winner-Takes-All (WTA) loss for various hypotheses. In regression situations, the prevailing MCL versions focus on combining the hypotheses, which may compromise the diversity of the predictions.\nIn contrast, rMCL employs a new learned scoring system, which is supported by a mathematical framework based on Voronoi tessellations of the output space. This approach allows for a probabilistic interpretation of the results. The authors tested rMCL using synthetic data and found it to be effective. They also applied it to the problem of sound source localization, demonstrating its practical utility and the relevance of its interpretation.\n\nSTRENGTHS:\nNovel Approach: The proposed Resilient Multiple Choice Learning (rMCL) provides a fresh perspective to tackle the Multiple Choice Learning (MCL) overconfidence problem, especially in regression settings. This offers a new method for researchers and engineers to approach this issue.\n\nLearned Scoring: rMCL is based on a learned scoring scheme that handles multi-target settings. This flexibility could allow rMCL to perform well across a variety of tasks and datasets.\n\nProbabilistic Interpretation: The authors provide a probabilistic interpretation of the model, which could help in understanding the model's behavior, tuning its performance, or extending it to new applications.\n\nEvaluation and Application: The paper demonstrates the practical utility of rMCL, especially in the context of sound source localization. It illustrates the resilience of the model in both synthetic and real-world datasets, showing that the method can handle real-world complexity.\n\nAdvantages Over Previous Methods: rMCL, applied to the Sound Source Localization (SSL) problem, seems to alleviate the issues related to imbalanced spatial positions and the source permutation problem. It does not require prior knowledge of the number of sources, which is a significant advantage in practical applications.\n\nWEAKNESSES:\nThe method heavily relies on the dependence on high-quality data: Like many machine learning models, the performance of rMCL might be significantly impacted by the quality and representativeness of the training data. Further evaluation might be required based on whether the model is affected in the presence of noise in the data and the quality of the labels. \n\nThe performance of the rMCL approach may not be suitable when one is approaching to scale the data wherein the size and complexity of the dataset would affect the nature of the machine learning task at hand especially when one considers the hypothesis space which is large. \n\nIt's not clear how rMCL would respond to different types of noise in the data or how robust it is to outliers.\n\nQUESTIONS:\nBased on the weakness it would be helpful for authors to provide comments on how the rMCL approach might work in case of noise in the data and whether there is any comparative analysis of WinnerTakesAll type approaches which take into consideration of the label data noise. \n\nDespite the probabilistic interpretation offered by the rMCL approach, it might still be challenging to understand and explain the model's decision-making process, which could be a limitation in certain applications where interpretability is crucial. How does authors see the probabilistic interpretation aiding/modified to overcome the interpretability of the model learnt? It might be the case where the probabilistic interpretation may yield incorrect result with overfitting of the model.\n\nLIMITATIONS:\nAuthors do state the limitations of the work clearly. \n\nThey review the performance of the model and conclude that it appears that the performance of Resilient Multiple Choice Learning (rMCL) is influenced by the number of hypotheses being considered. Particularly, attempting to predict an overly large number of hypotheses without prior knowledge can introduce errors into score predictions. In addition, using rMCL to address the overconfidence issue inherent in Winner-Takes-All (WTA) variants can slightly degrade the performance of the best hypothesis, though the reason for this is not currently known. Lastly, while ε-WTA seems to behave as expected with rMCL, the same is not observed for top-n-WTA, necessitating further study.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a technique for resilient Multiple Choice Learning (rMCL), which extends the vanilla Muliple Choice Learning (MCL) paradigm to conditional distributions for regression where multiple targets maybe sampled for each training input. It is known that MCL uses multiple scoring heads to score multiple hypothesis for a given input and suffers from the twin challenges of hypothesis collapse (where only a small subset of the possible prediction heads are trained well, as a result of the Winner Take All strategy) and overconfident predictions (where rare classes are overly represented). This work focuses principally on addressing  the latter issue. A key feature of the method is the use of a learned scoring scheme based on Voronoi tessellations which lends itself to a probabilistic interpretation. Results are reported on the sound source localization task and on synthetic data and compared against some standard alternatives.\n\nSTRENGTHS:\nThe following are the key strengths of the proposed approach:\n1. This is possibly the first work that extends MCL to a regression setting.\n2. The proposed solution attempts to overcome the overconfidence problem of a standard MCL system by casting the MCL as a conditional distribution estimation technique while allowing for a probabilistic interpretation of the same.\n3. Experiments show that the method, which is capable of assigning prediction probabilities to low density regions, is also interpretable.\n4. The code for the proposed method has been made available.\n\nWEAKNESSES:\nThe following are some of the principal weaknesses of the proposed approach:\n1. The proposed method does not deal with the hypothesis collapse problem of MCL methods.\n2. The contrast with prior approaches is not well brought out in the Related Works section.\n3. No performance is reported on more recent sound source localization datasets such as LOCATA or the DCASE 2019 dataset\n4. Moreover, the proposed approach falls way short of competing methods on the Oracle metrics, on both the reported datasets. The authors present too much emphasis on just the multimodal EMD results without justifying why the drop in Oracle distance metric should be ignored.\n\nQUESTIONS:\nFollowing are some of the questions/comments for the authors:\n1. How does the current work compare against prior works in Multilabel Learning? [1, 2]\n2. How do the methods perform on the Euclidean distance measure, compared with the Oracle Error?\n3. This reviewer fails to see the value in ensembling the proposed approach with WTA (Table 3). Why not combine with IE or PIT then?\n4. Please rephrase Line 111: \"More precisely....\"\n\nReferences:\n[1] Zhu, X., Li, J., Ren, J., Wang, J. and Wang, G., 2023. Dynamic ensemble learning for multi-label classification. Information Sciences, 623, pp.94-111.\n[2] Kim, Y., Kim, J.M., Akata, Z. and Lee, J., 2022. Large loss matters in weakly supervised multi-label classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 14156-14165).\n\nLIMITATIONS:\nYes the limitations of the method have been well addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}