Resilient Multiple Choice Learning: A learned scoring scheme with application to audio scene analysis
Authors: Victor Letzelter
Mathieu Fontaine
Mickael Chen
Patrick Perez
Slim Essid
Gaël Richard
Venue: NeurIPS 2023 poster (2023)
Keywords: Multiple Choice Learning
Audio processing.
TL;DR: Resilient Multiple Choice Learning is an extension of Multiple Choice Learning approaches for conditional distribution estimation in regression settings, relying on an interpretable scoring scheme and applicable for sound source localization.

Abstract: We introduce Resilient Multiple Choice Learning (rMCL), an extension of the MCL approach for conditional distribution estimation in regression settings where multiple targets may be sampled for each training input.
Multiple Choice Learning is a simple framework to tackle multimodal density estimation, using the Winner-Takes-All (WTA) loss for a set of hypotheses. In regression settings, the existing MCL variants focus on merging the hypotheses, thereby eventually sacrificing the diversity of the predictions. In contrast, our method relies on a novel learned scoring scheme underpinned by a mathematical framework based on Voronoi tessellations of the output space, from which we can derive a probabilistic interpretation.
After empirically validating rMCL with experiments on synthetic data, we further assess its merits on the sound source localization problem, demonstrating its practical usefulness and the relevance of its interpretation.

Resilient Multiple Choice Learning: A learned scoring scheme with application to audio scene analysis
Victor Letzelter!iz
Mathieu Fontaine
Mickael Chen
Patrick Pérez!
Slim Essid
Gaël Richard
valco.ai, ai, Paris, France
LTCI, Télécom Paris, Institut Polytechnique de Paris, France
Abstract
We introduce Resilient Multiple Choice Learning (rMCL), an extension of the MCL approach for conditional distribution estimation regression settings where multiple targets may be sampled for each training input. Multiple Choice Learning is a simple framework to tackle multimodal density estimation, using the Winner- Takes-All (WTA) loss for set of hypotheses. In regression settings, the existing MCL. variants focus on merging the hypotheses, thereby eventually sacrificing the diversity of the predictions. In contrast, scoring scheme underpinned by a mathematical framework based on Voronoi tessel- lations of the output space, from which we can derive probabilistic interpretation. After empirically validating rMCL with experiments on synthetic data, we further assess its merits on the sound source localization task, demonstrating its practical usefulness and the relevance of its interpretation.
method relies novel learned
1 Introduction
Machine learning models are commonly trained produce, for any given input, single prediction. In most cases, for instance, when minimizing the empirical risk with quadratic loss, this prediction can be interpreted as the conditional output expectation given the input. However, there are many tasks for which the conditional output distribution can be multimodal, either by the nature of the task or due various sources of uncertainty. Temporal tracking and forecasting, for instance, are problems of this type [5, 32, 44]. In such cases, the mean might fall in low-density region of the conditional probability function, and it would be beneficial to predict multiple hypotheses instead [34, 20]. In this context, Multiple Choice Learning (MCL) has emerged as simple pragmatic solution [17, 29]. Thanks network architecture with multiple heads, MCL models can produce multiple hypotheses, one per head. During supervised training, the gradients are only computed for the head that provides the best prediction given the current input sample. This Winner-Takes- All (WTA) training scheme allows each head to specialize in region of the output space. Accordingly, Rupprecht et al. [34] have proposed a probabilistic interpretation MCL based on Voronoi tessellations.
On that account, MCL suffers from two significant issues: hypotheses collapse and overconfidence. Hypotheses collapse occurs during training when a head takes the lead, the others being almost never selected under WTA and thus not updated. This leads to situation where most heads are not trained effectively and produce meaningless outputs during inference. Overconfidence can also be observed at inference time when looking at MCL under the lens of its probabilistic interpretation. Hypotheses that correspond to rare events tend be over-eepreeented thus not reflecting the true distribution of
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

outputs well. Multiple variants of MCL have been developed to alleviate these issues, in particular for classification or segmentation tasks [29, 27, 42, 13].
In this paper, we are interested in MCL in the context of regression settings, which has been mostly overlooked. More specifically, we propose rMCL., for resilient Multiple Choice Learning, a MCL variant with a learned scoring scheme to mitigate overconfidence. Unlike prior work, rMCL is designed handle more general settings where multiple targets may be sampled for each input during training. Moreover, we propose a probabilistic interpretation of rMCL, and show that rMCL's scoring fixes overconfidence by casting multi-choie regression as conditional distribution estimation. While introduced in the context of the original WTA loss, the proposed scoring approach is also compatible with other variants of WTA. We validate our claims with experiments on synthetic data, and we further evaluate the performance of rMCL on the sound source localization problem. The accompanying code with a rMCL implementation made available.
2 Related Work
Uncertainty estimation. The estimation of machine learning model's uncertainty can be studied from different perspectives depending on the type of ambiguity being considered. The uncertainty can concern either the model (epistemic) the data (aleatoric) [7, 12]. Kendall and Gal [22] showed that those types of uncertainty can be combined in deep learning settings, placing both a distribution on the weights and the outputs of the model. Epistemic uncertainty estimation may, for instance, be achieved by variational approximation of the weights posterior [31, 14]. Independent ensembles (IE) of neural networks [18, 11] is another widespread technique for epistemic uncertainty approximation. Several neural networks, trained from different random initializations, provide samples for approximating the
weights posterior distribution [26].
Multiple Choice learning. Originally introduced by Guzman-Rive et al. [17] and adapted to deep learning settings by Lee et al. [28, 29], MCL is a framework in which multiple output heads propose different possible predictions. Since many datasets contain only a single output realization for each input, the heads trained with a Vinner-Takes-Al scheme where only the head that made the best prediction is updated. It can be understood as a specific case of dependent ensembles [8, 46, 4, 47] where the members interact with each other during training. Alternatively, [34] have shown that MCL can be adapted successfully for multi-label classification [23].
The Winner-Takes--e training scheme, however, can cause two main issues known as hypotheses collapse and overconfidence [34, 6, 10, 19, 10, 27, 42]. Most related previous works tackle either or both of those questions. Hypotheses collapse occurs during training when some heads receive little no updates due to bad initial values under the WTA loss and other hypotheses taking the lead. To fix this issue, Rupprecht al. [34] proposed relaxed allows the update of non-winner heads, albeit with gradients scaled by a small constant E. While this approach ensures that every head receives gradients updates, it also tends level the different heads, which counters the initial goal of WTA and thus needs careful tuning. Finally, one can leverage the evolving Winner-Takes-Ar
Sainer-Taaes-AAle
(RWTA, or e-WTA) loss that
loss [32], with the top-n heads getting updated (top-n-WTA) instead of only the best one. The authors validate that their approach, with a scheduled decreasing value for n, achieves improved conditioning of the outputs by reducing inconsistent hypotheses in the context of future frames prediction.
On the other hand, overconfidence is an issue that can be observed in inference when evaluating model as a density estimator. In this case, one would want the different outputs to be distributed across the different modes of the true conditional probability of the outputs. Empirical observations [27, 42] show that this is not usually the case, and rare events tend to be overly represented, rendering such model inadequate for integration within real-world Docision-making systems. In this context, Lee et al. [27] proposed Confidení Multiple Choice Learning (CMCL) for solving overconfidence in classification setup. Additionally to the WTA loss, CMCL is trained by maximizing the entropy of the class distributions predicted by the non-selected hypotheses. The classifier's final prediction is based on a simple average of the discrete class distributions predicted by each of the heads. Although designed for tackling the overconfidence problem, CMCL reduces the diversity of the hypotheses. On that account, Versatile Multiple Choice Learning [42] (vMCL) proposed to address the issue by leveraging a choice network aiming at predicting a score for each hypothesis head. This may be seen
'https:// pettps://githeg http:///ggghhubbbo/eeeeeeeee htt.///icttoeeeeeeeeeeeeooooooo.

---

as variant of Mixture-of ofaeee-of-Expeee Ex-eets approaches [33], in which the choice network is supervised with explicit targets. The final prediction vMCL is derived by weighting the class distributions from each hypothesis by their respective scores. These works focus on classification tasks and aggregate the hypotheses. thereby losing individual information from each head. In contrast, we address regression tasks and demonstrate how to benefit from the diversity of predictions without aggregation.
We propose revisit the vMCL approach for regression, single-target or multi-target, and to extend accordingly the mathematical interpretation of the WTA proposed by Rupprecht et al. [34].
3 rMCL regression and its probabilistic interpretation
After reviewing MCL, this section introduces resilient Multiple Choice Learning, dubbed rMCL. This proposed variant handles multi-target settings thanks to its scoring scheme. We then provide a distribution learning interpretation of rMCL. that is relevant to regression tasks, thus completing the MCL. probabilistic interpretations of prior works [9, 34].
3.1
Fixing the overconfidence issue in Multiple Choice Learning
Let X Rd and y IR be the input and output spaces in supervised learning setting, such that the training set S is composed of samples (æ., ya) from an underlying joint distribution p(æ, y) over X x V.
Multiple choice learning was proposed to address tasks with ambiguous outputs, ie., for which the ground-truth distribution p(y æ)) is multimodal [17, 29, 10]. Adapted by Lee et al. [29] for deep-leaminng settings in Stochastic Multiple Choice Learning (sMCL), it leverages several models fo _ (fp..... f5) E F(X, referred to as K hypotheses, trained using the Winner-takes-al all (or Oracle) loss. It consists, given underlying loss function l and for each sample (æs, ys) in the current batch, of first the computation of
((So(1z)) y.) - min e ( (ars), ys)
kE[1,K]
after each forward pass, followed by backpropagation on the winner hypothesis, that the minimizing one. Assuming now that set Y. of targets is available in the training set for input æ., Firman et al. [10] have shown that (1) can be generalized by updating the best hypothesis per target using the meta-loss
K
C (fo (2)), = Σ
Σ
(ye y (a.)) e (fo (x.)
(2)
BEY, k=1
where
)²x)) A Ey e (1)())))) < ((((()))))) Vr #k.
This loss can, however, lead to poor conditioning the output set of predictions (hypotheses collapse), with only one or a few hypotheses being exclusively selected for backpropagation. Furthermore, sMCL is subject to the overconfidence problem. Following Theorem I of [34], a necessary condition for minimizing the risk
K
/ E y*( (*())
(f)(z). 8) p(x, y)dydæ.
that each prediction (xx) coincides with the centroid of (xx) with respect to p(y x). We say
that the components ()i (x)}a form a centroidal Voronoi tessellation [9]. If l for instance the l-- loss, this condition means that for each non-zero probability cell k fö (x) amounts the conditional mean of p(y) æ)) within the Voronoi cell y* (z:). However, this theorem tells us nothing about the predictions in very low probability zones; in such regions, the inference-time predictions f (æ:) will be meaningless. During inference, the sMCL indeed faces a limitation: it cannot solely rely on the predicted hypotheses to identify Voronoi cells in the output space with low probability mass (see Figure 1). This observation leads us to propose hypothesis-scoring heads 7 ....... 7 F(X, [0, 1]). Those aim to predict, for unseen input æ. the probability P(Y,, E (*))) where Y,, ~ p(y x)) with the aim of mitigating this overconfidence issuc.
3

---

As highlighted above, the hypothesis output can be interpreted, in the context of risk minimization, as the conditional mean of the Voronoi cell )**(2) defines, providing that the cell has non-zero probability. Furthermore, given (6), the output of the score head 7 (az) can be interpreted as an While MCL variants have often been proposed for classification tasks, we are, to the best of our knowledge, the first to propose solve this issue for targe-t regression, interpreting the problem as multimodal conditional distribution estimation.
3.2 Resilient Multiple Choice Learning
We consider hereafter a problem of estimation of a multimodal conditional distribution denoted p(y æ²) for each input æ E X. In real-world setting, only one several samples (the targets) drawn from p(y) æ) are usually accessible. Sound source localization is concrete instance such multimodal prediction for whose p(y) a:,) represents the sound source position for an input audio clip Vg X at a given time t.
In this last example, each target sample in Y, may represent the location of a mode of the ground- -ruth multimodal distribution. For such multi-output regression tasks, the MCL training of randomly initialized multi- mupth-es model with scoring functions, (fg...., 4K ........ 7o .o......
can be adapted
as follows.
For each training sample (æ.,, Ys), let
(x,) k+ E [L, By E Yo.k* E argmin e(fō (x. (x.).
(5)
k
and KC_ (xx.) _ [1, K] -K+ (ars) be the set of positive (or winner) and negative hypotheses respec- tively. It is then possible to combine the multi-target WTA loss L in (2) with hypothesis scoring loss
Escoring (0) A
Σ
log T8 (x.)+
Σ
log (1 (1- (ac)))
(6)
1o+ (x.))
A EK (z.)
in a compound loss C B This novel approach differs from previous MCL variants [42] in its ability predict multimodal distributions in regression settings and by the introduction of separated scoring branches that are updated based on loss (6), such that the target for the scoring branch is the probability that hypothesis i is among the winners for that sample.
resource-efficien implementation of rMCL is achieved by deriving the hypotheses and score heads from shared representation, with distinct parameters at the final stages of the architecture (e.g., through independent fully connected layers). Designed as such, is also possible to reduce memory cost by updating only fraction of score heads associated with the negative hypotheses at each training step, typically with distinct samples k ~ (x.)) This trick could also alleviate the imbalanced binary classification task that the scoring heads face with large number of hypotheses (x=(x;;) ) > (x.)). (typically. the number of target samples at disposal is small relative to the number of hypotheses K). The variants of the WTT (e.g., top-n-WTT WTA, E-WTA, see Sec. 2) are also compatible with rMCL. Inference with the proposed rMCL model is outlined in Algorithm 1.
approximation of the probability of sample from p(y a:) to belong to this cell.
Algorithm Inference in the rMCL model
Input: Unlabelled input x E X. Trained hypotheses and score heads (fo..... f/ . To F(X x F(X', [0, 1))
Output: Prediction of the output conditional distribution p(y x)).
Perform a forward pass by computing fø (z)...... .. (z), rb (x)..... 70 (x).
2: Construct the associated Voronoi components (**( (3) with y 1K
3: Normalize the predicted scores (x) <-
7(())
4: If Yg ~ p(y x). then for such that (x)) > 0, interpret the predictions as estimations of
{
(x) P(Y, (*))
(7)
(x) Y Y, E (=))

---

3.3 Probabilistic interpretation at inference time
Let us consider trained rMCL model, such as the one described in the previous section. Following the theoretical interpretation of [34] and the motivation of solving the overconfidence issue, as explained in Section 3.1, the output objectives of the model can be summarized in (7). Although is possible, assuming that those properties are verified, to deduce the following law of total expectation
Y Σ (x)
(8)
the above quantity may not be informative enough, especially if the law of Yv pyy x)) is multimodal. Indeed, to be able to derive complete probabilistic interpretation, we still need to characterize how rMCL approximates the full conditional density of Yı.. For that purpose, it is sufficient to fix law Th for Y₂ within each Voronoi cell k to accurately represent the predicted distribution in each region of the output space per the rMCI. model, as shown in Proposition 1.
Proposition (Probabilistic interpretation of rMCL). With the above notations, let us consider a multi-hypothesis model with properties of (7). Let us furthermore assume that
Y T-
i.e., the law YE of y conditioned by the event (Ya E (2)} is described by my for each input X with constraint [[Y 11 (a)). Then, for each measurable set A C.C.
K Σ
P(Y, E A)= /
(
(10)
YEA k=
Proof. We have
P(Y, E A)
EsP(Ys E (n)*()))
(by sigma-additivity) (by Bayes' theorem) (by (9)).
Ex P(Y E (a)))), E A e (x)))))
E. (x(xx) Lyea Tn(y x)dy
Following the principle of maximum entropy [38], the uniform distribution inside each cell Y ~ (x)) is the insormnffree distribution, assuming that the output space y is with finite volume and the hypotheses lie in the geometric center of their associated cell. With this prior, the predicted distribution can be interpreted as
1 1 E (x)) ply x)) = Σ (x)
K
(11)
(x))
where V()*(a)) (a)) A (x)))(()) dy is the volume of y (az). Similarly, if we model the output distribu- tion as mixture of Dirac deltas such that in each cell k, 7x(x) (x) > 0 = Y 6 (2)) then
K
p(y x)) = 7 (3) (g).
(12)
3.4 Toy example
We build upon the toy example formalized in [34, $4.1] validate the proposed algorithm and its interpretation. In particular, we seek to illustrate how rMCL handles the overconfidence problem in regression task. The toy problem involves the prediction of a 2D distribution p(y x)) based on scalar input æ E X, where X' [0. 1]. The training dataset is constructed by selecting for each random
input z ((0, 1]) one of the four sections S 1- x (-1.0), Sg 1-, x [0. 1). S₃ [0. x] (-,, S4 = [0. 1] x [0, 1], with probabilities p (St) = (Sa) Whenever a region selected, a point is then sampled uniformly in this region. To slightly increase the complexity of the dataset, it has been chosen to enable the sampling of two (instead of one) target training samples with probability ((x) for each input æ. Here, q a piece-wise affine function with q(0) 1. g(t)) 0, and q(1) 1.
(Sa) p
5

---

We compared the behavior of our proposed approach rMCL against sMCL and independent ensembles (IE) using 3-layer perceptron backbone with 256 hidden units and 20 output hypotheses, and as the underlying distance. For rMCI, we used scoring weight 8 = 1. In addition to the full-fledged rMCL., we assess variant denoted 'rMCI." where single negative hypothesis is uniformly selected during training. For independent ensembles, we trained with independent random initialization 20 singlc-hypothesis models, the predictions of which being combined to be comparable to multi- hypothesis models. The training processes were executed until the convergence of the training loss. report the checkpoints for which the validation loss was the lowest. All networks were optimized using Adam [24]. We show in Figure 1 (top right) test predictions of the classical sMCL compared to those of the rMCL model. visualizing the predictions where the ground truth distribution is known, observe empirically that the centroidal tessellation property, ]| fă (a), is verified with good approximation (see red points Figure 1, bottom right). We otherwise notice the overconfidence problem of the sMCL. model in low-density zones when the output distribution is multimodal by looking at the output prediction, e.g., for x 0.1 and x = 0.9 in Figure where the sMCL predictiiss fail in low-density zones. In contrast, rMCL solves this issue, assigning each Voronoi cell score that approximates the probability mass of the ground-truth distribution in this zone (see Figure 1, bottom). Furthermore, the independent ensembles (triangles) suffer from a collapse issue around the conditional mean of the ground-truth distribution. This behavior is expected as the minimizer of the continuous formulation of the risk (4) on the whole output space is the conditional expectation the output given the input.
EMD vs.
Predictions
0.4
rMCL
SMCL rMCL
0.20
0.6
Voronal tessellation
0.
036
0.2
0.6
Figure 1: Comparisons on 2D regression toy problem. Comparing sMCL with standard WTA ('sMCL', pink diamonds), proposed rMCL (blue circles), variant of rMCL with single negative hypothesis chosen uniformly at training ('MMCL*) purple crosses), and Independent Ensemble ('IE', yellow triangles). (Lefi) Test EMD for different inputs æ;; (Right) Inference Voronoi tessellations generated by the hypotheses. Green points are samples from the ground-truth distribution at the corresponding input. Cells conditional mean (red points) matches with the predicted hypotheses. The score predicted in each cell, displayed color saturation of the blue circles, per scale on the right, approximates the corresponding proportion points. At x 0.1 and 0.9, we can observe how rMCL tackles well the overconfidence issue in the low-density zones.
0.1, 0.6, 0.9 and
From a quantitative perspective, as demonstrated in Fig. 1, we can evaluate the effectiveness of rMCL in addressing the overconfidence issue compared the classical sMCL. To ensure a fair comparison with sMCL settings, we have opted for using the Earth Mover's Distance (EMD) metric, which measures the discrepancy between the ground truth and predicted distributions. Predicted distributions are considered mixtures of Dirac deltas as in (12) for the metric computation, with sMCL predictions assigning uniform mode weights. As depicted in Fig. rMCL outperforms sMCL when the target distribution is multimodal, i.e., for extreme values of z. However, in the unimodal scenario, performances of rMCL and sMCL are fairly similar. Furthermore, the 'rMCL** variant (depicted by crosses) that is lighter regarding memory consumption exhibits performance comparable to the standard rMCL. Additional assessments, which are detailed Appendix B.2, have demonstrated the robustness of the rMCL when corrupting the training dataset with outliers in the output space.

---

Experiments with audio data
4.1 Sound source localization as a conditional distribution estimation problem
Sound source localization (SSL) is the task of predicting the successive angular positions of audio sources from multichannel input audio clip E X [15]. This problem can be cast either a regression problem (estimating the continuous positions of the sources, e.g., [37]) or a classification one (segmenting the output space into zones in which we wish to detect a source, e.g., [2]). While not requiring the number of sources to be known in advance, the classification approach suffers from the lack of balanced data and limited spatial resolution. On the other hand, the regression approach enables off- fff-grid localization but typically suffers from the source permutation problem [40]. Assuming that the maximum number of overlapping sources M is known, a solution for handling this multi-target regression problem is to predict, at each time frame I. depending on the chosen output resolution, vector accounting for the sources' activity a; E {0, } as well as azimuth and elevation angles Φ E RM and 0, E RM, A model can be trained with a permutation invariant training (PIT) approach [1, 37, 45] using an optimization criterion of the form
(()) = Σ min lce (a(ā)), az) + lg (l((2)). (6.: dESM
(.. 0:))
(13)
where lcE and fg correspond respectively to cross-entropy term and geometrical loss, the latter being computed only for active sources indexes. SM is the set of permutations of M elements and the notation o (z) stands for the M-dim vector z with its components permuted according to In the following, we will denote Y, the set of source positions at time t.
With the distribution learning mindset, this task can be seen as an attempt to estimate, at each time step, the sound source position distribution p'y æ)), which can be viewed as a Dirac mixture, EY, 6y (y). if we suppose that the targets are point-wise, with another Dirac mixture representing the predicted active modes at predicted positions. Therefore, a natural way to evaluate such SSL regression models is to solve the linear assignment problem, e.g., using Hungarian matching underlying metric. To handle more general distributions, propose to
p(y| æ) x Ey,er,
with spherical distance generalize the metric used to the Earth Mover's Distance (see Sec. 4.2).
The rMCL framework is well suited to SSL as it allows one to benefit from both the advantages of the regression and classification viewpoints in the same spirit as [41]. There no need for prior knowledge of the number of sources, and it avoids challenges related to imbalanced spatial positions and the source permutation problem of (13). This method comes at a low computational cost regarding added parameters when opting for a low-level representation shared by the hypotheses and scores heads (see S.2...2) Furthermore, it allows for producing a heat map of the sound sources' positions with probabilistic prediction, which could otherwise account for their spatial dispersion depending on the chosen law TT& selected in the Voronoi cells. Modeling the sound sources as point sources, delta law will be selected following the formulation of Proposition 1.
4.2 Experimental setup
Datasets. We conducted our experiments on increasingly complex SSL datasets originally introduced by Adavanne et al. [1]: i) ANSYN, derived from DCASE 2016 Sound Event Detection challenge with spatially localized events anechoic conditions and ii) RESYN, variant of ANSYN in reverberant conditions. Each dataset is divided into three sub-datasets depending the maximum number of overlapping events (1, 2, or 3, denoted as D1, D2, and D3). These sub-datasets are further divided into three splits (including one for testing). Each training split contains 300 recordings (30s), of which 60 are reserved for validation. Preprocessing is detailed in Appendix A.2. For each experiment, we used all training splits from D1, D2, and D3. Evaluation was then conducted for each test set based on the overlapping levels. Moreover, we present results from additional SSL datasets, including REAL [1] and DCASE19 99 [3], in Appendix B.3.
Metrics. To assess the performance of the different methods, we employed the following metrics:
. The 'Oracle' error (L): (FEn Yv) |..| Ey,,er, minke[1.k] d (JH yy))
The Earth Mover's Distance (EMD f): also known as the 1st Wasserstein distance, it is distance measure between two probability distributions. In this context, these
7

---

are the predicted distribution, p(3 xn) = Ex= (e( (..)) and the ground- truth one, ply an) - [8,, Ey,,. EY, 5ym (1). The EMD solves the optimization prob- lem W1
[x.] = min Ex=1 Eyn,e,, Vk,md(f (zn), vm), where vb (v (zn), y vm)) k. is transport plan and the set of all valid transport plans [21].
While the oracle error assesses the quality of the best hypothesis predicted for each target, the EMD metrie looks all hypotheses. It provides insight into the overall consistency of the pre- dicted distribution. The EMD also penalizes the overconfidence issue of the WTA with sMCL described in S.c.3.4. In order to fit the directional nature of the sound source localiza- tion task, these metrics are computed in a spherical space equipped with distance d(ū, y) = arccos(sin()) sin( + cos( cos(1) cos(r) - &)}. where (ő) o) and (ø, 0) are source positions 9 and y, respectively, expressed as azimuth and elevation angles.
Neural network backbone. We employed the CRNN architecture of SeldNet [1] as a backbone, which we modified enable multi- multi-hypotheer hhtthheyootttssssseeee predictions. This adaptation involves adjusting the output format and duplicating the last fully-connected layers for hypothesis and scoring heads. As only the prediction heads are adjusted, the number of parameters added the architecture is negligible. Refer to Appendix A.2 for additional architecture and training details.
Baselines. We compared the proposed rMCL with several baselines, each utilizing the same feature- extractor backbone as previously described. The baselines include a Permutation Invariant Training variant ('PIT variant') proposed in [37] for SSL., the conventional WTT setup ("WTA, 5 hyp.") and its single hypothesis variant ('WTA, hyp.'). ) its e-relaxed version with & = 0.5 ('e-WTA, 5 hyp.") and its top-n variant with ensure a fair comparison, we used the same number of 5 hypotheses for the WTAs, sufficient to cover the maximum of 3 sound sources in the dataset (refer to Sec. 4.5 and the sensitivity study in Table 4). For the single hypothesis WTA, the head is updated by only considering the best target, as this fares better than using one update per target. IE was constructed from five such single hypothesis WTA models trained independently with random initialization.
(" ('top-n- "t.p-n--TA. 5 hyp.'). )) as well independent ensembles ('IE"). To
4.3 Comparative evaluation of rMCL in SSL
Table 1: Source localization in anechoic conditions. Average scores (+: standard deviation) on ANSYN dataset for PIT, IE, various WTs-based methods, and proposed rMCL. The 'EMD' evaluates all the hypotheses jointly, while the 'Oracle" eгror only looks at the best hypotheses. DI corresponds to the single-target setting, while D2 and D3 consider up 2 and 3 targets, respectively.
Datasct: ANSYN
EMD D1
EMD D2
EMD D3
Oracle D1
Oracle D2
Oracle D3
PIT variant
6.22 0.80
14.65
1.22
23.41 + 1.39
3.58 + 0.46
10.58 0.91
18.10 +: 1.04
IE (5 members)
4.05 0.48
21.64 + 2.29
34.34
+ 2.37
1.24 + 0.24
16.91 + 2.05
28.82 + 2.25
WTA, I hyp-
3.97 0.55
24.69 + 2.72
39.66
+ 2.67
3.97 + 0.55
24.69 + 2.72
39.66 + 2.67
WTA, 5 hyp.
48.22 1.78
44.41 + 1.25
41.83
3.56
6.53 = 0.44
10,44 0.58
top-n-WTA, 5 hyp.
5.14 0.80
18.09 + 1.32
25.12
1.30
3.33 0.46
7.48 - 0.79
13.54 + 0.96
e- WTA 5 hyp.
5.51 0.66
19.20
+ 1.78
28.39
1 1.68
3.62 - 0.57
10.86 = 1.26
17.44 +: 1.22
IMCL. 5 hyp.
7.04 0.58
13.87
+ 0.99
20.76 + 1.04
3.85 = 0.46
7.16 = 0.67
11.29 + 0.78
Table 2: Source localization in reverberant conditions. Results on RESYN dataset, with same table layout as in Table 1.
Dataset: RESYN
EMD D1
EMD D2
EMD D3
Oracle D1
Oracle D2
Oracle D3
PIT variant
10.53 1 0.90
25.09 1 2.14
35.05 + 1.98
4.89 1 0.55
15.92 1 1.36
24.95 1: 1.62
IE (5 members)
8.24 = 1.02
26.65
+ 2.49
38.7
+ 2.62
3.77 + 0.61
20.95
+ 2.27
32.85 + 2.48
WTA, hyp.
8.32 + 1.28
29.26
+ 2.85
43.25
+ 2.94
8.32 + 1.28
29.26
+ 2.85
43.25 + 2.94
WTA. hyp.
57.88 - 1.71
51.74
1.36
47.38
+ 1.26
5.81 + 0,58
9.46 + 0.71
13.33 + 0.69
top-n-WTA. 5 hyp.
42.74 1:44::2 2.86
37.25
1 1.88
36.48
1.40
6.21 0.79
11.02 1 1.00
17.32
1 1.11
c- WTA, 5 hyp.
8.84
1.09
27.3 1 2.52
38.43
2.42
6.48
1 0.95
20.54
1: 2.33
30.18
2.15
MCC, hyp.
12.14 + 1.12
24.45 + 1.91
32,28
1.85
5.74 0.66
10.5 + 0.87
14.6 + 0.87
8

---

We present in Tables and 2 the results of rMCL and baselines in the anechoic (ANSYN) and rever- berant (RESYN) conditions, respectively. The consistency of the results should be noted. For each model, the metrics tend to improve as the number of sources decreases and, under similar conditions, results are marginally lower for more challenging datasets. Unsurprisingly. the single hypothesis approach exhibits strong performances in the unimodal cases (D1), but performs significantly worse in the multimodal cases (D2 and D3). Ensembling (IE) improves its performance but still displays the lack of diversity already exposed in Section 3.4. As for the 5-hypothesis WTA, in its original form, it ill-suited for multi-target regression due to the overconfidence issue: despite a strong oracle metric, the EMD results are very poor. On the other hand, rMCL surpasses its competitors and shows the best EMD every multimodal setting. It also consistently obtains the second- -best oracle error, only slightly above WTA, while not suffering from overconfidence.
We also present results for REAL and DCASE19 Appendix B.3, that display similar trends. We did not observe the collapse issue (see Section 3.1) in our settings, neither with WTA nor with the proposed rMCL model. We believe solved in practice by the variability of the data samples the stochastic optimization during training [19]; refer to Appendix B.1 for further discussions.
4.4 Combining rMCL with other WTA variants
Table 3: Combining proposed rMCL with other WTA variants. Average scores (# standard
deviation) for source localization in anechoic (top) and reverberant (bottom) conditions.
Dataset: ANSYN
EMD DI
EMD D2
EMD D3
Oracle DI
Oracle D2
Oracle D3
(MCL, hyp-
7.04 0.58
13.87 + 0.99
20.76 1.04
3.85 0.46
7.16 + 0.67
11.29 0.78
top-n-rMCL, hyp.
5,46 0.62
13.88
1.06
21.45
1.10
4.2 0.55
8.04 + 0.74
13.72 0.89
s-MMLL, 5 hyp.
5.89 0.69
12.13 + 0.98
19.95 1.16
3.6 0.54
8.76 + 0.89
14.47 1.02
Datasct: RESYN
EMD
EMD D2
EMD D3
Oracle D1
Oracle
Oracle D3
IMCL, 5 hyp.
12.14 1.12
24.45 + 1.91
32.28 + 1.85
5,74 0.66
10.5 + 0.87
14.6 + 0.87
top-n-rMCL, 5 hyp. 6-1MLL, hyp.
9.3 + 1.15
23.81
+ 2.22
33.33 2.06
6.99 0.85
13.43 + 1.40
19.72 1.29
8.64 + 1.03
22.82
2.12
32.47 1.97
6.08 0.91
18.39 = 2.07
26.92 1.94
As noted in Section 3.2, rMCL's 's improvements are, in theory, orthogonal to that of other WTA variants. In this section, we combine the rMCL with top-n-Wi WTA and g- ETAA approaches into respectively top-n-rMCL and E-MMCL. The results shown in Tables 3. We note that while top-n rMCL does not provide significant improvement to rMCL, E-MMCL, on the other hand improves EMD scores at the expense of increased oracle error. We observe similar effects when this method is applied to WTA, indicating that there are indeed some additive effects of e- WTA and rMCL. Also note that with regards to Tables 1 and 2, all the proposed variants still get better EMD than all competitors under multimodal conditions, showing the robustness of our method.
4.5 Effect of the number of hypotheses
The impact of varying the number of hypotheses on the performance of the rMCL model is presented in Table 4 and Figure B.4. First and foremost, notice the anticipated trend of the oracle metric improving as the number of hypotheses increases. Concerning the EMD metric, the single hypoth- esis model, which avoids errors from negative hypotheses, is most effective in handling unimodal
Table 4: Sensitivity analysis. Effect of the number of hypotheses the performance of rMCL.
Dataset: ANSYN
EMD D1
EMD D2
EMD D3
Oracle D1
Oracle D2
Oracle D3
PIT variant
6.22 0.80)
14.65
1.22
23.41
1.39
3.58 + 0.46
10.58 + 0.91
18.10 + 1.04
WTA, hyp.
3.97 0.55
24.69
2.72
39.66
2.67
3.97 + 0.55
24.69 + 2.72
39.66 + 2.67
rMCL, 3 hyp.
9.89 0.95
14.37
0.91
20.96
1.03
5.65
0.73
0.63
13.42 : 0.86
rMCL, 5 hyp.
7.04 0.58
13.87
0.99
20.76
1.04
3.85
7.16
11.29 1
rMCL, 10 hyp.
9,14:00
15.2 0.84
21.28
0.96
2.94
4.76 0.39
7.54 0.50
MMCL, 20 hyp.
9.13 0.71
16.04 0.84
22.55 0.87
2.06 0.22
3.61 0.30 0.30
5.83 0.37

---

distributions. In the case of the EMD metric applied to multimodal distributions, we observe that multiple hypothesis models improve the results, but excessively increasing the number of hypotheses may marginally degrade performances. A further study could examine this phenomenon in detail, which may be related to the expressive power of the hypothesis heads or the accumulation of errors in score heads predictions.
Discussion and limitations
In the audio experiments, the performance of rMCL was found to be affected by the number of hypotheses to tune depending on the complexity of the dataset. Moreover, as with the other variants of WTA, fixing the overconfidence issue with rMCL slightly degrades the performance of the best hypothesis (oracle error) for a reason that is yet to be determined. Otherwise, while e-WTA behaves expected with rMCL. trading off the quality of the best hypotheses for overall performance, top-n-WTA does not exhibit the same behavior. This discrepancy warrants further investigation.
The probabilistic interpretation of rMCL, presented in Algorithm and in Sec. 3.3, states that the different hypotheses would be organized in an optimal partition of the output space, forming Voronoi tessellation. Ideally, each hypothesis would capture a region of the distribution, and the scores representing how likely this zone would activate a given context. This interpretation remains theoretically valid whenever train and test examples are sampled from the same joint distribution p(a, y). However, in realistic settings, the hypotheses might not adhere to meaningful regions, but this could be controlled and evaluated, provided that external validation dataset is available. In future work, we intend to use calibration techniques [16, 39] to identify and alleviate these issues.
To sum up, this paper proposes new Multiple Choice Learning variant, suitable for tackling the MCL. overconfidence problem in regression settings. Our method is based on a learned scoring scheme that handles situations where set of targets is available for each input. Furthermore, we propose a probabilistic interpretation of the model, and we illustrate its relevance with an evaluation on synthetic data. Its practical usefulness is also demonstrated in the context of point-wise sound source localization. Further work could include a specific study about the specialization of the predictors, and the validation of the proposed algorithm in increasingly complex real- wolld datasets.
Acknowledgments
This work funded by the French Association for Technological Research (ANRT CIFRE contract 2022-1854). We would like to thank the reviewers for their valuable feedback.
References
[1] Sharath Adavanne, Archontis Politis, Joonas Nikunen, and Tuomas Virtanen. Sound event localization and detection of overlapping sources using convolutional recurrent neural networks. IEEE Journal of Selected Topics in Signal Processing, 13(1):34 48, 2018. 7, 8, 14, 15, 16, 17
[2] Sharath Adavanne, Archontis Politis, and Tuomas Virtanen. Direction of arrival estimation for multiple sound sources using convolutional recurrent neural network. In 2018 26th European Signal Processing Conference (EUSIPCO), pages 1462-1466. IEEE, 2018. 7
[3] Sharath Adavanne, Archontis Politis, and Tuomas Virtanen. A multi-room reverberant dataset for sound event localization and detection. arXiv preprint arXiv: (ee.i/9050..4.. 2019. 7, 17
[4] Kazi Md Rokibul Alam, Nazmul Siddique, and Hojjat Adeli. A dynamic ensemble learning algorithm for neural networks. Neural Computing and Applications, 32:8675-8699, 2020.
[5] Samuel S Blackman. Multiple hypothesis tracking for multiple target tracking. IEEE Aerospace and Electronic Systems Magazine, 1981)-5-18, 2004.
[6] Mike Brodie, Chris Tensmeyer, Wes Ackerman, and Tony Martinez. Alpha model domination in multiple choice learning. In 20/8 17th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 879-884. IEEE, 2018. 2
10

---

[7] Armen Der Kiureghian and Ove Ditlevsen. Aleatory or epistemic? does it matter? Structural safery, 31(2):105-112 2009.
[8] Thomas G Dietterich. Ensemble methods machine learning. International workshop on multiple classifier systems, pages 1-15. Springer, 2000.
[9] Qiang Du, Vance Faber, and Max Gunzburger. Centroidal voronoi tessellations: Applications and algorithms. SIAM review, 41(4))637-676, 1999. 3
[10] Michael Firman, Neill DF Campbell, Lourdes Agapito, and Gabriel J Brostow. Diversenet: When one right answer is not enough. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5598-5607, 2018. 3
[11] Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspective. arXiv preprint arXiv: 1 192..7777 2019.
[12] Yarin Gal et al. Uncertainty in deep learning. 2016. 2
[13] Nuno Cruz Garcia, Sarah Adel Bargal, Vitaly Ablavsky, Pietro Morerio, Vittorio Murino, and Stan Sclaroff. Distillation multiple choice learning for multimodal action recognition. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 7764, 2021.
[14] Alex Graves. Practical variational inference for neural networks. Advances in neural information processing systems, 24, 2011.
[15] Pierre-Amaury Grumiaux, Srdan Kitić, Laurent Girin, and Alexandre Guérin. A survey of sound source localization with deep learning methods. The Journal of the Acoustical Society of America, 152(11:007-111, 2022. 7
[16] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration modern neural networks. In International conference machine learning, pages 1321-1330. PMLR, 2017. 10
[17] Abner Guzman- Rivera, Dhruv Batra, and Pushmeet Kohli. Multiple choice learning: Learning to produce multiple structured outputs. Advances in neural information processing systems, 25,
2012. 1, 2,
[18] Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE transactions pattern analysis and machine intelligence, 12110):9331001, 1990. 2
[19] Eddy 1lg, Ozgun Cicek, Silvio Galesso, Aaron Klein, Osama Makansi, Frank Hutter, and Thomas Brox. Uncertainty estimates and multi-hypotheses networks for optical flow. In Proceedings of the European Conference on Computer Vision (ECCV), pages 652-667, 2018. 2, 16
[20] Ehsan Imani and Martha White. Improving regression performance with distributional losses. In International conference on machine learning, pages 2157-2166. PMLR, 2018.
[21] Leonid V Kantorovich. On the translocation of masses. In Dokl. Akad. Nauk. USSR (NS), volume 37, pages 199-201, 1942. 8
[22] Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances neural information processing systems, 30, 2017.
[23] Youngwook Kim, Jae Myung Kim, Zeynep Akata, and Jungwoo Lec. Large loss matters in weakly supervised label classification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14165, 2022.
[24] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014. 14
[25] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955. 15

---

[26] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017. 2
[27] Kimin Lee, Changho Hwang, KyoungSoo Park, and Jinwoo Shin. Confident multiple choice learning. In International Conference on Machine Learning, pages 2014-2023. PMLR, 2017. 2
[28] Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why m heads are better than one: Training a diverse ensemble of deep networks. arXiv preprint arXiv: 1X11:331 2015. 2
[29] Stefan Lee, Senthil Purushwalkam Shiva Prakash, Michael Cogswell, Viresh Ranjan, David Crandall, and Dhruv Batra. Stochastic multiple choice learning for training diverse deep ensembles. Advances in Neural Information Processing Systems, 29, 2016. 1, 2,
[30] Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2018. 15
[31] David JC MacKay. A practical bayesian framework for backpropagation networks. Neural computation, 4(3):448 472, 1992.
[32] Osama Makansi, Eddy 1gg. Ozgun Cicek, and Thomas Brox. Overcoming limitations of mixture density networks: sampling and fitting framework for multimodal future prediction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7144-7153, 2019. 1, 2, 16
[33] Saeed Masoudnia and Reza Ebrahimpour. Mixture of experts: a literature survey. The Artificial Intelligence Review, 42(2):275, 2014.
[34] Christian Rupprecht, Iro Laina, Robert DiPietro, Maximilian Baust, Federico Tombari, Nassir Navab, and Gregory D Hager. Learning in an uncertain world: Representing ambiguity through multiple hypotheses. In Proceedings of the IEEE international conference on computer vision, pages 3591-3600, 2017. 1, 2, 3, 14
[35] Justin Salamon, Christopher Jacoby, and Juan Pablo Bello. dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM international conference on Multimedia, pages
1041-1044, 2014. 17
[36] Christopher Schymura, Benedikt Bönninghoff, Tsubasa Ochiai, Mare Delcroix, Keisuke Ki- noshita, Tomohiro Nakatani. Shoko Araki, and Dorothea Kolossa. PILOT: Introducing trans- formers for probabilistic sound event localization. In Interspeech 2021, pages 2117-2121. ISCA, 2021. 14, 15, 16
[37] Christopher Schymura, Tsubasa Ochiai. Mare Delcroix, Keisuke Kinoshita, Tomohiro Nakatani, Shoko Araki, and Dorothea Kolossa. Exploiting tectures for sound event localization. In 2020 28th European Signal Processing Conference (EUSIPCO), pages 231-235. IEEE, 2021. 7, 8, 14, 15, 16
iitentionbbasee sequence-to-sequenn archi-
[38] John Shore and Rodney Johnson. Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy. IEEE Transactions on information theory, 26 26(1):26-37, 1980.
[39] Hao Song, Tom Diethe, Meelis Kull, and Peter Flach. Distribution calibration for regression. In International Conference on Machine Learning, pages 5897-5906. PMLR, 2019. 10
[40] Aswin Shanmugam Subramanian, Chao Weng, Shinji Watanabe, Meng Yu, and Dong Yu. Deep learning based multi-source localization with source splitting and its effectiveness in multi-talker speech recognition. Computer Speech & Language, 75:101366 2022.
[41] Harshavardhan Sundar, Weiran Wang, Ming Sun, and Chao Wang. Raw waveform based end-to-end deep convolutional network for spatial localization of multiple acoustic sources. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4642-4646. IEEE, 2020.
12

---

[42] Kai Tian, Yi Xu, Shuigeng Zhou, and Jihong Guan. Versatile multiple choice learning and its application to vision computing. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6349 6357, 2019. 2, 4
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017. 15
[44] Zehan Wang, Sihong Zhou, Yuyao Huang, and Wei Tian. Dsmel: Dual-level stochastic multiple choice learning for multi-modal trajectory prediction. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pages 1-6. IEEE, 2020.
[45] Dong Yu, Morten Kolbæk, Zheng-Hua Tan, and Jesper Jensen. Permutation invariant training of deep models for Conference on Acousties, Speech and Signal Processing (ICASSP), pages 241-245. IEEE, 2017.
multi-talked speech separation. In 2017 IEEE International
7
[46] Zhi-Hua Zhou, Jianxin Wu, and Wei Tang. Ensembling neural networks: many could be better than all. Artificial intelligence, (37(1-2):29--663, 2002. 2
[47] Xiaoyan Zhu, Jiaxuan Li, Jingtao Ren, Jiayin Wang, and Guangtao Wang. Dynamic ensemble learning for multi-label classification. Information Sciences, 623:94-111, 2023. 2
13


Reviews:
[{"review_id": "4YZ7prus1j", "reviewer": "Reviewer_yLZS", "summary": "This paper proposes a technique for resilient Multiple Choice Learning (rMCL), which extends the vanilla Muliple Choice Learning (MCL) paradigm to conditional distributions for regression where multiple targets maybe sampled for each training input. It is known that MCL uses multiple scoring heads to score multiple hypothesis for a given input and suffers from the twin challenges of hypothesis collapse (where only a small subset of the possible prediction heads are trained well, as a result of the Winner Take All strategy) and overconfident predictions (where rare classes are overly represented). This work focuses principally on addressing  the latter issue. A key feature of the method is the use of a learned scoring scheme based on Voronoi tessellations which lends itself to a probabilistic interpretation. Results are reported on the sound source localization task and on synthetic data and compared against some standard alternatives.", "questions": "Following are some of the questions/comments for the authors:\n1. How does the current work compare against prior works in Multilabel Learning? [1, 2]\n2. How do the methods perform on the Euclidean distance measure, compared with the Oracle Error?\n3. This reviewer fails to see the value in ensembling the proposed approach with WTA (Table 3). Why not combine with IE or PIT then?\n4. Please rephrase Line 111: \"More precisely....\"\n\nReferences:\n[1] Zhu, X., Li, J., Ren, J., Wang, J. and Wang, G., 2023. Dynamic ensemble learning for multi-label classification. Information Sciences, 623, pp.94-111.\n[2] Kim, Y., Kim, J.M., Akata, Z. and Lee, J., 2022. Large loss matters in weakly supervised multi-label classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 14156-14165).", "limitations": "Yes the limitations of the method have been well addressed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The following are the key strengths of the proposed approach:\n1. This is possibly the first work that extends MCL to a regression setting.\n2. The proposed solution attempts to overcome the overconfidence problem of a standard MCL system by casting the MCL as a conditional distribution estimation technique while allowing for a probabilistic interpretation of the same.\n3. Experiments show that the method, which is capable of assigning prediction probabilities to low density regions, is also interpretable.\n4. The code for the proposed method has been made available.", "weaknesses": "The following are some of the principal weaknesses of the proposed approach:\n1. The proposed method does not deal with the hypothesis collapse problem of MCL methods.\n2. The contrast with prior approaches is not well brought out in the Related Works section.\n3. No performance is reported on more recent sound source localization datasets such as LOCATA or the DCASE 2019 dataset\n4. Moreover, the proposed approach falls way short of competing methods on the Oracle metrics, on both the reported datasets. The authors present too much emphasis on just the multimodal EMD results without justifying why the drop in Oracle distance metric should be ignored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and comments, as well as for the suggestions for extending the experimental results of the paper. We provide here a detailed answer to the raised concerns.\n\n**The metrics interpretation**\n\n> “The authors present too much emphasis on just the multimodal EMD results”\n\nWe insist on multimodal settings rather than unimodal ones because rMCL is suited for multimodal density estimation. The proposed approach is particularly relevant when the output conditional mean is not the best solution, which generally corresponds to multimodal distributions to predict.\n\n> “without justifying why the drop in Oracle distance metric should be ignored.”\n\nIn our experiments, the focus was primarily on the EMD metric, as the Oracle does not address the issue of overconfidence. It indeed only considers the best hypotheses, without accounting for the global consistency of the prediction; if a low probability density zone is overestimated, this phenomenon will not be measured by the Oracle metric. This precision will be added to the paper.\n\n> “How do the methods perform on the Euclidean distance measure, compared with the Oracle Error?”\n\nAs highlighted in L.257, the Oracle error is computed using an underlying distance $d$ adapted to the output geometry, e.g., the Euclidean distance in the toy example or the spherical distance in the SSL task. For those benchmarks, we are only interested in the angular instead of Cartesian positions, therefore computing an Euclidean distance is not representative of the task. This point will be clarified in the paper. \n\n> “This reviewer fails to see the value in ensembling the proposed approach with WTA (Table 3). Why not combine with IE or PIT then?”\n\nThese choices are explained by the fact that our approach can be seen as an extension of the WTA training scheme. As such we wanted to evaluate how our approach would combine with commonly used WTA extensions (top-$n$ and $\\varepsilon$). It is not straightforward to combine it with PIT or IE. Indeed PIT is not related to MCL and IE was constructed from single-hypothesis WTA models that are not amenable to our method.\n\n**Performance on more SSL datasets**\n\nFollowing the reviewer's suggestion, we conducted more experiments on SSL datasets: REAL [1] and DCASE 2019 [A], where the maximum overlapping events are three and two. As with the manuscript's tables, model evaluation occurred separately in unimodal and multimodal conditions. For REAL and DCASE19, we computed mean metric values after 2 and 3 training runs, respectively, presented in Tables A and B.\n\n**Tables A & B reveal trends aligned with the paper's findings:**\n\n- Consistent with Section 4.5's analysis, increasing the number of hypotheses improves the oracle, but also degrades slightly the EMD while still surpassing the PIT baseline in multimodal settings.\n- The winner-takes-all approach and the IE variant, with one hypothesis and a single target update (see L.21-23 in Supplementary material), still outperform the other methods whenever a single source position is to be predicted. \n- While vanilla WTA slightly outperforms our method on the Oracle metric with the same number of hypotheses, a significant gap remains in EMD metric when predicting multiple hypotheses (e.g., 5).\n- rMCL shows consistent performance across datasets, where top-n and $\\varepsilon$-WTA can show a very wide disparity. \n\nThe optimal number of hypotheses on DCASE19 is lower than in previous datasets, probably due to its less multimodal nature. We conducted various visualizations to compare baselines and the proposed approach, confirming the competitive performance on those two SSL datasets.\n\n**The collapse problem**\n\nIn our audio experiments, neither the vanilla WTA nor the proposed rMCL model exhibited collapse. We confirmed this by analyzing histograms of winner hypothesis heads from trained models during testing, as illustrated by Fig. A's 20-hypothesis rMCL model trained on ANSYN. As mentioned in [13, p.8], we think collapse is in practice solved by the variability of the data samples and the training stochasticity. This is why we did not study the collapse problem, but we will include this discussion in the Supplementary. \n\n**Comparison with prior work in Multi-label learning [A,B]**\n\nWe thank the reviewer for the suggested references, to be included in the Related Work. [A] approaches the topic of Multi-label learning from the ensemble learning perspective, while [B] addresses multi-label classification in images with missing labels, presenting a variant of multiple choice learning. Each hypothesis in this context would aim to predict a possible class present in the image while expecting diverse predictions [18,20,29]. These two approaches are, however, classification methods whereas our paper focuses specifically on MCL for regression tasks. Adapting such methods to multi-source regression is not straightforward.\n\n**About the related work section**\n\nThe related work section will be enhanced, focusing on contrasting it with the paper's contribution.\n- The link between uncertainty estimation and MCL will be made clearer through ensembling [20] (L.59-60 in the manuscript). \n- The contrast with previous works in MCL [13,20,29,18,11,24,22,6] will be emphasized; ours tackles the overconfidence problem in regression settings without merging the hypotheses (e.g., [13,24,22]), by revisiting [29]. This results in a gain in diversity of the hypotheses and is suitable for extending the MCL probabilistic interpretation proposed in [24]. \n\nAs suggested, the sentence in L.111 will be rephrased.\n\n[A] Zhu, X., Li, J., Ren, J., Wang, J., & Wang, G. (2023). Dynamic ensemble learning for multi-label classification. Information Sciences, 623, pp. 94-111. \n\n[B] Kim, Y., Kim, J. M., Akata, Z., & Lee, J. (2022). Large loss matters in weakly supervised multi-label classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 14156-14165)."}, {"review_id": "VcH1m6NypO", "reviewer": "Reviewer_YyHX", "summary": "The authors propose Resilient Multiple Choice Learning (rMCL), a modification of the Multiple Choice Learning (MCL) approach, for conditional distribution estimation in regression contexts where each input can have multiple target samples. While MCL is a straightforward strategy for multimodal density estimation, it uses the Winner-Takes-All (WTA) loss for various hypotheses. In regression situations, the prevailing MCL versions focus on combining the hypotheses, which may compromise the diversity of the predictions.\nIn contrast, rMCL employs a new learned scoring system, which is supported by a mathematical framework based on Voronoi tessellations of the output space. This approach allows for a probabilistic interpretation of the results. The authors tested rMCL using synthetic data and found it to be effective. They also applied it to the problem of sound source localization, demonstrating its practical utility and the relevance of its interpretation.\n", "questions": "Based on the weakness it would be helpful for authors to provide comments on how the rMCL approach might work in case of noise in the data and whether there is any comparative analysis of WinnerTakesAll type approaches which take into consideration of the label data noise. \n\nDespite the probabilistic interpretation offered by the rMCL approach, it might still be challenging to understand and explain the model's decision-making process, which could be a limitation in certain applications where interpretability is crucial. How does authors see the probabilistic interpretation aiding/modified to overcome the interpretability of the model learnt? It might be the case where the probabilistic interpretation may yield incorrect result with overfitting of the model. ", "limitations": "Authors do state the limitations of the work clearly. \n\nThey review the performance of the model and conclude that it appears that the performance of Resilient Multiple Choice Learning (rMCL) is influenced by the number of hypotheses being considered. Particularly, attempting to predict an overly large number of hypotheses without prior knowledge can introduce errors into score predictions. In addition, using rMCL to address the overconfidence issue inherent in Winner-Takes-All (WTA) variants can slightly degrade the performance of the best hypothesis, though the reason for this is not currently known. Lastly, while ε-WTA seems to behave as expected with rMCL, the same is not observed for top-n-WTA, necessitating further study.\n", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Novel Approach: The proposed Resilient Multiple Choice Learning (rMCL) provides a fresh perspective to tackle the Multiple Choice Learning (MCL) overconfidence problem, especially in regression settings. This offers a new method for researchers and engineers to approach this issue.\n\nLearned Scoring: rMCL is based on a learned scoring scheme that handles multi-target settings. This flexibility could allow rMCL to perform well across a variety of tasks and datasets.\n\nProbabilistic Interpretation: The authors provide a probabilistic interpretation of the model, which could help in understanding the model's behavior, tuning its performance, or extending it to new applications.\n\nEvaluation and Application: The paper demonstrates the practical utility of rMCL, especially in the context of sound source localization. It illustrates the resilience of the model in both synthetic and real-world datasets, showing that the method can handle real-world complexity.\n\nAdvantages Over Previous Methods: rMCL, applied to the Sound Source Localization (SSL) problem, seems to alleviate the issues related to imbalanced spatial positions and the source permutation problem. It does not require prior knowledge of the number of sources, which is a significant advantage in practical applications.\n\n", "weaknesses": "The method heavily relies on the dependence on high-quality data: Like many machine learning models, the performance of rMCL might be significantly impacted by the quality and representativeness of the training data. Further evaluation might be required based on whether the model is affected in the presence of noise in the data and the quality of the labels. \n\nThe performance of the rMCL approach may not be suitable when one is approaching to scale the data wherein the size and complexity of the dataset would affect the nature of the machine learning task at hand especially when one considers the hypothesis space which is large. \n\nIt's not clear how rMCL would respond to different types of noise in the data or how robust it is to outliers.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their insightful comments. \n\n> Based on the weakness it would be helpful for authors to provide comments on how the rMCL approach might work in case of noise in the data and whether there is any comparative analysis of WinnerTakesAll type approaches that take into consideration of the label data noise.\n\nThe reviewer is asking about the robustness of the model in the presence of label noise, which is a challenging problem. The label noise corresponds to the aleatoric uncertainty [8,4], which can not be reduced with more data (e.g., corresponding to measurement error). \n\nTo the best of our knowledge, there is no comparative analysis of Winner-Takes-All type approaches in the context of noise; we provide, in the next section, insights about the resilience of the proposed model in the presence of outliers in the data. \n\n**Performance of rMCL in the presence of target outliers**\n\nWe thank the reviewer for their question which prompted some interesting investigation. We believe that rMCL, thanks to the partition into Voronoi tessellations and the scoring scheme, has good properties for handling outliers. To illustrate this, we propose the following experiment.\n\nLet's consider a setting where we have outliers in the training dataset, for instance, the toy-use case presented in the paper, where for each training example sampled, the probability of getting an outlier is $p \\ll 1$, e.g., modeled with a bivariate Cauchy distribution. Then, whenever an outlier is sampled, one hypothesis will be pushed towards it with its associated score heads updated. As the training goes on, some of the hypotheses will manage the outliers samples; let's name them the \"outlier hypotheses\". Thanks to the proposed hypothesis scoring heads, the model will also learn the probability that an outlier hypothesis is chosen for a given training sample. Provided that the outlier likelihood is $p\\ll 1$, the scoring heads will therefore prevent outlier hypotheses output from deteriorating the quality of the predicted distribution by rMCL. In Fig.A an illustration of this phenomenon is proposed using a Cauchy distribution (we used $p=0.02$). We notice the above-explained phenomenon, where the so-called outlier hypotheses account for the outlier samples, while the other hypotheses lie in the square $[-1,1]^{2}$ representing the samples from the ground-truth distribution. \n\nProvided that the probability of sampling an outlier $p$ is small enough and the outliers are far enough from the ground-truth distribution to predict, the proposed rMCL model is therefore potentially robust to outliers. In this case, some specific hypotheses, namely the outlier hypotheses, will be assigned to them, preventing the non-outliers hypothesis from being heavily affected. At inference time, it will indeed be possible to set to zero the very low-score hypotheses given an arbitrary threshold, so that the outlier hypotheses are not taken into account. \n\n> \"Despite the probabilistic interpretation offered by the rMCL approach, it might still be challenging to understand and explain the model's decision-making process, which could be a limitation in certain applications where interpretability is crucial. How do authors see the probabilistic interpretation aiding/modified to overcome the interpretability of the model learnt? It might be the case where the probabilistic interpretation may yield incorrect results with overfitting of the model.\"\n\nThe probabilistic interpretation allows us to state that the different hypotheses would be organized in an optimal partition of the output space forming a Voronoi tessellation, providing insights about the distribution to predict through the score heads. In an ideal case, this would lead to each hypothesis capturing a region of the distribution, and the scores being how likely this zone would activate in a given context. Of course, in a realistic setting, the hypothesis might not adhere to a meaningful region, but this can be controlled and evaluated provided we have additional annotations in our data.\n\nAnother asset of our approach is that the scoring function, similarly to a classifier, should provide an insight about the confidence of our model on each hypothesis. While the predicted probabilities can be wrong as stated by the reviewer, we can evaluate and calibrate the scoring function independently on a validation set [A,B], which should allow us to identify and alleviate such issues.\n\nWe hope that we understood the concerns of the reviewer correctly and that these considerations address them. We would be happy to further discuss these topics and clarify any potential misunderstanding.\n\n[A] Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017, July). On calibration of modern neural networks. In International conference on machine learning (pp. 1321-1330). PMLR.\n\n[B] Song, H., Diethe, T., Kull, M., & Flach, P. (2019, May). Distribution calibration for regression. In International Conference on Machine Learning (pp. 5897-5906). PMLR."}, {"review_id": "eUJETKhhnI", "reviewer": "Reviewer_3WKf", "summary": "The paper tackles the problem of multiple-choice learning (MCL) in the regression setting with specific focus on the overconfidence problem and hypothesis collapse problem found in previous approaches for MCL where predictions from the heads corresponding to rare events are overestimated. The proposed rMCL model frames the problem as a multimodal conditional distribution estimation problem. The authors propose a new loss function that adds a new hypothesis scoring loss to the existing multi-target winner takes all (WTA) loss. \n\nThe proposed algorithm is compared against other baseline models on both a toy dataset, as well as a sound source localization (SSL) task. The experiments on the toy datasets visually show the effectiveness of the rMCL approach in tackling the overconfidence problem as compared to a method that does not use the additional loss term. The results on the SSL dataset show that the rMCL algorithm is practically applicable to problems with multimodal outputs with consistently better performance in the case of multiple sound sources as compared to the other algorithms.", "questions": "See above", "limitations": "The authors have addressed limitations with the presented work.", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The presented approach builds on top of existing literature and solves the overconfidence problem using an additional term to the existing loss function. \n- The authors visually demonstrate the effectiveness of the approach using a simple toy dataset, and follow up with application to a real-world regression task (SSL).", "weaknesses": "- The experiment on SSL is not very clear to me and might benefit from some additional details. Specifically, I am not clear on the interpretation of the EMD and oracle error. My understanding is that for each input sound snippet, the model predicts the angles of each source from each of the output heads. Does the oracle only count the best prediction out of all the source angles predicted, while the EMD accounts for the overall error?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the remarks and the feedback on the paper.\n\nThe reviewer is indeed correct in that the EMD (Earth mover's distance or Wasserstein-1 metric [A]) considers all the hypotheses predicted and their associated scores. When it comes to the oracle metric, it evaluates only the best prediction for each target source (e.g., [11,20,29,24,22,18,6]). To be more detailed, we elaborate here-after on the SSL task, and the metrics related to multiple choice learning. We will clarify their presentation in the paper accordingly.\n\nGiven audio tracks recorded from a microphone array, the SSL task consists of predicting, at a given temporal rate, the positions of specific sound sources one is interested in. Sound sources can appear or disappear in the record, and their number can vary. As for the position, in those benchmarks, we are only interested in the angular position.\n\nBecause of the uncertain and multi-modal nature of the prediction task, it can be useful to cast it in a multiple choice learning or distribution learning framework. In such settings, the models output different plausible predictions or hypotheses for any given input. The difficulty that arises then is how to assess the quality of the multiple predictions of the model when only one realization of the ground-truth distribution is observed for any given input.\n\nWhenever a single target is available, which corresponds to one source to localize in SSL, one possibility is to compute the error for the prediction closest to the ground truth, that is the Oracle metric: we get 0 oracle error if the ground-truth is in the pool of predictions. When multiple targets are present, the oracle metric averages over the best hypothesis for each target [6]. Heavily used in the MCL framework [11,20,29,24,22,18,6], this metric therefore informs about the mean quality of the best hypotheses predicted.\n\nThe other metric, the Earth Mover’s distance (EMD), computes the optimal transport cost [A] between the ground truth and predicted distribution, both being cast as a mixture of Diracs in our case. Indeed in those audio datasets, sound sources can be assumed to be point-wise, and the ground-truth distribution to predict can be cast as a uniform mixture of Diracs. This considers all the hypotheses, weighted by their normalized score, and allows for a more complete evaluation; it informs about the global consistency of the hypotheses predicted [22]. \n\nBoth the Oracle and the EMD are equipped with an underlying distance adapted to the geometry of the problem. Since we are dealing with angles only, this underlying distance is angular distance here.\n\n[A] Kantorovitch, L. (1958). On the translocation of masses. Management science, 5(1), 1-4."}]

Author rebuttal:
We would like to thank the reviewers for their remarks and suggestions, which will allow us to improve the quality of the paper.

We summarize here the main changes that will be made to the submission in a next revision, in accordance with the reviewers inputs. Please refer to the individual responses for more detailed comments about those changes, as well as answers to the questions.

- We provide results on two additional sound source localization (SSL) datasets; REAL [1] and DCASE19 [A] in Tables A and B of the rebuttal. The manuscript will be updated accordingly [Reviewer yLZS].
- We will add new insights about the resilience of the proposed rMCL model in the presence of label noise and outliers (See Figure B of rebuttal) in the Supplementary [Reviewer YyHX].
- We will also expand on the motivations for the probabilistic interpretation and include the discussion in the main paper [Reviewer YyHX].
- A discussion regarding the collapse problem will be added in the Supplementary material [Reviewer yLZS].
- The interpretations of the metrics used, the EMD and Oracle will be clarified in the paper [Reviewers YyHX and 3WKf].
- Related work will be improved [Reviewer yLZS]. 

[A] Adavanne, S., Politis, A., & Virtanen, T. (2019). A multi-room reverberant dataset for sound event localization and detection. arXiv preprint arXiv:1905.08546.

Meta-review: This paper proposes a novel scoring scheme for Multiple Choice Learning (rMCL) in the multiple-target regression setting. This new scoring scheme is supported by casting the output space in Voronoi tessellations, overcoming the need to merge hypotheses (which sacrifices the diversity in predictions) and allows for a probabilistic interpretation.

All of the scores are positive. MCL often suffers from two challenges, hypothesis collapse and overconfident predictions. All reviewers felt the paper’s synthetic experiments demonstrated how the new method overcomes the overconfident predictions. One reviewer felt the collapse problem wasn’t addressed, and the authors illustrated in the rebuttal that the problem was not a concern in the application of sound source localization (SSL). In response to reviewers request, the authors also conducted new experiments on two more recent SSL datasets and found that the results aligned with the paper’s findings.

Initially, there was concern that the proposed method may not work well in the presence of noise and outliers. In the rebuttal the authors gave a detailed response on how prior work approached different kinds of noise and how their method is able to handle outliers. The authors also provided additional explanations on how their method offers interpretability. Reviewers found the insights and proposed experiments valuable.

Overall, reviewers found the work novel, technically sound, and practically applicable. Hence the recommendation is to accept the paper.
Decision: Accept (poster)