SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We compared the behavior of our proposed approach rMCL against sMCL and independent ensembles (IE) using a 3-layer perceptron backbone with 256 hidden units and 20 output hypotheses, and  $\ell_2$  as the underlying distance. For rMCL, we used scoring weight  $\beta = 1$ . In addition to the full-fledged rMCL, we assess a variant denoted 'rMCL*' where a single negative hypothesis is uniformly selected during training. For independent ensembles, we trained with independent random initialization 20 single-hypothesis models, the predictions of which being combined to be comparable to multi-hypothesis models. The training processes were executed until the convergence of the training loss. We report the checkpoints for which the validation loss was the lowest. All networks were optimized using Adam [24]. We show in Figure 1 (top right) test predictions of the classical sMCL compared to those of the rMCL model. In visualizing the predictions where the ground-truth distribution is known, we observe empirically that the centroidal tessellation property,  $\mathbb{E}[Y_x^k] = f_\theta^k(x)$ , is verified with good approximation (see red points in Figure 1, bottom right). We otherwise notice the overconfidence problem of the sMCL model in low-density zones when the output distribution is multimodal by looking at the output prediction, e.g., for  $x = 0.1$  and  $x = 0.9$  in Figure 1 where the sMCL predictions fail in low-density zones. In contrast, rMCL solves this issue, assigning to each Voronoi cell a score that approximates the probability mass of the ground-truth distribution in this zone (see Figure 1, bottom). Furthermore, the independent ensembles (triangles) suffer from a collapse issue around the conditional mean of the ground-truth distribution. This behavior is expected as the minimizer of the continuous formulation of the risk (4) on the whole output space is the conditional expectation of the output given the input.
Figure 1: Comparisons on a 2D regression toy problem. Comparing sMCL with standard WTA ('sMCL', pink diamonds), proposed rMCL (blue circles), a variant of rMCL with a single negative hypothesis chosen uniformly at training ('rMCL*', purple crosses), and Independent Ensemble ('IE', yellow triangles). (Left) Test EMD for different inputs  $x$ ; (Right) Inference at  $x = 0.1, 0.6, 0.9$  and Voronoi tessellations generated by the hypotheses. Green points are samples from the ground-truth distribution at the corresponding input. Cells conditional mean (red points) matches with the predicted hypotheses. The score predicted in each cell, displayed as color saturation of the blue circles, as per scale on the right, approximates the corresponding proportion of points. At  $x = 0.1$  and 0.9, we can observe how rMCL tackles well the overconfidence issue in the low-density zones.
From a quantitative perspective, as demonstrated in Fig. 1, we can evaluate the effectiveness of rMCL in addressing the overconfidence issue compared to the classical sMCL. To ensure a fair comparison with sMCL settings, we have opted for using the Earth Mover's Distance (EMD) metric, which measures the discrepancy between the ground truth and predicted distributions. Predicted distributions are considered mixtures of Dirac deltas as in (12) for the metric computation, with sMCL predictions assigning uniform mode weights. As depicted in Fig. 1, rMCL outperforms sMCL when the target distribution is multimodal, i.e., for extreme values of  $x$ . However, in the unimodal scenario, the performances of rMCL and sMCL are fairly similar. Furthermore, the 'rMCL* variant (depicted by crosses) that is lighter regarding memory consumption exhibits performance comparable to the standard rMCL. Additional assessments, which are detailed in Appendix B.2, have demonstrated the robustness of the rMCL when corrupting the training dataset with outliers in the output space.
4 Experiments with audio data
4.1 Sound source localization as a conditional distribution estimation problem
Sound source localization (SSL) is the task of predicting the successive angular positions of audio sources from a multichannel input audio clip  $\pmb{x} \in \mathcal{X}$  [15]. This problem can be cast either as a regression problem (estimating the continuous positions of the sources, e.g., [37]) or a classification one (segmenting the output space into zones in which we wish to detect a source, e.g., [2]). While not requiring the number of sources to be known in advance, the classification approach suffers from the lack of balanced data and limited spatial resolution. On the other hand, the regression approach enables off-grid localization but typically suffers from the source permutation problem [40]. Assuming that the maximum number of overlapping sources  $M$  is known, a solution for handling this multi-target regression problem is to predict, at each time frame  $t$  depending on the chosen output resolution, a vector accounting for the sources' activity  $\pmb{a}_t \in \{0, 1\}^M$ , as well as azimuth and elevation angles  $\phi_t \in \mathbb{R}^M$  and  $\vartheta_t \in \mathbb{R}^M$ . A model can be trained with a permutation invariant training (PIT) approach [1, 37, 45] using an optimization criterion of the form
$$
\mathcal {L} (\theta) = \sum_ {t} \min  _ {\sigma \in \mathcal {S} _ {M}} \ell_ {\mathrm {C E}} \left(\sigma \left(\hat {\boldsymbol {a}} _ {t}\right), \boldsymbol {a} _ {t}\right) + \ell_ {g} \left(\left(\sigma \left(\hat {\boldsymbol {\phi}} _ {t}\right), \sigma \left(\hat {\boldsymbol {\vartheta}} _ {t}\right)\right), \left(\boldsymbol {\phi} _ {t}, \boldsymbol {\vartheta} _ {t}\right)\right), \tag {13}
$$
where  $\ell_{\mathrm{CE}}$  and  $\ell_g$  correspond respectively to a cross-entropy term and a geometrical loss, the latter being computed only for active sources indexes.  $S_M$  is the set of permutations of  $M$  elements and the notation  $\sigma(z)$  stands for the  $M$ -dim vector  $z$  with its components permuted according to  $\sigma$ . In the following, we will denote  $\mathbf{Y}_t$  the set of source positions at time  $t$ .
With the distribution learning mindset, this task can be seen as an attempt to estimate, at each time step, the sound source position distribution  $p(\boldsymbol{y} \mid \boldsymbol{x})$ , which can be viewed as a Dirac mixture,  $p(\boldsymbol{y} \mid \boldsymbol{x}) \propto \sum_{\boldsymbol{y}_i \in \mathbf{Y}_t} \delta_{\boldsymbol{y}_i}(\boldsymbol{y})$ , if we suppose that the targets are point-wise, with another Dirac mixture representing the predicted active modes at predicted positions. Therefore, a natural way to evaluate such SSL regression models is to solve the linear assignment problem, e.g., using Hungarian matching with spherical distance as an underlying metric. To handle more general distributions, we propose to generalize the metric used to the Earth Mover's Distance (see Sec. 4.2).
The rMCL framework is well suited to SSL as it allows one to benefit from both the advantages of the regression and classification viewpoints in the same spirit as [41]. There is no need for prior knowledge of the number of sources, and it avoids challenges related to imbalanced spatial positions and the source permutation problem of (13). This method comes at a low computational cost regarding added parameters when opting for a low-level representation shared by the hypotheses and scores heads (see Sec. 3.2). Furthermore, it allows for producing a heat map of the sound sources' positions with a probabilistic prediction, which could otherwise account for their spatial dispersion depending on the chosen law  $\pi_{k}$  selected in the Voronoi cells. Modeling the sound sources as point sources, a delta law will be selected following the formulation of Proposition 1.
Datasets. We conducted our experiments on increasingly complex SSL datasets originally introduced by Adavanne et al. [1]: i) ANSYN, derived from DCASE 2016 Sound Event Detection challenge with spatially localized events in anechoic conditions and ii) RESYN, a variant of ANSYN in reverberant conditions. Each dataset is divided into three sub-datasets depending on the maximum number of overlapping events (1, 2, or 3, denoted as D1, D2, and D3). These sub-datasets are further divided into three splits (including one for testing). Each training split contains 300 recordings (30 s), of which 60 are reserved for validation. Preprocessing is detailed in Appendix A.2. For each experiment, we used all training splits from D1, D2, and D3. Evaluation was then conducted for each test set based on the overlapping levels. Moreover, we present results from additional SSL datasets, including REAL [1] and DCASE19 [3], in Appendix B.3.
Metrics. To assess the performance of the different methods, we employed the following metrics:
The 'Oracle' error  $(\downarrow)$ :  $\mathcal{O}(\pmb{x}_n, \pmb{Y}_n) = \frac{1}{|\pmb{Y}_n|} \sum_{\pmb{y}_m \in \pmb{Y}_n} \min_{k \in [1, K]} d(f_\theta^k(\pmb{x}_n), \pmb{y}_m)$ .
- The Earth Mover's Distance (EMD  $\downarrow$ ): also known as the 1st Wasserstein distance, it is a distance measure between two probability distributions. In this context, these
are the predicted distribution,  $\hat{p} (\boldsymbol {y}\mid \boldsymbol {x}_n) = \sum_{k = 1}^{K}\gamma_\theta^k (\boldsymbol {x}_n)\delta_{f_\theta^k (\boldsymbol {x}_n)}(\boldsymbol {y})$ , and the ground-truth one,  $p(\boldsymbol {y}\mid \boldsymbol {x}_n) = \frac{1}{|\mathbf{Y}_n|}\sum_{\boldsymbol {y}_m\in \mathbf{Y}_n}\delta_{\boldsymbol {y}_m}(\boldsymbol {y})$ . The EMD solves the optimization problem  $W_{1}\left(p(\cdot |\boldsymbol{x}_{n}),\hat{p} (\cdot |\boldsymbol{x}_{n})) = \min_{\boldsymbol {\psi}\in \Psi}\sum_{k = 1}^{K}\sum_{\boldsymbol {y}_{m}\in \mathbf{Y}_{n}}\psi_{k,m}d(f_{\theta}^{k}(\boldsymbol{x}_{n}),\boldsymbol{y}_{m})\right.$ , where  $\psi = \big(\psi (f_{\theta}^{k}(\boldsymbol{x}_{n}),\boldsymbol{y}_{m})\big)_{k,m}$  is a transport plan and  $\Psi$  is the set of all valid transport plans [21].
While the oracle error assesses the quality of the best hypothesis predicted for each target, the EMD metric looks at all hypotheses. It provides insight into the overall consistency of the predicted distribution. The EMD also penalizes the overconfidence issue of the WTA with sMCL as described in Sec. 3.4. In order to fit the directional nature of the sound source localization task, these metrics are computed in a spherical space equipped with distance  $d(\hat{\pmb{y}}, \pmb{y}) = \arccos[\sin(\hat{\vartheta}) \sin(\vartheta) + \cos(\hat{\vartheta}) \cos(\vartheta) \cos(\phi - \hat{\phi})]$ , where  $(\hat{\phi}, \hat{\vartheta})$  and  $(\phi, \vartheta)$  are source positions  $\hat{\pmb{y}}$  and  $\pmb{y}$ , respectively, expressed as azimuth and elevation angles.
Neural network backbone. We employed the CRNN architecture of SeldNet [1] as a backbone, which we modified to enable multi-hypothesis predictions. This adaptation involves adjusting the output format and duplicating the last fully-connected layers for hypothesis and scoring heads. As only the prediction heads are adjusted, the number of parameters added to the architecture is negligible. Refer to Appendix A.2 for additional architecture and training details.
Baselines. We compared the proposed rMCL with several baselines, each utilizing the same feature-extractor backbone as previously described. The baselines include a Permutation Invariant Training variant ('PIT variant') proposed in [37] for SSL, the conventional WTA setup ('WTA, 5 hyp.') and its single hypothesis variant ('WTA, 1 hyp.'), its  $\varepsilon$ -relaxed version with  $\varepsilon = 0.5$  (' $\varepsilon$ -WTA, 5 hyp.) and its top- $n$  variant with  $n = 3$  ('top- $n$ -WTA, 5 hyp.), as well as independent ensembles ('IE'). To ensure a fair comparison, we used the same number of 5 hypotheses for the WTAs, sufficient to cover the maximum of 3 sound sources in the dataset (refer to Sec. 4.5 and the sensitivity study in Table 4). For the single hypothesis WTA, the head is updated by only considering the best target, as this fares better than using one update per target. IE was constructed from five such single hypothesis WTA models trained independently with random initialization.
4.3 Comparative evaluation of rMCL in SSL
Table 1: Source localization in anechoic conditions. Average scores (± standard deviation) on ANSYN dataset for PIT, IE, various WTA-based methods, and proposed rMCL. The 'EMD' evaluates all the hypotheses jointly, while the 'Oracle' error only looks at the best hypotheses. D1 corresponds to the single-target setting, while D2 and D3 consider up to 2 and 3 targets, respectively.
Dataset: ANSYN	EMD D1	EMD D2	EMD D3	Oracle D1	Oracle D2	Oracle D3
PIT variant	6.22 ± 0.80	14.65 ± 1.22	23.41 ± 1.39	3.58 ± 0.46	10.58 ± 0.91	18.10 ± 1.04
IE (5 members)	4.05 ± 0.48	21.64 ± 2.29	34.34 ± 2.37	1.24 ± 0.24	16.91 ± 2.05	28.82 ± 2.25
WTA, 1 hyp.	3.97 ± 0.55	24.69 ± 2.72	39.66 ± 2.67	3.97 ± 0.55	24.69 ± 2.72	39.66 ± 2.67
WTA, 5 hyp.	48.22 ± 1.78	44.41 ± 1.25	41.83 ± 0.96	3.56 ± 0.39	6.53 ± 0.44	10.44 ± 0.58
top-n-WTA, 5 hyp.	5.14 ± 0.80	18.09 ± 1.32	25.12 ± 1.30	3.33 ± 0.46	7.48 ± 0.79	13.54 ± 0.96
ε-WTA, 5 hyp.	5.51 ± 0.66	19.20 ± 1.78	28.39 ± 1.68	3.62 ± 0.57	10.86 ± 1.26	17.44 ± 1.22
rMCL, 5 hyp.	7.04 ± 0.58	13.87 ± 0.99	20.76 ± 1.04	3.85 ± 0.46	7.16 ± 0.67	11.29 ± 0.78
Table 2: Source localization in reverberant conditions. Results on RESYN dataset, with same table layout as in Table 1.
Dataset: RESYN	EMD D1	EMD D2	EMD D3	Oracle D1	Oracle D2	Oracle D3
PIT variant	10.53 ± 0.90	25.09 ± 2.14	35.05 ± 1.98	4.89 ± 0.55	15.92 ± 1.36	24.95 ± 1.62
IE (5 members)	8.24 ± 1.02	26.65 ± 2.49	38.7 ± 2.62	3.77 ± 0.61	20.95 ± 2.27	32.85 ± 2.48
WTA, 1 hyp.	8.32 ± 1.28	29.26 ± 2.85	43.25 ± 2.94	8.32 ± 1.28	29.26 ± 2.85	43.25 ± 2.94
WTA, 5 hyp.	57.88 ± 1.71	51.74 ± 1.36	47.38 ± 1.26	5.81 ± 0.58	9.46 ± 0.71	13.33 ± 0.69
top-n-WTA, 5 hyp.	42.74 ± 2.86	37.25 ± 1.88	36.48 ± 1.40	6.21 ± 0.79	11.02 ± 1.00	17.32 ± 1.11
ε-WTA, 5 hyp.	8.84 ± 1.09	27.3 ± 2.52	38.43 ± 2.42	6.48 ± 0.95	20.54 ± 2.33	30.18 ± 2.15
rMCL, 5 hyp.	12.14 ± 1.12	24.45 ± 1.91	32.28 ± 1.85	5.74 ± 0.66	10.5 ± 0.87	14.6 ± 0.87
We present in Tables 1 and 2 the results of rMCL and baselines in the anechoic (ANSYN) and reverberant (RESYN) conditions, respectively. The consistency of the results should be noted. For each model, the metrics tend to improve as the number of sources decreases and, under similar conditions, results are marginally lower for more challenging datasets. Unsurprisingly, the single hypothesis approach exhibits strong performances in the unimodal cases (D1), but performs significantly worse in the multimodal cases (D2 and D3). Ensembling (IE) improves its performance but still displays the lack of diversity already exposed in Section 3.4. As for the 5-hypothesis WTA, in its original form, it is ill-suited for multi-target regression due to the overconfidence issue: despite a strong oracle metric, the EMD results are very poor. On the other hand, rMCL surpasses its competitors and shows the best EMD in every multimodal setting. It also consistently obtains the second-best oracle error, only slightly above WTA, while not suffering from overconfidence.
We also present results for REAL and DCASE19 in Appendix B.3, that display similar trends. We did not observe the collapse issue (see Section 3.1) in our settings, neither with WTA nor with the proposed rMCL model. We believe it is solved in practice by the variability of the data samples in the stochastic optimization during training [19]; refer to Appendix B.1 for further discussions.
4.4 Combining rMCL with other WTA variants
Table 3: Combining proposed rMCL with other WTA variants. Average scores (± standard deviation) for source localization in anechoic (top) and reverberant (bottom) conditions.
Dataset: ANSYN	EMD D1	EMD D2	EMD D3	Oracle D1	Oracle D2	Oracle D3
rMCL, 5 hyp.	7.04 ± 0.58	13.87 ± 0.99	20.76 ± 1.04	3.85 ± 0.46	7.16 ± 0.67	11.29 ± 0.78
top-n-rMCL, 5 hyp.	5.46 ± 0.62	13.88 ± 1.06	21.45 ± 1.10	4.2 ± 0.55	8.04 ± 0.74	13.72 ± 0.89
ε-rMCL, 5 hyp.	5.89 ± 0.69	12.13 ± 0.98	19.95 ± 1.16	3.6 ± 0.54	8.76 ± 0.89	14.47 ± 1.02
Dataset: RESYN	EMD D1	EMD D2	EMD D3	Oracle D1	Oracle D2	Oracle D3
rMCL, 5 hyp.	12.14 ± 1.12	24.45 ± 1.91	32.28 ± 1.85	5.74 ± 0.66	10.5 ± 0.87	14.6 ± 0.87
top-n-rMCL, 5 hyp.	9.3 ± 1.15	23.81 ± 2.22	33.33 ± 2.06	6.99 ± 0.85	13.43 ± 1.40	19.72 ± 1.29
ε-rMCL, 5 hyp.	8.64 ± 1.03	22.82 ± 2.12	32.47 ± 1.97	6.08 ± 0.91	18.39 ± 2.07	26.92 ± 1.94
As noted in Section 3.2, rMCL's improvements are, in theory, orthogonal to that of other WTA variants. In this section, we combine the rMCL with top- $n$ -WTA and  $\varepsilon$ -WTA approaches into respectively top- $n$ -rMCL and  $\varepsilon$ -rMCL. The results are shown in Tables 3. We note that while top- $n$  rMCL does not provide significant improvement to rMCL,  $\varepsilon$ -rMCL, on the other hand improves EMD scores at the expense of increased oracle error. We observe similar effects when this method is applied to WTA, indicating that there are indeed some additive effects of  $\varepsilon$ -WTA and rMCL. Also note that with regards to Tables 1 and 2, all the proposed variants still get better EMD than all competitors under multimodal conditions, showing the robustness of our method.
4.5 Effect of the number of hypotheses
The impact of varying the number of hypotheses on the performance of the rMCL model is presented in Table 4 and Figure B.4. First and foremost, we notice the anticipated trend of the oracle metric improving as the number of hypotheses increases. Concerning the EMD metric, the single hypothesis model, which avoids errors from negative hypotheses, is most effective in handling unimodal
Table 4: Sensitivity analysis. Effect of the number of hypotheses on the performance of rMCL.
Dataset: ANSYN	EMD D1	EMD D2	EMD D3	Oracle D1	Oracle D2	Oracle D3
PIT variant	6.22 ± 0.80	14.65 ± 1.22	23.41 ± 1.39	3.58 ± 0.46	10.58 ± 0.91	18.10 ± 1.04
WTA, 1 hyp.	3.97 ± 0.55	24.69 ± 2.72	39.66 ± 2.67	3.97 ± 0.55	24.69 ± 2.72	39.66 ± 2.67
rMCL, 3 hyp.	9.89 ± 0.95	14.37 ± 0.91	20.96 ± 1.03	5.65 ± 0.73	8.6 ± 0.63	13.42 ± 0.86
rMCL, 5 hyp.	7.04 ± 0.58	13.87 ± 0.99	20.76 ± 1.04	3.85 ± 0.46	7.16 ± 0.67	11.29 ± 0.78
rMCL, 10 hyp.	9.14 ± 0.76	15.2 ± 0.84	21.28 ± 0.96	2.94 ± 0.35	4.76 ± 0.39	7.54 ± 0.50
rMCL, 20 hyp.	9.13 ± 0.71	16.04 ± 0.84	22.55 ± 0.87	2.06 ± 0.22	3.61 ± 0.30	5.83 ± 0.37
distributions. In the case of the EMD metric applied to multimodal distributions, we observe that multiple hypothesis models improve the results, but excessively increasing the number of hypotheses may marginally degrade performances. A further study could examine this phenomenon in detail, which may be related to the expressive power of the hypothesis heads or the accumulation of errors in score heads predictions.
In the audio experiments, the performance of rMCL was found to be affected by the number of hypotheses to tune depending on the complexity of the dataset. Moreover, as with the other variants of WTA, fixing the overconfidence issue with rMCL slightly degrades the performance of the best hypothesis (oracle error) for a reason that is yet to be determined. Otherwise, while  $\varepsilon$ -WTA behaves as expected with rMCL, trading off the quality of the best hypotheses for overall performance, top- $n$ -WTA does not exhibit the same behavior. This discrepancy warrants further investigation.
The probabilistic interpretation of rMCL, as presented in Algorithm 1 and in Sec. 3.3, states that the different hypotheses would be organized in an optimal partition of the output space, forming a Voronoi tessellation. Ideally, each hypothesis would capture a region of the distribution, and the scores representing how likely this zone would activate in a given context. This interpretation remains theoretically valid whenever train and test examples are sampled from the same joint distribution  $p(\pmb{x}, \pmb{y})$ . However, in realistic settings, the hypotheses might not adhere to meaningful regions, but this could be controlled and evaluated, provided that an external validation dataset is available. In future work, we intend to use calibration techniques [16, 39] to identify and alleviate these issues.
To sum up, this paper proposes a new Multiple Choice Learning variant, suitable for tackling the MCL overconfidence problem in regression settings. Our method is based on a learned scoring scheme that handles situations where a set of targets is available for each input. Furthermore, we propose a probabilistic interpretation of the model, and we illustrate its relevance with an evaluation on synthetic data. Its practical usefulness is also demonstrated in the context of point-wise sound source localization. Further work could include a specific study about the specialization of the predictors, and the validation of the proposed algorithm in increasingly complex real-world datasets.
This work was funded by the French Association for Technological Research (ANRT CIFRE contract 2022-1854). We would like to thank the reviewers for their valuable feedback.
[1] Sharath Adavanne, Archontis Politis, Joonas Nikunen, and Tuomas Virtanen. Sound event localization and detection of overlapping sources using convolutional recurrent neural networks. IEEE Journal of Selected Topics in Signal Processing, 13(1):34-48, 2018. 7, 8, 14, 15, 16, 17
[2] Sharath Adavanne, Archontis Politis, and Tuomas Virtanen. Direction of arrival estimation for multiple sound sources using convolutional recurrent neural network. In 2018 26th European Signal Processing Conference (EUSIPCO), pages 1462-1466. IEEE, 2018. 7
[3] Sharath Adavanne, Archontis Politis, and Tuomas Virtanen. A multi-room reverberant dataset for sound event localization and detection. arXiv preprint arXiv:1905.08546, 2019. 7, 17
[4] Kazi Md Rokibul Alam, Nazmul Siddique, and Hojjat Adeli. A dynamic ensemble learning algorithm for neural networks. Neural Computing and Applications, 32:8675-8690, 2020. 2
[5] Samuel S Blackman. Multiple hypothesis tracking for multiple target tracking. IEEE Aerospace and Electronic Systems Magazine, 19(1):5-18, 2004. 1
[6] Mike Brodie, Chris Tensmeyer, Wes Ackerman, and Tony Martinez. Alpha model domination in multiple choice learning. In 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 879-884. IEEE, 2018. 2
[42] Kai Tian, Yi Xu, Shuigeng Zhou, and Jihong Guan. Versatile multiple choice learning and its application to vision computing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6349-6357, 2019. 2, 4
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 15
[44] Zehan Wang, Sihong Zhou, Yuyao Huang, and Wei Tian. Dsmcl: Dual-level stochastic multiple choice learning for multi-modal trajectory prediction. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pages 1-6. IEEE, 2020. 1
[45] Dong Yu, Morten Kolbaek, Zheng-Hua Tan, and Jesper Jensen. Permutation invariant training of deep models for speaker-independent multi-talker speech separation. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 241-245. IEEE, 2017.
[46] Zhi-Hua Zhou, Jianxin Wu, and Wei Tang. Ensembling neural networks: many could be better than all. Artificial intelligence, 137(1-2):239-263, 2002. 2
[47] Xiaoyan Zhu, Jiaxuan Li, Jingtao Ren, Jiayin Wang, and Guangtao Wang. Dynamic ensemble learning for multi-label classification. Information Sciences, 623:94-111, 2023. 2