Uncertainty Quantification and Calibration for Audio-driven Disease Diagnosis

Venue: NeurIPS_cc_2024_Workshop_BDU
Published: 2024-08-29 13:41:39

Abstract: Deep learning excels in analyzing multi-modal signals for healthcare diagnostics but lacks the ability to quantify confidence in the predictions, which can lead to overconfident, erroneous diagnoses. In this work, we propose to predict model output independently and estimate the corresponding uncertainty. We present a unified audio-driven disease detection framework incorporating uncertainty quantification (UQ). This is achieved using a Dirichlet density approximation for model prediction and independent kernel distance learning in feature latent space for UQ. This approach requires minimum modifications to existing audio encoder architectures and is extremely parameter efficient compared to k-ensemble models. The uncertainty-aware model improves prediction reliability by producing confidence scores that closely match the accuracy values. Evaluations using the largest publicly available respiratory disease datasets demonstrate the advantage of the proposed framework in accuracy, training and inference time over ensemble and dropout methods.
	The proposed model improves speech and audio analysis for medical diagnosis by identifying and calibrating uncertainties, enabling better decision-making and risk assessment. This is shown by high uncertainty scores at low model accuracy. The proposed model contributes to speech technologies for healthcare by enhancing model transparency and reliability.

Main paper: 
 introduction: 
 The increase in general awareness and interest in speech technologies for disease diagnosis has generated significant growth in recorded public health datasets Song et al. (2023) ; Novikova and Balagopalan ([n. d.] ) across different modalities such as audio, imaging and time series (EEG). As the healthcare industry increasingly embraces data-driven approaches, the accurate interpretation of these subtle and complex multi-modal signals has become paramount for informed decision-making and improved patient outcomes. However, for these models to be useful in practical implementation, the outputs of such models must be explainable for medical decision making Miller (2019) . Multimodal medical datasets have been extensively researched for the task of disease diagnosis, symptom identification and monitoring Kulkarni et al. (2023) ; Wang and Wang (2022); Bae et al. (2023) .Popularly, large-scale convolutional neural network (CNN) architectures Demir et al. (2020) such as ResNet Gairola et al. (2021) ; Bengs et al. ([n. d.] ) trained on spectrogram images of audio inputs are used for this task. Recently, direct waveform speech encoders (Wav2Vec Baevski et al. (2020) , and PASE Ravanelli et al. (2020) ) have shown improved speech feature representations for respiratory monitoring Kulkarni et al. (2023) . After featurisation, a classification layer followed by softmax is used to produce output scores. However, fixed softmax scores may result in fundamentally incorrect outputs without indicating that the estimate is uncertain. Thus, achieving a statistically nuanced understanding of model outputs via uncertainty quantification (UQ) is crucial in safety-critical applications such as disease detection.Submitted to Workshop on Bayesian Decision-making and Uncertainty, 38th Conference on Neural Information This can be illustrated with a simple example of speech-driven disease detection. Input audio can be either "healthy" or COVID-"positive." A softmax-based classifier gives scores that express the likelihood of two different classes. Figure 1 (left) shows a histogram of the softmax scores coloured according to the correctness of the predicted output. The plot shows that irrespective of correctness of the prediction, the output confidence is always greater than 50%. The confidence score for two inputs (one predicted correctly and another incorrectly) lying on a vertical will be exactly same (healthy = 0.89, positive = 0.11) and (healthy = 0.11, positive = 0.89). Without UQ model, there is no way to decide the reliability of either prediction based on just softmax probabilities. An independent UQ estimate can quantify high uncertainty for false predictions, as shown in Figure 1 (right). An uncertainty-aware audio classification model enables 1) prediction of confidence scores independent of model outputs and 2) calibration of model such that estimated uncertainty closely follows model accuracy.In this work, we present a novel framework for uncertainty-aware disease detection using speech and non-speech inputs through quantification and disentanglement of sample uncertainty and model calibration. The framework comprises of a probabilistic classification head on top of a self-supervised audio encoder and model uncertainties are quantified using a feature distance-based metric. A training scheme is proposed to optimize uncertainty estimation independent of model prediction or classification training. A novel formulation of learnable transformation matrix in latent space is used to maximise feature space diversity for distance calculation. Evaluations show that the uncertaintyaware model produces low confidence scores at low accuracy values, thus improving output reliability.Experiments on the largest public respiratory disease datasets show that the proposed UQ model is generalizable, computationally efficient at training and enables fast evaluation during inference without sacrificing classification performance. Specifically, our contributions are as follows -• Advocate the use of a probabilistic classifier in place of softmax scores to quantify irreducible uncertainties inherent in learning problem for audio-driven disease diagnosis and medical decision making• Emphasize the necessity of model calibration for reducible uncertainties in audio-driven disease diagnosis. we show that combining probabilistic classifier simple k-ensembles (even with small k=5) significantly improves model calibration score• Propose a novel single inference method of uncertainty quantification with minimal changes to large encoder models for high-fidelity datasets such as audio and speech. The proposed model performs as well as k-ensembles at a fraction of compute and memory costs To best of our knowledge, this is the first systematic study of uncertainties quantification and model calibration associated with audio driven disease diagnosis. 
model: 
 Lets denote a(t) ∈ A as an input audio waveform and (y j = y j + ϵ j ) is its corresponding noisy label which takes a value from label space j ∈ {1, . . . J} and ϵ j is the label noise due to data gathering process or the noise inherent to the mapping problem G : A → J. We decompose above function 2. An uncertainty aware calibration training to estimate UQ as a function of feature space density. We use a novel learnable Mahalanobis distance-based metric, which ensures the latent space is bi-Lipschitz continuous and captures a measure of data distribution.In the subsequent sections, we describe these two component of the proposed UQ frameworkA deterministic softmax classifier only outputs a single scaled vector s(x) corresponding a input x such that j s(x) = 1. In contrast, the probabilistic classifier head is trained to predict a vector of concentration parameters α = (α 1 . . . α J ) one for each class label j ∈ J, and a strength parameter α 0 := j (α j ). This set of concentration parameters define a Dirichlet distribution Dir(α) with probability density given by equation 1, where Γ(•) denotes Gamma function. This is used to sample a class probability vector p as a random vector p ∼ Dir(α), At the inference time, a sample from Dirichlet distribution gives indicative probability p j of input x belonging to class j. The expected probability (mean) and the variance for a single input x is given byEQUATIONThus, the classifier head is a model with uncertainty that outputs two quantities corresponding to label distribution, the mean µ(x) and the variance σ(x). for input X as an empirical expectation over all inferences k. A combination of Deep EnsembleVar * (x) = 1 K k σ 2 k (x) + 1 K k µ 2 k (x) -µ 2 * (x) = E k [σ 2 k (x)] + E k [µ 2 k (x)] -E k [µ k (x)] 2 = E k [σ 2 k (x)] Aleatoric Uncertainty + Var k [µ k (x)]Epistemic Uncertainty and Dirichlet Probabilistic classifier gives an estimate for the Irreducible Aleatoric Uncertainty and Model Uncertainty (Epistemic). However, it is neither possible to treat each term separately nor to reduce epistemic part of uncertainty. Despite the limitations, the k-ensemble approach is shown to be the state-of-the art for uncertainty prediction on several benchmarks Mukhoti et al. (2021) . Both these methods improve performance and uncertainty estimation through model diversity but incur high computational costs during training and inference. In next section, we describe the second part of the proposed framework -an alternative to k-ensemble for quantifying approximate Epistemic uncertainty in single forward pass.In contrast to multiple feed-forward evaluation models, we propose single-shot UQ estimation using smoothness and sensitivity Lee et al. ([n. d.] ). We first describe the proposed single shot approach with intuitive modifications to training scheme that address the aforementioned problems.The uncertainty estimation flow is shown in Figure 2 (a). A centroid vector Z ∈ R m is initialised randomly and assigned to each label class in a set of classes J. Let X t (i) ∈ R d be the set of audio encodings of a mini-batch during training. A distance transformation matrix W j (m, d) is initialised using a Gaussian prior per class, where d is the feature encoding dimension and m < d is the size of the centroid vector. Weight matrix W j acts as a learnable linear dimensionality reduction on feature vectors, enabling a compact representation for distance computation Ren et al. (2021) ; Venkataramanan et al. (2023) . The class-dependent nature of W j enables class separation in latent space and is crucial for minimising the likelihood of feature collapse.A weighted feature distance D j between the model output and centroids is computed as:D j (X t , Z j ) = ||W j X t -Z j || 2 2mσ 2 jwhere length scale σ j is a trainable parameter and acts as class dependent normalising hyperparameter.If the matrix W is assumed to be Identity Matrix the above formulation computes Mahalanobis distance (MD) from the centroids. The learnable nature of W acts as an adaptive dimensionality reduction on the latent space X and the output W X can be expected to represent global distributions as well as class dependent local distributions.During the forward pass, a class label for each sample is given by softmax of distance scoresy i = ArgminZ j (X i )as the maximum correlation (minimum distance) between data point X i and class centroids Z j . For the UQ estimate, the set of Mahalanobis distances is normalised through the division of maximum class distance. The model uncertainty is given by mixture of the Gaussian models fitted at each class centroid d UQ = jN (D j |z j , σ j ).The class centroids, Z j , are updated for every mini-batch of training using an exponential moving average of the feature vectors of data points corresponding to class j:Z t+1,j = γZ t,j + 1 n j (1 -γ) i (W j X i )where n j is number of samples in the j th class, and γ is a hyper-parameter similar to momentum gradient descent. After each update, the class vectors are normalised such that ||Z j || 2 = 1.Class dependent triplet Loss formulation is used to maximise the distance between distinct class centroids and minimise intra-class separation, following Kumar et al. (2020) ; Hermans et al. (2017) . Audio embeddings obtained from the encoder network were used as an anchor point X a . Let Z a be the centroid vector of the class corresponding to true label y a , while Z j indicates remaining centroid vectors such that {j ∈ J∀j ̸ = a}, The loss with margin ϵ ∈ (0.1 -0.5) is given byL triplet = a,j max (||W X a -Z a || -||W X a -Z j || + ϵ, 0)During the training process, this loss is averaged over a mini-batch of data points, the class centroids are updated to new locations as per predicted labels and stochastic gradient descent (SGD) is performed for θ and W j . Audio encoder output latent vectors usually have high dimensions and the above loss may suffer poorly due to involved distance computation. Low rank nature of W ensures that distance computation in above loss function is sensible.Feature Regularisation High dimensional feature space embedding suffer from feature collapse and feature redundancy in latent space which can adversely affect uncertainty prediction Liu et al. 2020)L 1 * ||x 1 -x 2 || X ≤ ||f W (x 1 ) -f W (x 2 )|| H ≤ L 2 * ||x 1 -x 2 || X This ensures the mapping ||f W (x 1 ) -f W (x 2)|| H has meaningful correspondence in input space with respect to a well defined distance measure ||x 1 -x 2 || X Liu et al. (2020) . This condition also ensures smoothness in latent space such that the audio embeddings are not too sensitive to small variations in input.We 3 ExperimentsWe will now demonstrate the utility of proposed framework in quantifying uncertainties of audio driven disease diagnosis. We first start with a brief description of datasets, evaluation criterion and implementation details. (detailed description and data histograms are covered in Appendix) 
datasets: 
 We conduct extensive experiments using two popular audio-driven healthcare diagnosis datasets.The Self-supervised learning (SSL) is an attractive approach for healthcare audio datasets where the data size is limited and manual annotation is expensive Sharma et al. (2020) ; Rocha et al. (2018) .Three different SSL models are employed as audio encoders for the empirical evaluation. Let a(t) ∈ A be an input audio waveform and y = 1, . . . J be its corresponding label. The feature encoder gives embedding vectors X w (a) ∈ R d , where d = 256 is the fixed latent dimension.All audio files were resampled to a fixed rate of 22.05kHz. The ICBHI respiratory sounds were cropped/padded to max a length of 7s Gairola et al. (2021) ; Kulkarni et al. (2023) , while COSWARA speech were fixed to 10s length Sharma et al. (2020) . In the case of ResNet, each audio was transformed to log Mel-spectrogram using 128 frequency bins. An input size of (128, 350) was used for ICBHI, whereas, for COSWARA, the input size was (128, 500). For both cases, the dataset was divided into three non-overlapping portions such that the test set (20%) and validation set (20%) contained audio records from different patients than that of the train set (60%).For measuring accuracy of model, sensitivity ( T P T P +F N ), and specificity (  
results and discussion: 
 The first goal of the experiments is to answer the question 'whether model uncertainty score follows model accuracy'.  
uncertainty visualisation and decision making: 
 The outputs produced by sampling from Dirichlet distribution (output of probabilistic classifier for a single input) satisfy the property that j (p j ) = 1, where p j is probability P [y = j|X]. For a three class problem (ICBHI -wheeze, crackle, healthy) , each of these samples fall on the 2D plane defined by j (p j ) = 1. The UQ model performs well on unseen datasets, as seen from results on non-intersecting inter-patient data splits, and is equally applicable to more general datasets. Results also show the effectiveness and applicability of the Mahalanobis distance-based metric for different general-purpose audio encoders.Finally, the proposed framework enables fast and lightweight UQ estimation, making it more suitable for implementation in mobile and IoT devices for continuous health monitoring owing to its small size and lower number of trainable parameters.We consider the probabilistic function learning problem between input audio space X ∈ X and The training dataset D n = {(x i , y i )∀i = 1 . . . N } (a subset of joint space µ XY ) is used to train an estimator Ŷ = g(X; W ), where g denotes a neural network with parameter W . Further, the collected dataset itself can be inherently noisy or error-prone, which results in data uncertainty (also known as Aleatoric uncertainty). The noise in the dataset is indicated by x i = x + η i and y i = ŷ + ϵ j where the the observed noisy dataset is given by, D n = {(x i , y i )}. This aleatoric uncertainty is irreducible and can only be estimated as expected variance in the output for a fixed input X, and a given estimator f (x|W ). However is not the only source of uncertainty in the estimator, the output variance does not capture the uncertainty in estimator or the learning process itself.The total predictive probability, can be expanded as follows -EQUATIONThe The inference integral is often approximated using multiple forward pass via Dropout or Ensemble modelling. The goal of the proposed distance based model is to provide an efficient single forward pass alternative to approximate the marginal and inference integrals.Given that we can view an ensemble member as a single deterministic model and vice versa, this provides an intuitive explanation for why single deterministic models report inconsistent and widely varying predictive entropies and confidence scores for OoD samples for which a Deep Ensemble would report high epistemic uncertainty (expected information gain) and high predictive entropy.Assuming that p(y|x, ω) only depends on p(y|x) and I[Y ; w|x], we model the distribution of p(y|x, ω)(as a function of ω) using a Dirichlet distribution Dir(α) which satisfies:EQUATIONThen, we can model the softmax distribution using a random variable p ∼ Dir(α) as:EQUATIONThe E p(p|α) H[Y | p] = ψ (α 0 + 1) - K y=1 α i α 0 ψ (α i + 1)Proof. Applying the sum rule of expectations and 3 from 1.1 we can writeEH[Y | p] = E - K i=1 p i log p i = - i E [p i log p i ] = - i α i α 0 (ψ (α i + 1i -ψ (α 0 + 1))The result follows after rearranging and making use of i αi α0 = 1. introduces a more sophisticated approach by masking portions of the latent space rather than the raw audio. Wav2Vec 2.0 significantly improves upon the quality of learned representations and demonstrates exceptional performance in downstream speech tasks. We use Wav2vec2.0 as one of the backbone feature extractor in the proposed framework. The Wav2vec model predicts the masked latent representations, encouraging it to capture rich contextual information. The output embedding dimension of the Wev2vec encoder is fixed to be 256.Problem Agnostic Speech Encoder (PASE) is another self supervised audio feature encoder which employs multiple neural networks, termed "workers," to tackle various self-supervised tasks. These workers contribute to learning rich and discriminative representations. To ensure robust feature vectors with respect to small variations in input audio, PASE+ introduces an online speech distortion module that artificially corrupts the input audio, forcing the encoder to learn more invariant features.As shown in Figure 5 PASE+ also uses bidirectional attention layers to combine convolution ouputs to better capture both short-term and long-term speech dynamics.An image-based ResNet-50 is used as the backbone with a residual block of two 3 × 3 convolution layers and a skip connection between each block. The network is trained on the self-supervised task of spectral feature prediction and reconstruction of the log-Mel spectrogram. The network consists of a series of convolution layers. Each of these layers is defined with 64 channels, kernel strides (5, 2, 2, 2, 2, 2, 2), and kernel widths (7, 3, 3, 3, 3, 3, 2, 2), respectively, followed by batch normalization and ReLU activation. The interval between two sequential samples in the feature encoder output Z is 15ms, and the receptive audio field is 20 ms. The output from convolution layers is concatenated and passed to a multi-head attention layer and a fully connected layer with an embedding size of 256.Like PASE Ravanelli et al. (2020) , the final linear layer is used to predict speech features such as log (2020). These representations are then frozen to compute encoding for respiratory cycle datasets.Experiments show that no significant improvement are observed with additional complete finetuning on the ICBHI dataset during the training phase compared to the frozen representation.We train the classification using unweighted combination of negative log likelihood term L N LL and a KL-divergence term, following the Sensoy et al. (2018) ; Bachstein et al. (2019) . Appendix covers Loss function derivations and final expressions.The loss function expressions of L N LL and L KL are respectivelyEQUATIONEQUATION)in which αc = y c + (1 -y c )α c and ψ(•) is Digamma function.These two losses can viewed intuitively as a union of Bayes Risk Approximation losses, which is defined with respect to class conditional density prediction. We use Bayes risk formulation from PAC learning nomenclature as given below,EQUATIONDuring the forward pass, a class label for each sample is given by softmax of distance scores y i = ArgminZ j (X i ) as the maximum correlation (minimum distance) between data point X i and class centroids Z j . For the UQ estimate, the set of Mahalanobis distances is normalised through the division of maximum class distance. The model uncertainty is given by mixture of the Gaussian models fitted at each class centroid d UQ = jN (D j |z j , σ j ).The class centroids, Z j , are updated for every mini-batch of training using an exponential moving average of the feature vectors of data points corresponding to class j:Z t+1,j = γZ t,j + 1 n j (1 -γ) i (W j X i )where n j is number of samples in the j th class, and γ is a hyper-parameter similar to momentum gradient descent. After each update, the class vectors are normalised such that ||Z j || 2 = 1.Class dependent triplet Loss formulation is used to maximise the distance between distinct class centroids and minimise intra-class separation, following Kumar et al. (2020) ; Hermans et al. (2017) . Audio embeddings obtained from the encoder network were used as an anchor point X a . Let Z a be the centroid vector of the class corresponding to true label y a , while Z j indicates remaining centroid vectors such that {j ∈ J∀j ̸ = a}, The loss with margin ϵ ∈ (0.1 -0.5) is given byL triplet = a,j max (||W X a -Z a || -||W X a -Z j || + ϵ, 0)During the training process, this loss is averaged over a mini-batch of data points, the class centroids are updated to new locations as per predicted labels and stochastic gradient descent (SGD) is performed for θ and W j .For measuring accuracy of model, sensitivity ( T P T P +F N ), and specificity ( F P F P +T N ) scores were used. Each score measures class-wise prediction accuracy in the case of the unbalanced dataset. The notations T N, F N denote true and false negative rates and T P, F P denote true and false positive rates, respectively. Average of these two scores ( SP +SN 2) was used for comparison with SoTA models Rocha et al. (2018) . The area under the receiver operating curve (AUROC) was used as an indicative probability of correctly classifying a randomly selected unseen sample. ECE = 10 k=1 n k n |ascore(B k ) -uscore(B k )|where n k is number of samples in k th bin, ascore and uscore are average accuracy and uncertainty estimates for each bin B k . Experiments show that calibration error drastically reduces with the addition of UQ models while maintaining the model accuracy in AUROC scores. It can be interpreted as the probability that a positive example (in-distribution) will have a higher detection score than a negative example (out-of-distribution).We conduct extensive experiments using two popular audio-driven healthcare diagnosis datasets. for ICBHI, whereas, for COSWARA, the input size was (128, 500). For both cases, the dataset was divided into three non-overlapping portions such that the test set (20%) and validation set (20%) contained audio records from different patients than that of the train set (60%).1 In this second stage we show that, using off-the-shelf encoders it is possible to achieve and state of the art performance on popular disease diagnosis task.In the proposed uncertainty quantification framework weighted feature distance D j between the model output and centroids is computed as:D j (X t , Z j ) = ||W j X t -Z j || 2 2mσ 2 jwhere length scale σ j is a trainable parameter and acts as class dependent normalising hyperparameter.If the matrix W is assumed to be Identity Matrix the above formulation computes Mahalanobis distance (MD) from the centroids. The learnable nature of W acts as an adaptive dimensionality reduction on the latent space X and the output W X can be expected to represent global distributions as well as class dependent local distributions.Figure 8 shows comparison of uncertainty estimates obtained using distance based metric and ensemble based model. In contrast to multiple feed forward evaluation models, a single shot estimation of distance function in feature space gives an approximation of class conditional density.A Mahalanobis distance metric Lee et al. ([n. d.] ); Venkataramanan et al. (2023) between output and the class centroids is has already been shown to act as an approximation of class conditional density and outperform empirical ensemble models for the task of OOD detection. In medical diagnosis, a nonparametric model for UQ has been used as a noise metric for Diffusion MRI. Statistical UQ using ensemble models has also been used to augment clinical decision support in medicine Kang et al. (2021) . These models are resource-intensive due to multiple training runs to form an ensemble and/or several feed-forward evaluations for a single inference. However, the explainability via uncertainty quantification of speech and audio-driven disease classification remains under-explored and is paramount for system reliability and patient safety Xia et al. (2021) . In this work we treat model prediction and uncertainty quantification as independent tasks. We emphasise quantification of both reducible epistemic and aleatoric (irreducible) uncertainties using a single inference model with minimal modifications to backbone architecture.The proposed framework introduced a new way to measure how confident the model is in its predictions. This is called model uncertainty. Instead of just giving a single answer, the model provides a range of possible outcomes along with how likely each outcome is. This helps in making more reliable decisions, especially in medical applications.When the model is very uncertain about its prediction, it can be considered as a "flag" to say, "I'm not sure about this." In this case, the system can ask for more information, like re-recording the audio, before making a final decision. This helps prevent incorrect diagnoses.The framework was tested on two well-known datasets: COSWARA and ICBHI. It performed better than other popular methods, especially in detecting respiratory diseases. The model was able to correctly identify diseases more often and was less likely to make mistakes. One of the strengths of this framework is its ability to handle different types of audio data. It can work well with various In summary, the model offers a significant improvement in audio-based disease detection. It is more accurate, reliable, and practical than existing methods. By considering the model's uncertainty, it helps to reduce the risk of incorrect diagnoses and improve patient safety.Finally, Our results show that: there is a necessity of quantification and identification of prediction uncertainties in deep learning models for audio-driven disease estimation. Further it is necessary to distinguish between learning aleatoric and epistemic uncertainty, which is unexpected and violates assumptions on simple distribution based uncertainty quantification methods. We expect that our formulation and results help practitioners and researchers choose uncertainty methods and expand the use of disentangled uncertainties, as well as motivate additional research into this topic.Answer: [No] Justification: We use open source healthcare datasets and provide appropriate links for data download, no data is released along with this paper. Further, we provide step-by-step guidance to reproduce experiments conducted in this study.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Supplementary material further describes validation splits in details.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We use 10-fold CV for reporting all numerical figures in the results• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Covered in Supplementary Material Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification:Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification:Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.• The paper should discuss whether and how consent was obtained from people whose asset is used.• At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.• Justification:Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.Processing Systems (BDU at NeurIPS 2024). Do not distribute.Training and validation set labels shared as supplementary material. 
 
 Appendix:  



Reviews: review: title: title: Uncertainty Quantification and Calibration for Audio-driven Disease Diagnosis
main_review: main_review: This paper proposes a novel framework for incorporating uncertainty quantification (UQ) into audio-driven disease diagnosis models.

Pros:

Addresses a crucial limitation: Deep learning models often lack the ability to quantify confidence in predictions, leading to overconfident and erroneous diagnoses. This work proposes a method to address this by estimating model and data uncertainties.
Unified framework: The proposed approach integrates UQ with audio-driven disease detection using a Dirichlet density approximation and independent kernel distance learning.
Improved model reliability: The uncertainty-aware model produces confidence scores that closely match accuracy values, increasing overall model reliability.
Minimal modifications: The proposed method requires minimal modifications to existing audio encoder architectures and is computationally efficient compared to ensemble models.
Generalizability: Evaluations on large public respiratory disease datasets demonstrate the framework's generalizability and efficiency.

Cons:

Limited experimental details: While the paper mentions using three self-supervised audio encoders (ResNet-50, Wav2Vec, PASE), it lacks specifics on hyperparameter tuning, training details, and validation strategies.
Comparison with existing UQ methods: The paper mentions limitations of existing UQ methods like dropout and ensemble models but doesn't explicitly compare the proposed method's performance with these approaches.

Originality:

The combination of a probabilistic classifier with a learnable distance metric for UQ in audio-driven disease diagnosis appears to be novel.
The focus on single-shot UQ estimation with minimal alterations to existing encoder architectures offers a potentially efficient approach.


Significance:

This work contributes to the development of more reliable and interpretable AI models for medical diagnosis.
By enabling quantification of uncertainties, the proposed framework can improve decision-making in healthcare settings by highlighting areas of low model confidence.
The emphasis on computational efficiency further enhances the framework's potential for real-world applications.
title: title: Great Work (Length Concern (9 Pages vs. 4-Page Abstract)
main_review: main_review: **Major Concern (Note to AC/PC):** Paper consists of 9pgs of content instead of 4-page extended abstract.

### Pros
- This paper is one of the first work addressing UQ in audio classification.
- It proposes a novel framework for UQ in audio-driven disease detection, it uses a Probabilistic Classifier and a Single Inference method to measure uncertainty which utilizes latent feature maps created by the encoder to depict the class-conditional distribution.
- The authors validate their methodology on two medical audio detection datasets: ICBHI(4 class) and COSWARA(2 class) demonstrating a significant decrease in calibration error with their proposed technique.
- The proposed framework achieves SOTA performance on the ICBHI4 dataset.

### Cons
- **Major:** The primary section spans 9 pages, while the appendix, references, and NeurIPS checklist extend the total to 26 pages.
- The writing can be somewhat difficult to follow due to various formatting issues.(Especially citations with lack of appropriate spacing)

### Summary
This paper could serve as a valuable contribution to the workshop audience, provided it can be condensed to 4 pages.