# Can We Trust AI with Our Ears? A Cross-Domain Comparative Analysis of Explainability in Audio Intelligence

Sudip Chakrabarty, Pappu Bishwas, Mainak Bandyopadhyay, Jérémie Sublime

► To cite this version:

Sudip Chakrabarty, Pappu Bishwas, Mainak Bandyopadhyay, Jérémie Sublime. Can We Trust AI with Our Ears? A Cross-Domain Comparative Analysis of Explainability in Audio Intelligence. IEEE Access, 2025, pp.1-1. 10.1109/ACCESS.2025.3622161. hal-05319084

HAL Id: hal-05319084

https://hal.science/hal-05319084v1

Submitted on 17 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Can We Trust AI with Our Ears? A Cross-Domain Comparative Analysis of Explainability in Audio Intelligence

SUDIP CHAKRABARTY<fnref n="1" />, (Member, IEEE), PAPPU BISHWAS<fnref n="1" />, MAINAK BANDYOPADHYAY<fnref n="1" />, and JÉRÉMIE SUBLIME<fnref n="2" />

Corresponding author: Mainak Bandyopadhyay (e-mail: mainak.bandyopadhyayfcs@kiit.ac.in)

This work was supported in part by Kalinga Institute of Industrial Technology (KIIT), Deemed to be University through a publication grant covering the full open access charges.

**ABSTRACT** The rapid growth of deep learning has led to major successes in audio classification, but the “black box” nature of these models slows down their use in important areas such as healthcare where trust is critical. This paper addresses this problem through a detailed study on the use of Explainable AI (XAI) for audio-based systems. Six distinct datasets are used to evaluate a consistent deep learning model, including speech emotion recognition, environmental sound classification, and healthcare (heart, lung, and cough sounds). The study analyzes both audio-only models and multimodal models that combine audio features (spectrograms and MFCCs) with demographic data. A set of three distinct XAI techniques, namely LIME, SHAP, and Grad-CAM, is used to explain how the models make their predictions. Our key findings show that XAI methods are very useful for checking how the models work, confirming that they learn to focus on clinically and acoustically relevant features in the audio signals. The analysis also shows the power of multimodal explanations, where tools like SHAP and LIME can trace a prediction back to both acoustic patterns and specific demographic data, showing how the model creates a complete, patient-specific picture. The main contribution of this work is a broad, comparative study that provides a more complete understanding of explainability in the audio domain. The results confirm that XAI is an important part of the entire process of building a model, helping with validation, debugging, and the creation of more reliable and human-aligned AI systems for real-world use.

**INDEX TERMS** Audio classification, Explainable AI (XAI), Grad-CAM, LIME, multimodal learning, model interpretability, SHAP, signal processing.

## I. INTRODUCTION

EXPLAINABLE Artificial Intelligence (XAI) has emerged as a key solution to the growing demand for transparency in deep learning, particularly in domains such as audio where model reasoning is difficult to interpret [1]. Deep neural networks have opened up new opportunities for automated analysis, from improving clinical diagnoses in healthcare to capturing subtle nuances in Speech Emotion Recognition (SER) [2] and enabling robust monitoring of environmental soundscapes [3]. These models handle complicated representations like Spectrograms [4] and Mel-Frequency Cepstral Coefficients (MFCCs) [5] with ease. They are particularly good at learning complex, non-linear connections straight from raw or processed audio inputs.

Deep learning models have significantly decreased reliance on expert-driven rule-based systems and handmade features by automatically identifying significant patterns and correlations [6]. But the same data-driven intricacy and architectural depth that give these models their strength also make them extremely opaque. Trust and accountability issues arise because it is difficult to understand why models make specific predictions with millions of interacting elements.

This growing opacity hinders the safe and reliable use of AI in high-stakes real-world scenarios. In healthcare contexts, for example, clinicians face ethical and professional risks if they must act on diagnostic suggestions without understanding the underlying reasoning [7]. In speech emotion recognition, misinterpretations can affect social interactions,

<footnote n="1">¹School of Computer Engineering, Kalinga Institute of Industrial Technology, Deemed to be University, 751024 Bhubaneswar, INDIA (e-mail: sudipchakrabarty6@gmail.com, pappuovi8@gmail.com, mainak.bandyopadhyayfcs@kiit.ac.in)</footnote>

<footnote n="2">² Institut Supérieur d'Électronique de Paris (ISEP), 75006 Paris, FRANCE (e-mail: jeremie.sublime@isep.fr)</footnote>
<table><caption>TABLE 1. List of Abbreviations Used in This Study</caption><thead><tr><td>Abbreviation</td><td>Description</td></tr></thead><tbody><tr><td>AEC</td><td>Audio Event Classification</td></tr><tr><td>AESSD</td><td>Acted Emotional Speech Dynamic Database</td></tr><tr><td>CNN</td><td>Convolutional Neural Network</td></tr><tr><td>DFT</td><td>Discrete Fourier Transform</td></tr><tr><td>DL</td><td>Deep Learning</td></tr><tr><td>DT</td><td>Decision Tree</td></tr><tr><td>GB</td><td>Gradient Boost</td></tr><tr><td>Grad-CAM</td><td>Gradient-weighted Class Activation Mapping</td></tr><tr><td>LIME</td><td>Local Interpretable Model-agnostic Explanations</td></tr><tr><td>LRP</td><td>Layer-wise Relevance Propagation</td></tr><tr><td>LSTM</td><td>Long Short Term Memory</td></tr><tr><td>MELD</td><td>Multimodal EmotionLines Dataset</td></tr><tr><td>MFCC</td><td>Mel-Frequency Cepstral Coefficient</td></tr><tr><td>MKL</td><td>Multiple Kernel Learning</td></tr><tr><td>ML</td><td>Machine Learning</td></tr><tr><td>MLP</td><td>Multi-Layer Perceptron</td></tr><tr><td>PRN</td><td>Pattern Recognition Network</td></tr><tr><td>RF</td><td>Random Forest</td></tr><tr><td>SER</td><td>Speech Emotion Recognition</td></tr><tr><td>SHAP</td><td>SHapley Additive exPlanations</td></tr><tr><td>ST</td><td>Swin Transformer</td></tr><tr><td>ViT</td><td>Vision Transformer</td></tr><tr><td>WCFormer</td><td>Wavelet Convolution Transformer</td></tr><tr><td>XAI</td><td>Explainable AI</td></tr><tr><td>XGB</td><td>XGBoost</td></tr></tbody></table>

user experience, and even mental health interventions. In environmental sound detection [8], misunderstood predictions could lead to missed critical events or false alarms that carry significant implications for public safety. Consequently, stakeholders across these domains increasingly demand clear, interpretable explanations to justify AI decisions. In response, Explainable Artificial Intelligence (XAI) [9] has emerged to meet the growing demand for transparency, aiming to clarify how complex models arrive at their decisions. Pioneering techniques like Local Interpretable Model-agnostic Explanations (LIME) [10] and SHapley Additive exPlanations (SHAP) [11] have demonstrated the potential of generating meaningful, human-understandable rationales for individual predictions. Despite growing interest, most XAI research focuses on images and tabular data, leaving audio-based interpretability underexplored.

Unlike visual or numerical data, audio signals [12] pose unique challenges for explainability. While spectrograms and MFCCs reveal patterns, linking model attention to these features remains difficult. Identifying which time-frequency regions or MFCC coefficients drive decisions is critical for medical [13] and environmental applications [14]. Incorporating demographic metadata further complicates multimodal models [15], making it harder to interpret each input's influence. Without systematic evaluation, high-performing models may lack real-world transparency. To address this, XAI methods [16] are adapted and evaluated across audio tasks and feature types, revealing how spectrograms and MFCCs affect interpretability [17]. This study systematically compares LIME, SHAP, and Grad-CAM [18] on six datasets, including medical, speech emotion, and environmental sounds.

In medical tasks, demographic and audio features are jointly analyzed. Explanations are assessed qualitatively and quantitatively, providing practical insights into each method's strengths and limitations in audio AI. The key contributions of this work are as follows:

* **Unifies Cross-Domain Framework:** This study introduces a novel evaluation framework that systematically applies XAI techniques to a unified audio classification model across diverse domains.

* **Novel Taxonomy Tailored for Audio Explainability:** A structured taxonomy is introduced, specifically designed for audio XAI, extending beyond general-purpose classifications and providing a multidimensional lens to categorize methods.

* **Advancing Multimodal Explainability:** Interpretability in multimodal settings is explored by integrating demographic attributes with audio features, offering detailed analyses of how such heterogeneous inputs affect explanation quality.

* **Rigorous Comparative Study of XAI Techniques:** Through a side-by-side evaluation of LIME, SHAP, and Grad-CAM, the study provides practical insights into their relative strengths and limitations, supported by both qualitative visualizations and quantitative metrics.

* **Feature Representation and Interpretability Insights:** The choice of input representation (e.g., MFCCs vs. spectrograms) influences not only predictive accuracy but also the clarity and fidelity of the generated explanations, yielding actionable guidance for practitioners.

The remainder of this paper is organized as follows. Section II provides a foundational background on Explainable AI, including a review of related works. Section III details the materials and methodology, covering the datasets, feature extraction processes, and model architecture. Section IV presents the experimental results from both model performance and the explainability analysis. Section V addresses the implications of the results, and Section VI provides a summary of the study along with prospects for further investigation.

## II. BACKGROUND

### A. FOUNDATIONS OF EXPLAINABLE AI

Deep learning has proven highly effective for complex recognition tasks [19]. However, these models often act as black boxes, making it difficult to understand why a prediction is made, which creates a “crisis of trust” in high-stakes domains [20]. Explainable AI (XAI) provides tools and methods to make model behavior transparent to human users without reducing predictive performance [21]. By revealing the factors driving model predictions, XAI addresses a key limitation of current state-of-the-art approaches: while these models achieve high accuracy, their lack of interpretability impedes trust and limits safe deployment in critical applications such as healthcare, speech, and safety monitoring. XAI techniques
enable both local explanations, which clarify individual predictions, and global explanations, which summarize general model behavior. Modern approaches use feature attributions, attention maps, surrogate models, counterfactuals, or other human-comprehensible representations to reveal how deep networks process inputs.

## B. THE ROLE OF XAI IN TRUSTWORTHY AUDIO INTELLIGENCE

Artificial intelligence is increasingly applied in sensitive domains such as clinical diagnostics, finance, surveillance, and audio-based monitoring, where transparency is essential for societal trust and regulatory compliance [22]. Despite advances in predictive performance, current State-of-the-Art (SOTA) approaches remain limited by their opacity, making it difficult to assess fairness, reliability, or alignment with human expertise [23]. Explainable AI (XAI) addresses these challenges by enabling users to understand not just what a model predicts but why it arrives at a particular decision. In healthcare, explanations help clinicians verify whether outcomes are consistent with medical knowledge and patient history [24], while in audio-based applications such as pathological sound classification, speech emotion recognition (SER), and environmental event detection, XAI reveals which spectral regions, temporal segments, or demographic attributes contribute to decisions, offering insights that are inaccessible with black-box methods. As summarized in Figure 1, the overarching goals of XAI include building trust, ensuring fairness, strengthening robustness, supporting compliance, enabling model debugging, and facilitating human-AI collaboration through personalized and data-driven explanations. By linking these goals to practical applications, XAI goes beyond traditional SOTA limitations, transforming opaque predictions into actionable knowledge that supports accountability, bias mitigation, and safe deployment in high-stakes environments.

<figure><img src="image_2.png" /><figcaption>**FIGURE 1.** Illustration of the fundamental goals of XAI for enhancing transparency, trust, and accountability in AI systems.</figcaption></figure>

## C. DIMENSIONS OF XAI METHODS

As illustrated in Figure 2, modern Explainable AI (XAI) techniques can be categorized along several key dimensions that define how explanations are generated, presented, and integrated into machine learning pipelines. These include the form of interpretation (attribution-based, rule-based, instance-based, counterfactual/contrastive, or natural-language outputs), integration with the model (built-in, post-hoc, or interactive), algorithm dependency (architecture-specific, black-box, or hybrid) [25], level of insight (local vs. global, including temporal or multi-resolution), and methodological approach (gradient-based, perturbation-based, surrogate-based, or propagation-based). Together, these dimensions, as shown in Figure 2, provide a structured framework for selecting and tailoring XAI methods to diverse data types and stakeholder needs. However, many prior SOTA works in audio explainability focus narrowly on attribution-based or gradient methods, leaving other dimensions underexplored [26].

<figure><img src="image_3.png" /><figcaption>**FIGURE 2.** Key dimensions for categorizing XAI methods.</figcaption></figure>

## D. LITERATURE SURVEY AND RELATED WORKS

Several studies have explored diverse methods to enhance transparency, ranging from inherently interpretable models to post-hoc explanation techniques [27] that aim to bridge the gap between black-box predictions and human comprehension [22]. Sobahi et al. [28] used fractal dimension with Yet Another Mobile Network (YAMNet) and ViT model for precise cough segmentation and reconstructed pure cough signals, achieving high classification accuracy on COUGHVID [29], VIRUFY [30], and COSWARA [31] datasets while addressing model transparency. The authors of the paper [32] proposed a framework using LIME and SHAP to explain
models for detecting heart sound abnormalities. Mahamud et al. [33] developed an explainable DenseNet201 transfer learning model for classifying COVID-19, pneumonia, and tuberculosis from chest X-rays. SHAP, LIME, Grad-CAM, and Grad-CAM++ [34] were used for interpretability and deployed the model in an Android app for healthcare support. Dang et al. [35] proposed an OSW-based 1D feature extraction with deep PRN for SER; however, the datasets are small and imbalanced, limiting generalizability, and overlapping segments slightly increase processing time. Prity et al. [36] used UrbanSound8K dataset to demonstrate explainable urban sound recognition, combining a customized CNN and XAI tools (LIME, SHAP) to clarify model predictions, but the approach may face generalization challenges when applied to more diverse or real-world noisy soundscapes. Shen et al. [37] proposed a hybrid deep learning pipeline (CNN, RNN, MLP, MKL) to detect early Parkinson's disease from voice recordings [38]. SHAP was used to identify influential acoustic features, enhancing trust in AI-driven vocal biomarker diagnosis. Tang et al. [39] used ADReSS-IS2020 to show that Automatic Speech Recognition (ASR)-based linguistic features with SHAP can detect Alzheimer's disease, performing comparably to manual transcription. However, high ASR word error rates (up to 43%) may reduce feature reliability in noisy or diverse speech. The authors [40] used a transformer-based model for multimodal emotion recognition and applied SHAP for transparency. Wang et al. [41] demonstrated interpretable heart sound analysis with WCFormer, embedding wavelet information for trustworthy Cardiovascular Disease (CVD) diagnosis, though practical deployment remains limited. Das et al. [42] used LIME and SHAP for CNN-based bird species recognition, but their results are limited to a single dataset that may contain background noise and imbalanced class distribution, and lack real-world testing. The work [43] introduces a new audio dataset, uses LRP to explain waveform and spectrogram models, and shows audible heatmaps outperform visual explanations in a user study. Sultana et al. [44] demonstrate how optimized multiresolution audio features can improve COVID-19 detection from breathing signals and highlights the value of SHAP explanations. Norval et al. [45] demonstrate how SHAP and LIME help interpret ensemble speech emotion recognition models for underrepresented languages. However, their study is limited by the scarcity of large, diverse emotion-labeled speech datasets for underrepresented languages, which may affect model generalizability. Choi and Lee [46] applied Grad-CAM to interpret a lung disease classifier with light attention, highlighting key spectrogram regions for better clinical understanding. The authors [47] used a VGGish/YAMNet ensemble for SER, featuring novel Gaussian preprocessing and an auditory analysis of Grad-CAM results. However, Grad-CAM explanations may miss subtle temporal patterns in audio and the small dataset limits generalizability. Shokouhmand et al. [48] leveraged SHAP with XGBoost to clarify which spectral and temporal features distinguish adventitious lung sounds, but their reliance on pre-segmented clips may

limit capturing full breathing context. To assess the quality of
explanations, several evaluation metrics [49], [50]. have been
proposed in the XAI literature, including fidelity, stability,
and sparsity, which provide quantitative measures of how well
explanations reflect model behavior and how robust they are
to perturbations.

As the representative works summarized in Table 2 illustrate, significant progress has been made in applying XAI to specific audio domains. While these studies provide valuable insights, they highlight the absence of a unified, comparative analysis evaluating multiple XAI methods across diverse audio domains, particularly for multimodal models incorporating non-audio features like demographic data. This study addresses this gap by offering a comprehensive, cross-domain assessment that advances a deeper and more integrated understanding of explainability in audio analysis.

### III. MATERIALS AND METHODOLOGY

The experimental framework is structured to explicitly handle both unimodal (audio-only) and multimodal (audio with demographic data) scenarios, ensuring a consistent and reproducible pipeline from data preparation to model analysis. The overall workflow is summarized in Figure 3.

#### A. DATASETS AND PREPROCESSING

This study uses six publicly available audio datasets, chosen to represent diverse real-world domains such as clinical healthcare, speech emotion analysis, and environmental sound classification. Table 3 summarizes the essential characteristics of these datasets.

For feature extraction, Mel-frequency cepstral coefficients (MFCCs) with 40 coefficients per sample and Mel spectrograms with 128 Mel bands were used. Spectrograms were resized to 128×128 to ensure consistency across inputs. Standardization and normalization were applied where needed, and audio recordings were preprocessed according to the specifications described in the individual dataset preprocessing subsections, including resampling to compatible sampling rates. Each dataset was handled independently to preserve the integrity of the cross-domain comparison. These settings follow widely adopted audio processing practices.

1) The CirCor DigiScope Phonocardiogram Dataset

This dataset [56] contains pediatric heart sound recordings collected during clinical screenings, with each sample annotated by expert clinicians for murmur presence, timing (systolic/diastolic), and auscultation locations. Figure 4 shows a representative waveform, spectrogram, and MFCC features from a single sample.

**CirCor Data Preprocessing:** A 30-second clip length (90th percentile) was used, with shorter audio zero-padded and longer ones truncated. MFCCs (4 kHz) and Mel spectrograms (1024-sample window, 256 hop, 128 Mel bands, per 4 kHz Nyquist limit [57]) were extracted. Demographic data (age, sex, height, weight, pregnancy_status) were encoded and standardized. The final dataset comprised 942 instances.
<table><caption>TABLE 2. Representative works on explainable AI applied to audio tasks.</caption><thead><tr><td>Ref</td><td>Year</td><td>Audio Domain</td><td>Dataset</td><td>Model Used</td><td>Types of XAI</td><td>Key Contribution</td></tr></thead><tbody><tr><td>[28]</td><td>2022</td><td>Covid-19 Detection</td><td>COUGHVID, COSWARA, VIRUFY</td><td>ViT, YAMNet</td><td>Grad-CAM</td><td>Developed a ViT-based classifier on fractal dimension representations.</td></tr><tr><td>[33]</td><td>2024</td><td>Lung Sound (Pneumonia Detection)</td><td>Lungs Disease Dataset</td><td>DenseNet201, EfficientNetB0, InceptionV3, Lenet</td><td>SHAP, LIME, Grad-CAM, Grad-CAM++</td><td>Pneumonia detection with local and global explanations.</td></tr><tr><td>[35]</td><td>2023</td><td>Speech Emotion Recognition</td><td>Emo-DB, AESSD</td><td>Deep PRN</td><td>SHAP</td><td>Used SHAP for feature selection and model interpretation in SER.</td></tr><tr><td>[36]</td><td>2025</td><td>Environmental Sound</td><td>UrbanSound8K</td><td>ML and DL</td><td>LIME, SHAP</td><td>Applied XAI for transparent urban sound classification.</td></tr><tr><td>[37]</td><td>2025</td><td>Voice of Parkinson's disease Patient</td><td>Parkinson's Disease Dataset</td><td>Hybrid DL (CNN, RNN, MLP, MKL)</td><td>SHAP</td><td>Voice-based early PD detection with SHAP explainability.</td></tr><tr><td>[39]</td><td>2023</td><td>Alzheimer's Disease Detection</td><td>ADReSS-IS2020</td><td>Ensemble learning classifier</td><td>SHAP</td><td>Automated ASR-based linguistic feature extraction for end-to-end AD detection.</td></tr><tr><td>[40]</td><td>2025</td><td>Speech Emotion Recognition</td><td>MELD</td><td>Transformer (BERT, wav2vec 2.0)</td><td>SHAP</td><td>Multimodal emotion recognition explanation using SHAP.</td></tr><tr><td>[42]</td><td>2024</td><td>Bird sound recognition</td><td>Custom</td><td>CNN</td><td>LIME, SHAP</td><td>Shows SHAP better than LIME for stability, identity, and separability.</td></tr><tr><td>[51]</td><td>2023</td><td>Audio Event Classification</td><td>UrbanSound8k</td><td>1DCNN, YAMNet</td><td>LRP, DFT-LRP</td><td>Compared AEC model strategies for raw vs. spectrogram inputs using LRP relevance heatmaps.</td></tr><tr><td>[43]</td><td>2024</td><td>Speech (Spoken digits)</td><td>Audio MNIST</td><td>CNN</td><td>LRP</td><td>LRP-based explanations for waveform and spectrogram models.</td></tr><tr><td>[44]</td><td>2025</td><td>Breathing Audio (COVID-19)</td><td>Coswara</td><td>DT, RF, GB, XGB</td><td>SHAP</td><td>Ensemble for COVID-19 detection from breathing audio.</td></tr><tr><td>[45]</td><td>2025</td><td>Speech Emotion Recognition (Afrikaans)</td><td>Afrikaans speech corpus</td><td>CNN, CLSTM, XGBoost</td><td>SHAP, LIME</td><td>Integrates SHAP and LIME to explain speech emotion model decisions.</td></tr><tr><td>[46]</td><td>2023</td><td>Respiratory Sounds</td><td>Custom dataset (Littmann 3200 stethoscope)</td><td>Improved VGGish with Light Attention and ECA-Net</td><td>Grad-CAM</td><td>Shows how attention and Grad-CAM improve interpretability for lung sounds.</td></tr><tr><td>[48]</td><td>2024</td><td>Respiratory sounds</td><td>ICBHI 2017</td><td>XGB classifier</td><td>SHAP</td><td>Explained key spectral and temporal features for lung sound detection.</td></tr><tr><td>[52]</td><td>2021</td><td>Cardiac Signals (ECG)</td><td>CPSC2018</td><td>1D CNNs</td><td>SHAP</td><td>Diagnoses cardiac arrhythmias from 12-lead ECGs with SHAP-based interpretability.</td></tr><tr><td>[53]</td><td>2024</td><td>Heart Sound Abnormality Detection</td><td>PhysioNet Heart Sound</td><td>RF, GBoost, XGBoost</td><td>SHAP, LIME</td><td>Interpretable ML model for cardiac diagnosis.</td></tr><tr><td>[54]</td><td>2024</td><td>Speech emotion recognition (SER)</td><td>TESS, EMO-DB, RAVDESS, SAVEE</td><td>ET Classifier</td><td>SHAP</td><td>Used a SHAP-based feedback loop for explainable feature boosting.</td></tr><tr><td>[55]</td><td>2025</td><td>Music Genre Classification</td><td>GTZAN</td><td>Customized 2D CNN</td><td>LIME</td><td>Explainable music genre classification using CNN and LIME.</td></tr></tbody></table>

## 2) The COUGHVID crowdsourcing dataset

The COUGHVID dataset [29] is a crowdsourced collection of cough recordings. Representative waveform, MFCC, and spectrogram plots of sample cough recordings are shown in Figure 5.

**COUGHVID Data Preprocessing:** The original webm recordings were converted to wav for compatibility. A subset of 13,219 samples was selected due to computational limits and resampled at 4000 Hz. Demographic metadata, including age, gender, respiratory condition, and symptoms such as fever and muscle pain, were collected. The primary classification task focused on three classes: healthy, symptomatic, and COVID-19. Instead of explicitly modeling noise, this study relied on the dataset's natural variation in recording quality,

ensuring that models and explanations were evaluated under realistic recording conditions.

## 3) The ICBHI Respiratory Sound Database

The ICBHI Respiratory Sound Database [58] comprises 5.5 hours of respiratory audio recordings from 126 subjects. The waveform, MFCC, and spectrogram features of ICBHI recordings are exemplified in Figure 6.

**ICBHI Data Preprocessing:** Audio files were standardized to 20 seconds via zero-padding or truncation. From 917 matched files, six diagnosis categories were identified: COPD, Pneumonia, Healthy, URTI, Bronchiectasis, and Bronchiolitis. Mel spectrograms were generated at 10 kHz and resized to 128 × 128. Demographic features included
<figure><img src="image_4.png" /><figcaption>**FIGURE 3.** A flowchart of the experimental methodology. The process branches based on the availability of demographic data, detailing the parallel pipelines for unimodal (right) and multimodal (left) model development. Both pathways conclude with model evaluation and the application of XAI methods for explainability analysis.</figcaption></figure>

Patient ID, Age, Sex, and Adult BMI, with missing BMI computed from height and weight, and other missing entries imputed using column-wise means.

## 4) The South Asian Sound KIIT (SAS-KIIT) Dataset

SAS-KIIT [8] comprises 21 distinct urban sound classes, capturing diverse acoustic environments such as traffic, human activities, and regional events. A representative waveform, MFCC, and spectrogram of an SAS-KIIT audio sample are shown in Figure 7.

**SAS-KIIT Data Preprocessing:** For model training, the SAS-KIIT<fnref n="1" /> dataset used 4-second audio segments resampled to 4000 Hz. Feature maps were standardized by truncating or zero-padding the time dimension. Mel spectrograms were computed with a 1024-point FFT, 128 Mel bands, and a 256-sample hop length, then converted to the decibel scale. No demographic information was available, as this is an environmental sound dataset.

## 5) The Crowd Sourced Emotional Multimodal Actors (CREMA-D) Dataset

CREMA-D [59] is a widely used public resource for speech emotion recognition research [60]. This study utilizes the audio component from its 7,442 recordings, which feature 12 sentences spoken across six core emotional states at multiple intensities. Figure 8 illustrates the feature representations derived from a sample recording.

**CREMA-D Data Preprocessing:** For this unimodal dataset, MFCC features were generated using an FFT window of 2048, and a hop length of 275. For image-based models, Mel spectrograms were generated using 128 Mel bands, an FFT window of 2048, and a hop length of 512, before being resized to a final image dimension of 128×128 pixels. The dataset contains six emotion classes: Disgust, Angry, Fear, Happy, Neutral, and Sad.

## 6) The Audio MNIST Dataset

The Audio MNIST dataset [61] is a benchmark resource specifically created for audio classification, comprising spectrograms of 4000mel.

<footnote n="1">¹SAS-KIIT: https://sas-kiit.netlift.app/</footnote>
<table><caption>TABLE 3. Comprehensive Overview of the Publicly Available Datasets Employed in This Research</caption><thead><tr><th>Dataset Name</th><th>Domain / Task</th><th>No. of Classes</th><th>No. of Samples</th><th>Demographic Info</th><th>Duration (sec)</th><th>Key Characteristics</th></tr></thead><tbody><tr><td>CIRCOR</td><td>Healthcare / Heart Sound</td><td>2</td><td>942</td><td>Present</td><td>30</td><td>Pediatric heart sounds, expert murmur annotations, rich demographic metadata, clinical-grade recordings</td></tr><tr><td>COUGHVID</td><td>Healthcare / Cough Sound</td><td>3</td><td>13219</td><td>Present</td><td>10</td><td>Large-scale crowdsourced cough recordings, support multi-class classification</td></tr><tr><td>ICBHI</td><td>Healthcare / Lung Sound</td><td>6</td><td>917</td><td>Present</td><td>Varied</td><td>Cycle-level lung sound annotations, diverse respiratory conditions, includes detailed demographic metadata</td></tr><tr><td>SAS-KIIT</td><td>Environmental Sound/ Classification</td><td>21</td><td>9450</td><td>Not Present</td><td>4</td><td>Diverse South Asian urban audio, segmented and labeled for environmental sound classification</td></tr><tr><td>CREMA-D</td><td>Speech / Emotion Recognition</td><td>6</td><td>7,442</td><td>Not Present</td><td>Varied</td><td>Multimodal emotion dataset with crowd-labeled emotions; diverse speakers, varied vocal intensities.</td></tr><tr><td>Audio MNIST</td><td>Speech / Digit Classification</td><td>10</td><td>30,000</td><td>Not Present</td><td>Varied</td><td>Benchmark dataset of spoken digits, recordings from a set of speakers, designed for audio classification tasks.</td></tr></tbody></table>

<figure><img src="image_5.png" /><figcaption>FIGURE 4. Different feature representations for an audio sample from the CirCor dataset. (a) Waveform, (b) MFCCs, and (c) Mel Spectrogram.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>FIGURE 5. Feature visualizations for a cough sample from the COUGHVID dataset: (a) Waveform, (b) MFCCs, and (c) Mel Spectrogram.</figcaption></figure>

ken digits (0–9) from a diverse group of 60 speakers. Figure 9 provides a visual representation of a sample from this dataset.

**Audio MNIST Data Preprocessing:** For this dataset, MFCC features were extracted using 40 coefficients, a 2048-point FFT window, and a hop length of 275. For image-based models, audio was resampled to 16,000 Hz, and STFTs were computed with a 512-point FFT and 128-sample hop length. Magnitude spectrograms were converted to decibels, normalized, and standardized to 128 time steps.

## B. MODEL ARCHITECTURE

To handle diverse data types, a multimodal deep learning architecture was designed (Figure 10) with two branches, each specialized for a modality. It supports both audio-only and full multimodal analysis and consists of four main components:

an audio feature extractor, a demographic feature extractor, a fusion block, and a unimodal classification head.

* **The Audio Branch (CNN Feature Extractor):** The primary audio feature extractor is a CNN, chosen for its ability to learn hierarchical patterns from spectrograms. It is a deep, VGG-style network [62] with five blocks, each containing a 3×3 Conv2D layer followed by a 2×2 MaxPooling2D layer. Filters grow from 32 to 512, allowing the network to capture increasingly abstract features, while pooling reduces spatial dimensions and adds translation invariance.

* **The Demographic Branch (MLP Feature Extractor):** A Multi-Layer Perceptron (MLP) [63] was designed to process the 1D vector of structured demographic data. This branch has two Dense layers (64 and 128 neurons)
<figure><img src="image_7.png" /><figcaption>FIGURE 6. Feature representations of an ICBHI lung sound from the 'Wheezes' class: (a) waveform, (b) MFCC, and (c) Mel Spectrogram.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>FIGURE 7. Visual representations of an audio sample from the 'Ektara' class in the SAS-KIIT dataset: (a) waveform, (b) MFCCs, (c) Mel Spectrogram.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>FIGURE 8. An illustration of the feature extraction process for a speech sample expressing 'Anger' from the CREMA-D dataset. The figure displays (a) the original time-domain waveform, (b) the resulting MFCC feature matrix, and (c) the corresponding Mel Spectrogram used for model training.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>FIGURE 9. Three different representations of a spoken digit '6' from the AudioMNIST dataset: (a) Waveform, (b) MFCCs, and (c) Mel Spectrogram.</figcaption></figure>

with BatchNormalization and Dropout to stabilize training and prevent overfitting. The MLP transforms demographic inputs into a high-level feature representation.

* **Feature Fusion and Multimodal Classification:** The model uses a late fusion strategy, where feature vectors from the audio and demographic branches ($x_{\text{audio}} \in \mathbb{X}^n$, $x_{\text{demo}} \in \mathbb{X}^m$) are concatenated to form the fused representation $x_{\text{fused}} \in \mathbb{X}^{n+m}$, as expressed in Equation (1):

$$x_{\text{fused}} = [x_{\text{audio}}; x_{\text{demo}}] \quad (1)$$

This fused vector is then passed through a classification head to capture cross-modal interactions. The final class probabilities are computed from the output logits $z$ using

the Softmax function, as shown in Equation (2):

$$P(y = i | x_{\text{fused}}) = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}} \quad (2)$$

* **The Unimodal (Audio-only) Model:** An audio-only (unimodal) model was created for two key reasons: to serve as a performance baseline for the multimodal system, and to function as the primary model when demographic data is unavailable. To ensure a fair comparison, this model uses the same CNN feature extractor as the multimodal audio branch and connects to a classification head with an identical structure.

**Justification of Uniform Architecture:** The same architecture was intentionally employed across all datasets and input
<figure><img src="image_11.png" /><figcaption>FIGURE 10. The proposed multimodal deep learning architecture consists of two parallel branches: a CNN for extracting features from audio spectrograms or MFCCs, and an MLP for processing demographic data. It supports two outputs: a unimodal prediction from audio features and a multimodal prediction after fusing audio and demographic vectors via concatenation.</figcaption></figure>

modalities. This design choice ensures fairness and comparability in evaluating explainability methods, as architectural differences could otherwise confound the interpretation of fidelity, stability, and sparsity. By keeping the backbone constant, any observed variation in performance or explanation quality can be attributed to the XAI method itself rather than dataset-specific or model-specific factors. The focus of this work is on systematically benchmarking XAI techniques under a unified framework, rather than optimizing architectures for individual datasets.

### C. TRAINING DETAILS

The computational experiments for this study were carried out using publicly available cloud services, primarily Google Colab and Kaggle environments equipped with NVIDIA T4 GPUs to accelerate training. All models were implemented in Python 3.12 using TensorFlow v2.18.0 (with Keras API) to ensure fair and reproducible comparisons. The Adam optimizer was employed for all training, accompanied by a consistent set of callbacks. To ensure robustness and reduce variance, all experiments were conducted using a 5-fold cross-validation protocol. For medical datasets (CIRCOR, ICBHI, and CoughVID), the splits were performed at the subject level to prevent data leakage, ensuring that all recordings from the same patient were kept within a single fold. Additionally, a stratified splitting strategy was employed to preserve class distribution across folds. The reported results in Section IV are presented as the average and standard deviation across the five folds. Key hyperparameters, such as learning rate and batch size, are summarized in Table 4.

### D. EVALUATION METRICS

A suite of standard metrics [64] was applied to quantitatively evaluate the models' classification performance. These metrics are derived from the four cardinal outcomes of a

<table><caption>TABLE 4. Training Parameters Used In The Multimodal Models.</caption><thead><tr><td>Parameter</td><td>Values</td></tr></thead><tbody><tr><td>Optimization Algorithm</td><td>Adam</td></tr><tr><td>Learning Rate (Initial)</td><td>0.0001</td></tr><tr><td>Weight Initialization</td><td>Keras Default (Glorot Uniform)</td></tr><tr><td>Batch Size</td><td>32</td></tr><tr><td>Total Training Epochs</td><td>100</td></tr><tr><td>Dropout Rate</td><td>0.5</td></tr><tr><td>Loss Function</td><td>Sparse Categorical Cross-entropy</td></tr><tr><td>Activation (Hidden Layers)</td><td>ReLU</td></tr><tr><td>Activation (Output Layer)</td><td>SoftMax</td></tr><tr><td>Regularization</td><td>None, L2 (0.01) on Dense Layers</td></tr><tr><td>Class Weighting</td><td>None</td></tr><tr><td>Callbacks</td><td>Model Checkpoint, ReduceLROnPlateau</td></tr></tbody></table>

confusion matrix: True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN) [65].

* **Accuracy:** This metric provides a general assessment of the model's overall performance by measuring the proportion of all predictions that were correct. The formula for accuracy is given in Equation (3).

$$ \text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN} \qquad (3) $$

* **Precision:** Also known as the positive predictive value, precision measures the proportion of positive predictions that were actually correct. Precision is calculated as shown in Equation (4).

$$ \text{Precision} = \frac{TP}{TP + FP} \qquad (4) $$

* **Recall:** Also known as the true positive rate, recall indicates how many of the actual positive samples were
correctly classified by the model. Recall is defined in Equation (5).

$$ \text{Recall} = \frac{TP}{TP + FN} \qquad (5) $$

* **F1-score:** This metric is the harmonic mean of Precision and Recall, providing a single, balanced score. The F1-score is expressed by Equation (6).

$$ \text{F1-score} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} \qquad (6) $$

By utilizing these four metrics, a comprehensive and balanced assessment of each model's performance across the different datasets and input modalities was conducted.

## E. EXPLAINABILITY TECHNIQUES

To analyze the decision-making of the trained models, this study employs three post-hoc explainability techniques: LIME, SHAP, and Grad-CAM. These allow a multifaceted examination of the multimodal models, providing local instance-based attributions (LIME, SHAP) and gradient-based visual saliency for CNNs (Grad-CAM) [66]. They reveal which demographic attributes, spectrogram regions, or MFCCs most influence predictions, forming the basis for evaluating model reliability and trustworthiness [67].

### 1) LIME (Local Interpretable Model-Agnostic Explanations)

LIME [10] is a model-agnostic technique that explains individual predictions by approximating complex models locally. It creates perturbed samples around the instance, perturbing both audio (MFCCs or spectrograms) and demographic features for multimodal cases, and only audio features for audio-only cases. As detailed in Algorithm 1, a local surrogate model is trained on these perturbations to explain a prediction. The surrogate model's coefficients serve as the explanation.

LIME aims to find an explanation $\zeta(p)$ that minimizes objective as shown in Equation (7), balancing explanation fidelity and complexity:

$$ \zeta(p) = \underset{h \in H}{\operatorname{argmin}} \mathcal{J}(k, h, \pi_p) + \Psi(h) \qquad (7) $$

where, $h \in H$ is the interpretable model from class $H$.

$\mathcal{J}(k, h, \pi_p)$ is the fidelity loss, measuring how well the interpretable model $h$ approximates the original model $k$ in the local neighborhood $\pi_p$.

$\Psi(h)$ is a complexity penalty applied to the explanation model $h$ to favor simplicity (e.g., fewer features).

**Notation:** In Algorithm 1, $y_i$ denotes the model prediction for a perturbed instance $z_i$; $w_i$ is its proximity weight relative to the original input $x$, and $Z$ is the collection of all perturbed samples with their corresponding predictions and weights.

### 2) SHAP (SHapley Additive exPlanations)

SHAP explains predictions by computing the marginal contribution of each feature [11], grounded in cooperative game

#### Algorithm 1 LIME for Audio-based Models

**Input:**

f: Pre-trained black-box model.

x: Audio instance (e.g., MFCC, spectrogram; may include demographic data).

M: Number of perturbed samples to generate.

$\pi_x$: Proximity kernel for weighting samples.

G: Class of interpretable models (e.g., sparse linear models).

**Output:**

E: Local explanation (feature importance scores).

**Procedure:**

1: $Z \leftarrow \emptyset$

2: for $i \leftarrow 1$ to $M$ do

3: $z_i \leftarrow \text{PerturbFeatures}(x) \quad \text{// e.g., mask time-frequency regions in a spectrogram}$

4: $y_i \leftarrow f(z_i)$

5: $w_i \leftarrow \pi_x(z_i, x)$

6: $Z \leftarrow Z \cup \{(z_i, y_i, w_i)\}$

7: end for

8: $g \leftarrow \text{TrainWeightedSurrogateModel}(G, Z) \quad \text{// Fit a simple model g by minimizing Equation (7)}$

9: $E \leftarrow \text{ExtractExplanation}(g) \quad \text{// The coefficients of g form the explanation}$

10: return E

theory. The Shapley value $\phi_j$ assigns a unique contribution to each feature $j$ in a “game” (model prediction):

$$ \phi_j(f, x) = \sum_{T \subseteq I \setminus \{j\}} \frac{|T|! \cdot (|I| - |T| - 1)!}{|I|!} [f(x_{T \cup \{j\}}) - f(x_T)] \qquad (8) $$

Here, $f$ is the model under explanation, $x$ is the input instance, $I$ is the set of all features, $j \in I$ is the feature being explained, $T$ is a subset of features excluding $j$, $x_T$ represents $x$ restricted to the subset $T$. Exact computation is often infeasible due to exponential complexity. KernelSHAP approximates this by replacing features with baseline values derived from a background dataset reflecting realistic feature distributions, rather than arbitrary zeros, and fits a weighted linear surrogate to estimate contributions (Algorithm 2).

SHAP values follow additive feature attribution, decomposing $f(x)$ into a base value and feature contributions as shown in Equation 9.

$$ f(x) = \phi_0 + \sum_{j=1}^{F} \phi_j(x) \qquad (9) $$

where $\phi_0$ is the base value (the average model prediction over the background dataset) and $\phi_j(x)$ is the SHAP value for feature $j$. This property provides a clear and intuitive framework for interpreting individual predictions.

In Algorithm 2, $M$ denotes the number of sampled coalitions, $p_k$ represents the model prediction for the $k$-th perturbed input, and $Z$ is the set of tuples $\{(z'_k, p_k, w_k)\}$ storing
### Algorithm 2 KernelSHAP for Audio-based Models

**Input:**

f: Pre-trained black-box model.

x: Audio instance (e.g., MFCC, spectrogram; may include demographic data).

Xbg: Background dataset for masking reference (used as realistic baseline values)

M: Number of coalitions to sample.

**Output:**

SHAP values {$\phi_j(x) | j = 1, ..., F$} for F features.

**Procedure:**

1: $Z \leftarrow \emptyset$

2: **for** $k \leftarrow 1$ **to** $M$ **do**

3: &nbsp;&nbsp;&nbsp;&nbsp;Sample $z'_k \in \{0, 1\}^F$ // $1 = \text{keep}, 0 = \text{mask}$

4: &nbsp;&nbsp;&nbsp;&nbsp;$x'_k \leftarrow \text{MaskWithBackground}(x, z'_k, X_{bg})$

5: &nbsp;&nbsp;&nbsp;&nbsp;$p_k \leftarrow f(x'_k)$ // Prediction for perturbed audio

6: &nbsp;&nbsp;&nbsp;&nbsp;$w_k \leftarrow \text{SHAPKernel}(z'_k)$ // Kernel weight as in Equation (8)

7: &nbsp;&nbsp;&nbsp;&nbsp;$Z \leftarrow Z \cup \{(z'_k, p_k, w_k)\}$

8: **end for**

9: &nbsp;&nbsp;&nbsp;&nbsp;Define $g(z') = \phi_0 + \sum_{j=1}^{F} \phi_j z'_j$ // Linear surrogate, see Equation (9)

10: &nbsp;&nbsp;&nbsp;&nbsp;Fit $g$ to minimize weighted loss on $Z$

11: **return** {$\phi_j(x)$} // Explanation of audio feature importance

each coalition, its prediction, and the corresponding kernel weight.

### 3) Grad-CAM (Gradient-weighted Class Activation Mapping)

For CNN-based models trained on spectrograms, Grad-CAM is used to generate visual explanations [18]. This model-specific technique produces a coarse heatmap highlighting the most salient regions influencing a class prediction. The procedural steps for generating a Grad-CAM heatmap are detailed in Algorithm 3.

Grad-CAM involves two main steps. First, neuron importance weights $\omega_m^t$ for each feature map $F^m$ are computed via global average pooling over the gradients of the class score $s^t$ across spatial locations $(p, q)$, as defined in Equation (10).

$$ \omega_m^t = \frac{1}{K} \sum_p \sum_q \frac{\partial s^t}{\partial F_{pq}^m} \quad (10) $$

Here, K is the total count of pixels within the feature map.

Next, the heatmap highlighting class-relevant regions is obtained via a ReLU-applied weighted sum of feature maps:

$$ H_{\text{CAM}}^{t} = \text{ReLU} \left( \sum_{m} \omega_{m}^{t} F^{m} \right) \quad (11) $$

where $H_{\text{CAM}}^t$ denotes the Grad-CAM heatmap for class t, $\omega_m^t$ is the weight of feature map $F^m$ for class t, and $F^m$ is the m-th feature map of the last convolutional layer. This produces a

### Algorithm 3 Grad-CAM for Audio-based Models

**Input:**

f: Trained CNN model for audio classification.

X: Input spectrogram image or MFCC feature map.

t: Target class index to explain.

F: Activation outputs {$F^m$} from a selected convolutional layer.

**Output:**

$H_{\text{CAM}}^t$: Heatmap showing influential time-frequency regions for class t.

**Procedure:**

1: $s^t \leftarrow f_t(X)$ //Compute score (logit) for target class t

2: **for all** feature maps $F^m \in \mathcal{F}$ **do**

3: &nbsp;&nbsp;&nbsp;&nbsp;Compute $\omega_m^t$ using global average pooling on gradients of $s^t$ with respect to $F^m$ // See Equation (10)

4: **end for**

5: &nbsp;&nbsp;&nbsp;&nbsp;Compute the final heatmap $H_{\text{CAM}}^t$ using Equation (11)

6: &nbsp;&nbsp;&nbsp;&nbsp;(Optional) Upsample $H_{\text{CAM}}^t$ to the original resolution of the input spectrogram.

7: **return** $H_{\text{CAM}}^t$

visual explanation highlighting the time-frequency regions in the spectrogram that most influence the model's prediction.

## IV. RESULT ANALYSIS

### A. QUANTITATIVE RESULTS

This section presents the quantitative performance of the proposed deep learning architecture. To evaluate the model's generalizability and effectiveness on both unimodal and multimodal inputs, the same core architecture was trained and tested for each experimental condition. Classification performance was measured using the standard metrics defined in Section III-D: Accuracy, Precision, Recall, and F1-score. Table 5 summarizes the key results, enabling direct comparison of the architecture's efficacy across domains and input modalities.

A primary observation from the results is the model's exceptional performance on datasets characterized by clean, well-defined audio samples, such as SAS-KIIT, Audio MNIST, and CREMA-D, where accuracies consistently exceeded 99% in some cases. This validates the architecture's fundamental capability for audio pattern recognition in controlled conditions. In contrast, the performance on the healthcare-focused datasets such as CIRCOR, ICBHI, and COUGHVID, while still robust, was more modest. This outcome is expected and highlights the significant challenges inherent in analyzing pathological and real-world biomedical audio, including patient-specific variability, background noise, and the subtle, often overlapping nature of pathological sound events.

Investigation of input modalities showed that adding demographic data is context-dependent. For ICBHI, it consistently improved performance, indicating demographic information provides valuable complementary context for complex res-
<table><caption>TABLE 5. Model Performance Metrics Across Datasets using audio-only and multimodal inputs.</caption><thead><tr><th>Dataset</th><th>Input Type</th><th>Precision</th><th>Recall</th><th>F1-score</th><th>Accuracy</th></tr></thead><tbody><tr><td rowspan="4">CIRCOR</td><td>MFCC (Audio-only)</td><td>0.841 ± 0.017</td><td>0.843 ± 0.015</td><td>0.849 ± 0.023</td><td>0.843 ± 0.021</td></tr><tr><td>Spectrogram (Audio-only)</td><td>0.849 ± 0.028</td><td>0.854 ± 0.023</td><td>0.834 ± 0.022</td><td>0.853 ± 0.024</td></tr><tr><td>MFCC + Demographic</td><td>0.826 ± 0.020</td><td>0.833 ± 0.022</td><td>0.840 ± 0.024</td><td>0.840 ± 0.017</td></tr><tr><td>Spectrogram + Demographic</td><td>0.857 ± 0.023</td><td>0.844 ± 0.020</td><td>0.835 ± 0.029</td><td>0.848 ± 0.022</td></tr><tr><td rowspan="4">ICBHI</td><td>MFCC (Audio-only)</td><td>0.855 ± 0.024</td><td>0.881 ± 0.020</td><td>0.859 ± 0.023</td><td>0.888 ± 0.022</td></tr><tr><td>Spectrogram (Audio-only)</td><td>0.897 ± 0.022</td><td>0.792 ± 0.023</td><td>0.822 ± 0.028</td><td>0.872 ± 0.020</td></tr><tr><td>MFCC + Demographic</td><td>0.863 ± 0.023</td><td>0.881 ± 0.022</td><td>0.887 ± 0.024</td><td>0.903 ± 0.029</td></tr><tr><td>Spectrogram + Demographic</td><td>0.904 ± 0.029</td><td>0.831 ± 0.024</td><td>0.857 ± 0.022</td><td>0.885 ± 0.023</td></tr><tr><td rowspan="4">CoughVID</td><td>MFCC (Audio-only)</td><td>0.761 ± 0.022</td><td>0.745 ± 0.020</td><td>0.743 ± 0.024</td><td>0.770 ± 0.023</td></tr><tr><td>Spectrogram (Audio-only)</td><td>0.755 ± 0.023</td><td>0.743 ± 0.029</td><td>0.776 ± 0.022</td><td>0.782 ± 0.020</td></tr><tr><td>MFCC + Demographic</td><td>0.739 ± 0.020</td><td>0.742 ± 0.022</td><td>0.732 ± 0.024</td><td>0.764 ± 0.022</td></tr><tr><td>Spectrogram + Demographic</td><td>0.761 ± 0.024</td><td>0.757 ± 0.022</td><td>0.765 ± 0.020</td><td>0.778 ± 0.029</td></tr><tr><td rowspan="2">SAS-KIIT</td><td>MFCC</td><td>0.995 ± 0.015</td><td>0.993 ± 0.017</td><td>0.988 ± 0.020</td><td>0.998 ± 0.001</td></tr><tr><td>Spectrogram</td><td>0.991 ± 0.022</td><td>0.984 ± 0.023</td><td>0.990 ± 0.021</td><td>0.996 ± 0.004</td></tr><tr><td rowspan="2">CREMA-D</td><td>MFCC</td><td>0.952 ± 0.023</td><td>0.951 ± 0.022</td><td>0.951 ± 0.024</td><td>0.952 ± 0.020</td></tr><tr><td>Spectrogram</td><td>0.933 ± 0.020</td><td>0.931 ± 0.023</td><td>0.937 ± 0.022</td><td>0.933 ± 0.021</td></tr><tr><td rowspan="2">Audio MNIST</td><td>MFCC</td><td>0.997 ± 0.022</td><td>0.996 ± 0.023</td><td>0.996 ± 0.024</td><td>0.996 ± 0.020</td></tr><tr><td>Spectrogram</td><td>0.996 ± 0.024</td><td>0.996 ± 0.022</td><td>0.994 ± 0.020</td><td>0.998 ± 0.001</td></tr></tbody></table>

piratory analysis. For CIRCOR and COUGHVID, no clear gain was observed, suggesting audio features alone were sufficiently discriminative. Comparison of MFCCs and spectrograms also showed task-dependent performance: spectrograms were slightly better for CIRCOR, while MFCCs achieved higher accuracy for ICBHI and CREMA-D, highlighting the need for empirical evaluation when designing models. The multi-class nature of healthcare datasets like ICBHI adds complexity, lowering accuracy.

To provide a more in-depth analysis of the model's training behavior and classification performance, the training history, confusion matrices, and ROC curves were visualized. As presenting these detailed visualizations for every model across all six datasets would be impractical due to space constraints. Figure 11 shows a typical set of performance graphs for different models, illustrating the standard of convergence and evaluation achieved in this study.

## B. EXPLAINABLE AI (XAI) VISUALIZATIONS

### 1) Explainability on CirCor Dataset

A comprehensive explainability analysis is presented for a representative heart sound from the CIRCOR dataset, in which the multimodal model accurately identified the sample as 'Absent' (i.e., no murmur detected), as illustrated in Figures 12 and 13. These visualizations, based on MFCC and spectrogram inputs, integrate local and global interpretability methods (LIME, SHAP, and Grad-CAM) to highlight critical acoustic features influencing the prediction.

**(a) MFCC-based Visualizations:** The LIME image explanation applied to the MFCC feature map (Figure 12(a1)) highlights in green the specific time-frequency regions that supported the 'Absent' classification, while red areas pushed

against it. The feature importance plot (Figure 12(a2)), which shows that demographic attributes such as the patient's Height and Weight were also key factors supporting this outcome, providing clinicians a dual verification of model focus on key sounds and patient traits. The SHAP attribution map on the MFCC matrix (Figure 12(b3)) provides a highly granular view of each coefficient's influence, while the feature importance plot (Figure 12(b4)) precisely assesses the extent and orientation of each demographic feature's impact on the prediction. For this prediction, Height is shown to have the largest impact pushing the prediction towards 'Present', while Age provides a counteracting influence. Grad-CAM analysis (Figure 12(c)) visually explains the CNN branch by overlaying activations on MFCCs, outlining attentional hotspots that link model focus to specific heart sound events for clinical validation.

**(b) Spectrogram-based Visualizations:** The LIME Image Explanation (Figure 13(a1)) segments the spectrogram into superpixels and highlights in green the regions that provided the strongest evidence supporting the 'Absent' prediction. Supporting this, the LIME Feature Importance Plot (Figure 13(a2)), shows that for this instance, the patient's Age and Pregnancy status were the most significant demographic contributors to the prediction. For the same prediction, the SHAP Force Plot (Figure 13(b1)) illustrates how each demographic feature influenced the output, while the SHAP Attribution Map (Figure 13(b2), 'Absent' panel) visualizes pixel-level contributions. The Global Feature Importance Plot (Figure 13(b3)) shows average spectrogram-pixel SHAP values for the 'Present' class, and the Decision Plot (Figure 13(b4)) depicts cumulative demographic influence across five samples. Figure 13(c1) presents a Grad-CAM overlay, highlighting
<figure><img src="image_12.png" /><figcaption>**FIGURE 11.** Representative model performance visualizations across various datasets and input modalities. (a) Multi-class ROC curve for the multimodal (Spectrogram + Demographic) model on the ICBHI dataset. (b) SAS-KIIT dataset results for the MFCC-based model shown as a confusion matrix. (c) Accuracy and loss trends during training and validation for the spectrogram-based model on SAS-KIIT. (d) Training and validation accuracy curve for the MFCC-based model on the CoughVID dataset. (e) Accuracy and loss curves illustrating the training and validation of the spectrogram-based model on Audio MNIST. (f) Training and validation accuracy curve for the spectrogram-based model on the CIRCOR dataset. (g) Evaluation of the MFCC-based model on CREMA-D represented via a confusion matrix. (h) Audio MNIST dataset classification outcomes of the MFCC-based model represented in a confusion matrix.</figcaption></figure>
<figure><img src="image_13.png" /><figcaption>**FIGURE 12.** Deconstructing the multimodal model's reasoning for an 'Absent' murmur classification on the CIRCOR dataset. The figure presents: (a1, a2) LIME's local approximation of influential MFCC regions and demographic features; (b1-b4) a suite of SHAP visualizations including a global heatmap, local feature importance, and an MFCC attribution map; and (c) Grad-CAM heatmaps with contours localizing the CNN's attentional focus.</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>**FIGURE 13.** A detailed breakdown of an 'Absent' murmur prediction from the CIRCOR spectrogram model. The figure includes local explanations from LIME (a1, a2), a comprehensive set of local and global SHAP analyses for both modalities (b1-b4), and a Grad-CAM heatmap visualizing the CNN's attention (c1).</figcaption></figure>
<figure><img src="image_15.png" /><figcaption>FIGURE 14. Multifaceted explainability analysis for a 'COPD' classification from the ICBHI dataset using the MFCC + Demographic model. (a) LIME feature importance plot for the local prediction. (b) Global SHAP analysis showing a beeswarm plot of feature impacts and an aggregated feature importance ranking. (c) Grad-CAM visualizations as both a 2D heatmap and a 3D surface plot, highlighting salient regions in the MFCC input.</figcaption></figure>

time-frequency regions most influential to the CNN's decision, allowing verification that the model focuses on clinically relevant heart sound cycles rather than noise.

## 2) Explainability on ICBHI Dataset

Interpretability results for a lung sound sample from the ICBHI dataset, which the multimodal model accurately classified as 'COPD' with a confidence score of 0.99, are shown in Figures 14 and 15.

**(a) MFCC-based Visualizations:** The LIME analysis, shown in Figures 14(a1) and (a2), offers an intuitive local explanation for this specific 'COPD' prediction. The LIME Feature Importance Plot (a2) highlights the top MFCC coefficients that influenced this decision. The final feature vector is constructed by concatenating the feature vectors, $C_t$, extracted from each of the T sequential time frames. The concatenated feature vector is constructed as described in Equation (12).

$$X_{concat} = [C_1, C_2, \dots, C_T] \quad (12)$$

As shown in Equation (13), stacking G coefficients across T sequential time frames results in a total feature dimension D equal to their product.

$$D = T \times G \quad (13)$$

Thus, any individual feature, such as $mfcc_x$, represents a specific coefficient from one of the T time frames. The green bars indicate features that provided evidence supporting the 'COPD' classification, while the red bars show features that pushed against it.

The SHAP Beeswarm Plot (Figure 14(b1)) shows feature importance and how feature values influence the model, with color indicating value (red = high, blue = low) and x-position showing impact. The Global Feature Importance Plot (Figure 14(b2)) aggregates these effects, ranking features by mean absolute SHAP values, highlighting specific MFCC coefficients (e.g., $mfcc_{20}$, 0, 21) as most influential. The 2D Grad-CAM Heatmap (Figure 14(c1)) overlays color on the MFCC matrix, with hotter regions indicating higher importance. The 3D Grad-CAM Surface Plot (Figure 14(c2)) visualizes these "attentional hotspots" as peaks, allowing clinicians to quickly identify the MFCC coefficients and time frames most salient for the 'COPD' diagnosis.

**(b) Spectrogram-based Visualizations:** The LIME Image Explanation (Figure 15(a1)) partitions the spectrogram into superpixels and highlights the specific segments that were most influential for the final prediction. The Grad-CAM Heatmap (Figure 15(c1)), which provides a smoother localization of the CNN's attention, with high-intensity regions indicating the time-frequency areas the model deemed most salient. The SHAP Force Plot (Figure 15(b1)) shows how
<figure><img src="image_16.png" /><figcaption>**FIGURE 15.** Multifaceted explainability analysis for lung sound classification from the ICBHI dataset using the Spectrogram + Demographic model. (a1) LIME explanation highlighting influential superpixels. (b1) SHAP force plot for 'COPD' and (b2) SHAP decision plot showing feature contributions for 'Bronchiolitis'. (b3) SHAP attribution maps showing pixel-level importance across classes. (c1) Grad-CAM heatmap localizing CNN attention.</figcaption></figure>

features like Age and Adult BMI push the COPD' prediction, clarified by the SHAP Decision Plot (Figure 15(b2)) for 'Bronchiolitis'. Concurrently, the SHAP Attribution Maps (Figure 15(b3)) offer the most granular insight, displaying pixel-level SHAP values for the spectrogram across multiple potential diagnoses.

### 3) Explainability on COUGHVID Dataset

The explainability analysis for a cough sound instance from the COUGHVID dataset, correctly classified by the multi-modal model as 'Healthy', is shown in Figure 16.

**(a) MFCC-based Visualizations:** The LIME Feature Importance Plot (Figure 16(a2)) explains the model's reasoning in terms of a binary 'symptomatic' vs. 'not symptomatic' decision, showing that the absence of a respiratory condition and fever/muscle pain were the strongest factors supporting the 'not symptomatic' (and therefore 'Healthy') outcome. This is complemented by the SHAP Force Plot (Figure 16(b1)), where the total features are calculated by the Equations (12) and (13), acts as a competing force. Red bars represent features pushing the prediction towards a higher value ('symptomatic'), while blue bars push the prediction lower. The Grad-CAM Model Importance per Frequency Bin plot (Figure 16(c1)) provides a global view, highlighting MFCC coefficients 10–17 as most relevant. For the specific 'Healthy' prediction, the 3D Grad-CAM Surface Plot (Figure 16(c2)) offers a local explanation, showing attentional hotspots as peaks; the low, spread-out peaks indicate no strong acoustic patterns of a pathological cough.

**(b) Spectrogram-based Visualizations:** The explainability analysis for a COUGHVID sample correctly classified as 'COVID-19' is shown in Figure 17. LIME Image Explanations (Figures 17(a1), (a2)) segment the spectrogram into superpixels, highlighting in green the cough regions most supporting the 'COVID-19' prediction, verifying the model's focus on meaningful acoustic features. SHAP Attribution Maps (Figure 17(b1)) provide a granular explanation, displaying the original spectrogram alongside maps for 'Healthy', 'Symptomatic', and 'COVID-19'. The 'COVID-19' map shows strong positive (red) and negative (blue) SHAP values, while the other classes are mostly neutral, illustrating the model's confidence and discriminative reasoning. Grad-CAM visualizations (Figures 17(c1) and (c2)) show which regions the model attends to when making incorrect predictions. Contours and heatmaps highlight acoustic features erroneously associated with 'Healthy', helping diagnose the cause of misclassification.

### 4) Explainability on SAS-KIIT Dataset

Explainability analyses for an environmental sound instance from the SAS-KIIT dataset, which the model classified as 'Fish Market', are illustrated in Figures 18 and 19.

**(a) MFCC-based Visualizations:** The local explanations from LIME (Figures 18(a1), (a2)) and the SHAP Attribution Map (Figure 18(b)) provide a granular view of feature contributions. Red pixels indicate coefficients pushing the prediction higher, and blue pixels indicate those pushing it lower. For complex scenes like 'Fish Market Noise', these visualizations confirm that the model focuses on chaotic sound
<figure><img src="image_17.png" /><figcaption>**FIGURE 16.** Explainability analysis for a 'Healthy' classification from the COUGHVID dataset using the MFCC + Demographic model. (a) LIME feature importance plot. (b1) Local SHAP force plot showing individual feature contributions. (c1) Global Grad-CAM plot highlighting average importance of each MFCC coefficient. (c2) Local 3D Grad-CAM surface plot visualizing model attention on the MFCC input.</figcaption></figure>

<figure><img src="image_18.png" /><figcaption>**FIGURE 17.** Explainability analysis for a COUGHVID 'COVID-19' classification using spectrograms: (a1, a2) LIME highlights key superpixels, (b1) SHAP shows pixel-level importance across classes, and (c1, c2) Grad-CAM localizes attention, revealing focus on 'Healthy' features despite the true label.</figcaption></figure>

patterns rather than irrelevant background, aiding debugging and trust. The Grad-CAM Heatmap on MFCC (Figure 18(c1)) localizes CNN attentional hotspots, while the Global Model Importance Plot (Figure 18(c2)) shows the average importance of each MFCC bin across the dataset, revealing the model's overall strategy.

**(b) Spectrogram-based Visualizations:** The LIME Image Explanations (Figures 19(a1), (a2)) divide the spectrogram into superpixels, highlighting the regions most affecting classification. Complementing this, the Global SHAP Feature Importance Plot (Figure 19(b1)) and Local SHAP Heatmap (Fig-

ure 19(b2)) display SHAP values for individual pixels. The Grad-CAM Contours (Figure 19(c1)) outline the CNN's attentional hotspots, indicating salient time-frequency regions. The Global Attention Map (Figure 19(c2)), obtained by averaging Grad-CAM heatmaps, highlights consistent regions of focus, validating predictions and confirming attention to meaningful acoustic features.

### 5) Explainability on Creama-D Dataset

A comprehensive visualization of the explainability analyses for the same 'Angry' speech instance from the CREMA-D
<figure><img src="image_19.png" /><figcaption>**FIGURE 18.** Explainability analysis for a 'Fish Market Noise' classification from the SAS-KIIT dataset using the MFCC-based model. (a1, a2) LIME image explanations highlighting influential superpixels. (b) Local SHAP attribution map showing pixel-level importance on the MFCC matrix. (c1) Local Grad-CAM heatmap visualizing CNN attention for this instance. (c2) Global Grad-CAM plot showing average importance of each MFCC coefficient across the dataset.</figcaption></figure>

<figure><img src="image_20.png" /><figcaption>**FIGURE 19.** Explainability analysis for a 'Fish Market Noise' classification from the SAS-KIIT dataset using the spectrogram-based model. (a1, a2) LIME image explanations highlighting influential superpixels. (b1) Global SHAP feature importance plot and (b2) local SHAP importance heatmap. (c1) Local Grad-CAM contours visualizing CNN attention for this instance. (c2) Global attention map averaging Grad-CAM heatmaps across the dataset.</figcaption></figure>

dataset is shown in Figures 21 and 20.

**(a) MFCC-based Visualizations:** The LIME Heatmap (Figure 20(a1)) provides a local view of MFCC importance weights over time for the 'Angry' prediction, with darker reddish-brown areas showing features strongly supporting the classification. The SHAP Beeswarm Plot (Figure 20(b1)) offers a global view of feature contributions across predictions, revealing consistent high-impact features (e.g., F_44) that act as key acoustic markers of anger, as illustrated in Equations (12) and (13). At the instance level, the SHAP Waterfall Plot (Figure 20(b2)) shows how the baseline pre-

diction shifts step by step to the final 'Angry' classification. Complementing this, the Grad-CAM Frequency Importance plot (Figure 20(c1)) highlights the average importance of each MFCC coefficient, indicating the model primarily relies on lower-to-mid frequency ranges.

**(b) Spectrogram-based Visualizations:** The LIME Image Explanation (Figure 21(a1)) separates the spectrogram into regions that positively support the prediction (green) and those that oppose it (red). Similarly, the 3D Grad-CAM Surface Plot (Figure 21(c2)) visualizes the model's attentional hotspots as peaks in a landscape, clearly showing how the
<figure><img src="image_21.png" /><figcaption>**FIGURE 20.** Explainability analysis for an 'Angry' emotion classification from the CREMA-D dataset using the MFCC-based model. (a1) Local LIME explanation visualized as a heatmap on the MFCC matrix. (b1) Global SHAP beeswarm plot summarizing feature impacts across the dataset, and (b2) local SHAP waterfall plot for this instance. (c1) Global Grad-CAM feature importance plot showing average importance of each MFCC coefficient.</figcaption></figure>

<figure><img src="image_22.png" /><figcaption>**FIGURE 21.** Explainability analysis for a CREMA-D speech emotion classification using the spectrogram-based model: (a1) LIME highlighting spectrogram regions with positive and negative contributions; (b1) global SHAP heatmap (10-sample average) and (b2) local SHAP maps for top predicted classes; (c1) global Grad-CAM showing average frequency-bin importance; (c2) local 3D Grad-CAM surface visualizing attention for the specific prediction.</figcaption></figure>

model's focus shifts in both time and frequency. The Average SHAP Heatmap (Figure 21(b1)) reveals the time-frequency regions that the model, on average, considers most important for this emotion class. For the individual prediction, the SHAP Attribution Maps (Figure 21(b2)) are particularly powerful. They show why the model chose the correct emotion and rejected other candidates by highlighting evidence and lack thereof.

## 6) Explainability on Audio MNIST Dataset

The explainability analysis for a spoken digit instance from the Audio MNIST dataset, which the unimodal model correctly classified as '6', is shown in Figure 22.

**(a) MFCC-based Visualizations:** The LIME Heatmap (Figure 22(a1)) shows uniform MFCC importance over time, indicating reliance on the overall acoustic shape of the digit.

The SHAP Feature Importance Plot (Figure 22(b1)) ranks contributions, with red bars favoring '6' and blue bars opposing it, while the SHAP Attribution Map (Figure 22(b2)) provides a pixel-level view of positive (red) and negative (blue) coefficients. The Grad-CAM analysis (Fig. 22(c1)) visualizes salient MFCC regions influencing the CNN's '6' classification. The heatmap and its superimposed result reveal that the model's attention is concentrated on specific temporal and frequency segments of the spoken digit.

**(b) Spectrogram-based Visualizations:** A spoken digit instance from AudioMNIST, correctly classified as '7' by the unimodal spectrogram-based model, is analyzed in Figure 23. LIME (Figure 23(a1)) highlights influential superpixels for each class, revealing the distinct time-frequency patterns the model associates with different digits. SHAP Attribution Maps (Figure 23(b1)) provide a local, contrastive view: the
<figure><img src="image_23.png" /><figcaption>**FIGURE 22.** Deconstructing a '6' classification from the Audio MNIST dataset using various XAI methods on MFCC features. (a1) A local LIME importance heatmap visualizing feature weights over time. (b1) A local SHAP plot ranking the most impactful features. (b2) A SHAP attribution map showing pixel-level importance. (c1) Grad-CAM heatmap and the superimposed result, visualizing the CNN's attention.</figcaption></figure>

<figure><img src="image_24.png" /><figcaption>**FIGURE 23.** Explainability analysis for a '7' classification in the Audio MNIST spectrogram-based model: (a1) global LIME showing influential superpixels for all digit classes; (b1) local contrastive SHAP maps for '7' and competing classes; (c1) Grad-CAM plot of aggregated attention over time; (c2) Grad-CAM heatmap highlighting attention in the time-frequency domain.</figcaption></figure>

true class '7' shows strong positive (red) and negative (blue) values, while other digits like 0', 3', and 6' exhibit little evidence. Grad-CAM offers a dual perspective on model attention. The time-aggregated plot (Figure 23(c1)) shows attention distribution across time, while the 2D heatmap (Figure 23(c2)) visualizes joint focus over time and frequency.

## C. QUANTITATIVE AND QUALITATIVE EVALUATION OF XAI METHODS

To rigorously evaluate the interpretability of our trained models, we quantitatively assess XAI methods across multiple datasets and input types (MFCC and spectrogram). Unlike

purely visual explanations, this analysis uses *reproducible metrics* fidelity, stability, and sparsity to measure the quality and consistency of explanations. The same models evaluated for performance in Table 5 are used here for explainability assessment, ensuring consistency between model behavior and XAI evaluations.

### 1) Metrics Definition

**Fidelity ($F_\alpha$, ↑):** Fidelity measures how well an explanation reflects the model's true decision. It is computed using *Inspection AUC*, which quantifies the increase in model prediction confidence as the top features highlighted by the explanation
<table><caption>TABLE 6. Comparative evaluation of XAI methods across multiple datasets and input representations, measured in terms of fidelity, stability, and sparsity.</caption><thead><tr><th rowspan="2">Dataset</th><th colspan="5">MFCC-based Inputs</th><th colspan="5">Spectrogram-based Inputs</th></tr><tr><th>Input</th><th>Method</th><th>Fidelity</th><th>Stability</th><th>Sparsity</th><th>Input</th><th>Method</th><th>Fidelity</th><th>Stability</th><th>Sparsity</th></tr></thead><tbody><tr><td rowspan="6">CIRCOR</td><td rowspan="3">MFCC</td><td>LIME</td><td>0.78±0.03</td><td>0.82±0.04</td><td>0.024</td><td rowspan="3">Spectrogram</td><td>LIME</td><td>0.79±0.03</td><td>0.83±0.04</td><td>0.015</td></tr><tr><td>SHAP</td><td>0.80±0.02</td><td>0.80±0.05</td><td>0.016</td><td>SHAP</td><td>0.81±0.02</td><td>0.81±0.05</td><td>0.010</td></tr><tr><td>Grad-CAM</td><td>0.76±0.04</td><td>0.81±0.04</td><td>0.032</td><td>Grad-CAM</td><td>0.77±0.04</td><td>0.82±0.04</td><td>0.020</td></tr><tr><td rowspan="3">MFCC+Demo</td><td>LIME</td><td>0.77±0.03</td><td>0.81±0.05</td><td>0.024</td><td rowspan="3">Spec+Demo</td><td>LIME</td><td>0.80±0.03</td><td>0.82±0.04</td><td>0.015</td></tr><tr><td>SHAP</td><td>0.79±0.02</td><td>0.80±0.05</td><td>0.016</td><td>SHAP</td><td>0.82±0.02</td><td>0.81±0.05</td><td>0.010</td></tr><tr><td>Grad-CAM</td><td>0.75±0.04</td><td>0.81±0.04</td><td>0.032</td><td>Grad-CAM</td><td>0.78±0.04</td><td>0.83±0.04</td><td>0.020</td></tr><tr><td rowspan="6">ICBHI</td><td rowspan="3">MFCC</td><td>LIME</td><td>0.80±0.03</td><td>0.84±0.03</td><td>0.018</td><td rowspan="3">Spectrogram</td><td>LIME</td><td>0.81±0.03</td><td>0.82±0.04</td><td>0.015</td></tr><tr><td>SHAP</td><td>0.82±0.02</td><td>0.82±0.04</td><td>0.010</td><td>SHAP</td><td>0.83±0.02</td><td>0.80±0.05</td><td>0.010</td></tr><tr><td>Grad-CAM</td><td>0.78±0.04</td><td>0.83±0.03</td><td>0.020</td><td>Grad-CAM</td><td>0.79±0.04</td><td>0.81±0.04</td><td>0.020</td></tr><tr><td rowspan="3">MFCC+Demo</td><td>LIME</td><td>0.82±0.02</td><td>0.84±0.03</td><td>0.018</td><td rowspan="3">Spec+Demo</td><td>LIME</td><td>0.83±0.02</td><td>0.83±0.04</td><td>0.015</td></tr><tr><td>SHAP</td><td>0.84±0.02</td><td>0.82±0.04</td><td>0.010</td><td>SHAP</td><td>0.85±0.02</td><td>0.81±0.05</td><td>0.010</td></tr><tr><td>Grad-CAM</td><td>0.80±0.03</td><td>0.83±0.03</td><td>0.020</td><td>Grad-CAM</td><td>0.81±0.03</td><td>0.82±0.04</td><td>0.020</td></tr><tr><td rowspan="6">CoughVID</td><td rowspan="3">MFCC</td><td>LIME</td><td>0.75±0.04</td><td>0.78±0.05</td><td>0.024</td><td rowspan="3">Spectrogram</td><td>LIME</td><td>0.76±0.04</td><td>0.79±0.05</td><td>0.015</td></tr><tr><td>SHAP</td><td>0.77±0.03</td><td>0.77±0.06</td><td>0.016</td><td>SHAP</td><td>0.78±0.03</td><td>0.78±0.06</td><td>0.010</td></tr><tr><td>Grad-CAM</td><td>0.74±0.05</td><td>0.78±0.05</td><td>0.032</td><td>Grad-CAM</td><td>0.75±0.05</td><td>0.79±0.05</td><td>0.020</td></tr><tr><td rowspan="3">MFCC+Demo</td><td>LIME</td><td>0.74±0.04</td><td>0.77±0.06</td><td>0.024</td><td rowspan="3">Spec+Demo</td><td>LIME</td><td>0.76±0.04</td><td>0.78±0.05</td><td>0.015</td></tr><tr><td>SHAP</td><td>0.76±0.03</td><td>0.76±0.06</td><td>0.016</td><td>SHAP</td><td>0.78±0.03</td><td>0.77±0.06</td><td>0.010</td></tr><tr><td>Grad-CAM</td><td>0.73±0.05</td><td>0.77±0.05</td><td>0.032</td><td>Grad-CAM</td><td>0.75±0.05</td><td>0.78±0.05</td><td>0.020</td></tr><tr><td rowspan="3">SAS-KIIT</td><td rowspan="3">MFCC</td><td>LIME</td><td>0.98±0.01</td><td>0.97±0.02</td><td>0.010</td><td rowspan="3">Spectrogram</td><td>LIME</td><td>0.97±0.01</td><td>0.96±0.02</td><td>0.015</td></tr><tr><td>SHAP</td><td>0.99±0.01</td><td>0.98±0.01</td><td>0.010</td><td>SHAP</td><td>0.98±0.01</td><td>0.97±0.01</td><td>0.010</td></tr><tr><td>Grad-CAM</td><td>0.97±0.02</td><td>0.96±0.02</td><td>0.020</td><td>Grad-CAM</td><td>0.96±0.02</td><td>0.95±0.02</td><td>0.020</td></tr><tr><td rowspan="3">CREMA-D</td><td rowspan="3">MFCC</td><td>LIME</td><td>0.92±0.02</td><td>0.91±0.03</td><td>0.015</td><td rowspan="3">Spectrogram</td><td>LIME</td><td>0.90±0.02</td><td>0.89±0.03</td><td>0.015</td></tr><tr><td>SHAP</td><td>0.94±0.02</td><td>0.92±0.02</td><td>0.010</td><td>SHAP</td><td>0.92±0.02</td><td>0.90±0.02</td><td>0.010</td></tr><tr><td>Grad-CAM</td><td>0.91±0.03</td><td>0.90±0.03</td><td>0.020</td><td>Grad-CAM</td><td>0.89±0.03</td><td>0.88±0.03</td><td>0.020</td></tr><tr><td rowspan="3">Audio MNIST</td><td rowspan="3">MFCC</td><td>LIME</td><td>0.99±0.01</td><td>0.98±0.01</td><td>0.010</td><td rowspan="3">Spectrogram</td><td>LIME</td><td>0.98±0.01</td><td>0.97±0.02</td><td>0.015</td></tr><tr><td>SHAP</td><td>0.99±0.01</td><td>0.99±0.01</td><td>0.010</td><td>SHAP</td><td>0.99±0.01</td><td>0.98±0.01</td><td>0.010</td></tr><tr><td>Grad-CAM</td><td>0.98±0.01</td><td>0.97±0.02</td><td>0.020</td><td>Grad-CAM</td><td>0.97±0.02</td><td>0.96±0.02</td><td>0.020</td></tr></tbody></table>

are gradually inserted into a baseline input:

$$ \mathcal{F}_{\alpha} = \frac{1}{N} \sum_{i=1}^{N} \Delta \hat{y}_{i} \quad (14) $$

Here, *N* is the number of test samples, and $\Delta \hat{y}_i$ is the change in model prediction for sample *i* after inserting the top-*k* features identified by the explanation. A higher $\mathcal{F}_{\alpha}$ indicates that the explanation better captures the truly influential features.

**Stability** ($S_{\gamma}$, ↑). Stability quantifies the consistency of explanations across repeated runs or input perturbations. It is measured using the Jaccard Index between sets of top features obtained from different runs:

$$ S_{\gamma} = \frac{|\mathcal{H}_i \cap \mathcal{H}_j|}{|\mathcal{H}_i \cup \mathcal{H}_j|}, \quad i \neq j \quad (15) $$

where $\mathcal{H}_i$ and $\mathcal{H}_j$ are the sets of top-*k* features highlighted in runs *i* and *j*, respectively, and $| \cdot |$ denotes the set cardinality. Larger $S_{\gamma}$ values indicate more reproducible explanations.

**Sparsity** ($\mathcal{P}_{\theta}$, ↓). Sparsity measures the conciseness of an explanation, i.e., the fraction of features highlighted relative to the total number of input features:

$$ \mathcal{P}_{\theta} = \frac{|\mathcal{H}_i|}{F_{\text{tot}}} \quad (16) $$

Here, $|\mathcal{H}_i|$ is the number of highlighted features for sample *i*, and $F_{\text{tot}}$ is the total number of input features (MFCC coefficients or spectrogram bins). Since audio clips vary in length, $F_{\text{tot}}$ also varies across samples, which naturally introduces differences in sparsity values.

## 2) Unified Quantitative Results

Table 6 presents the fidelity, stability, and sparsity for LIME, SHAP, and Grad-CAM across all datasets and input modalities. SHAP consistently achieves the highest fidelity, indicating robust identification of influential features. Grad-CAM applied to MFCC inputs effectively highlights time-frequency regions, validating its methodological soundness. Incorporating demographic features slightly improves fidelity without affecting sparsity, confirming the complementarity of multimodal inputs. Sparsity values remain low overall, reflecting concise explanations; their variation is influenced by the total number of input features ($F_{\text{tot}}$), which depends on the audio length and feature extraction method. Stability remains consistently high, demonstrating the reproducibility of the explanations across runs.

## 3) Justification for Grad-CAM on MFCC Features

MFCCs are represented as 2D arrays (time × coefficient), enabling convolutional neural networks to capture local temporal-spectral patterns. Grad-CAM applied to CNN feature maps produces attention maps highlighting influential features.
<table><caption>TABLE 7. Summary of Key Explainability Findings Across All Datasets.</caption><thead><tr><th>Dataset</th><th>Input Type</th><th>LIME Findings</th><th>SHAP Findings</th><th>Grad-CAM Findings</th></tr></thead><tbody><tr><td rowspan="2">CIRCOR</td><td>MFCC with Demographic Data (Figure 12)</td><td>LIME revealed that demographic features like Age and Weight were key drivers of the prediction, alongside specific MFCC regions.</td><td>SHAP precisely quantified the contribution of each demographic feature and provided a granular attribution map on the MFCC matrix.</td><td>Heatmaps with contours localized the model's focus to distinct temporal segments of the heart sound cycle.</td></tr><tr><td>Spectrogram with Demographic Data (Figure 13)</td><td>Explanations confirmed that both demographic data and specific time-frequency regions in the spectrogram aided the final prediction.</td><td>Global plots revealed a learned "murmur template," while local plots showed how demographic features influenced the outcome.</td><td>Heatmaps clearly highlighted clinically relevant phases (e.g., systolic) of the heart sound.</td></tr><tr><td rowspan="2">ICBHI</td><td>MFCC with Demographic Data (Figure 14)</td><td>LIME identified demographic features (Age, BMI) as key contributors complementing the acoustic features.</td><td>The beeswarm plot revealed global trends, showing how certain features consistently influence the model across the dataset.</td><td>The 3D surface plot provided a highly intuitive visualization of the model's "attentional hotspots" on the MFCC input.</td></tr><tr><td>Spectrogram with Demographic Data (Figure 15)</td><td>SHAP attribution maps revealed distinct acoustic patterns learned for different respiratory conditions.</td><td>Heatmaps successfully localized patterns corresponding to wheezes and crackles.</td><td>Shows evidence that the model is not a "black box" and focuses on known pathological sounds.</td></tr><tr><td rowspan="2">CoughVID</td><td>MFCC with Demographic Data (Figure 16)</td><td>Explanations showed heavy reliance on user-reported symptoms (demographic data) like fever.</td><td>Global importance plots identified the key MFCC frequency bins the model learned to rely on.</td><td>Highlights possible model dependency on metadata over acoustic features, requiring careful validation.</td></tr><tr><td>Spectrogram with Demographic Data (Figure 17)</td><td>Contrastive maps showed the model can distinguish between COVID-19 and other symptomatic coughs.</td><td>Heatmaps localized focus on the explosive phase of the cough sound.</td><td>Confirmed the model's ability to learn discriminative acoustic features from a crowdsourced dataset.</td></tr><tr><td rowspan="2">SAS-KIIT</td><td>MFCC (Figure 18)</td><td>SHAP maps showed importance across the entire sound event.</td><td>Global plots highlighted key bands for environmental sounds.</td><td>Validated that the model learned the general acoustic signature.</td></tr><tr><td>Spectrogram (Figure 19)</td><td>LIME highlighted specific chaotic regions as important for the 'Fish Market' class.</td><td>Global attention maps confirmed the model's focus on relevant frequency bands.</td><td>Demonstrates the model's ability to identify key events in complex acoustic scenes.</td></tr><tr><td rowspan="2">CREMA-D</td><td>MFCC (Figure 20)</td><td>The heatmap visualized the importance of each MFCC coefficient over time for the 'Angry' prediction.</td><td>The beeswarm plot revealed global trends (e.g., features consistently indicating anger), while the waterfall plot explained a single prediction.</td><td>The global importance plot identified the key frequency bins the model uses to distinguish between emotions.</td></tr><tr><td>Spectrogram (Figure 21)</td><td>LIME separated the spectrogram into regions with positive (supporting) and negative (opposing) contributions to the emotion classification.</td><td>Contrastive maps explained why the model chose the correct emotion and rejected the other most likely candidates.</td><td>The 3D surface plot showed how the model's attention shifts in both time and frequency, corresponding to emotional prosody.</td></tr><tr><td rowspan="2">Audio MNIST</td><td>MFCC (Figure 22)</td><td>The importance heatmap showed a relatively uniform focus across the entire spoken digit.</td><td>The feature importance plot ranked the specific MFCC coefficients that most impacted the '6' prediction.</td><td>The heatmap localized the model's attention to the specific temporal segments of the spoken digit.</td></tr><tr><td>Spectrogram (Figure 23)</td><td>Global analysis revealed the learned "template" of influential regions for each of the ten digit classes.</td><td>Contrastive maps explained why the model chose '7' and, crucially, why it rejected other similar-sounding digits.</td><td>The attention-over-time plot showed *when* the model was focusing, while the importance mask showed *where* in the spectrogram.</td></tr></tbody></table>

time-frequency regions. This approach, consistent with prior audio AI studies, shows that Grad-CAM on MFCCs provides interpretable and meaningful insights.

### 4) Notes on Metrics

* Fidelity and stability values lie in [0,1], with higher values indicating better alignment with model predictions and greater reproducibility.

* Sparsity is normalized to [0,1]; lower values indicate more concise explanations highlighting fewer features.

* Insertion AUC is reported as fidelity; deletion AUC was also evaluated with consistent trends but omitted for brevity.

## V. DISCUSSION

This study conducted a cross-domain evaluation of a unified deep learning architecture for audio classification, with a central focus on analyzing model behavior through Explainable AI (XAI). By combining model performance outcomes (Table 5) with a quantitative evaluation of explanations (Table 6), we critically assess the reliability, complementarity, and practical implications of XAI in diverse audio intelligence tasks. The model's legitimacy is first established by predictive accuracy. On benchmark datasets like CREMA-D, Audio MNIST, and SAS-KIIT, performance was consistently strong (>95%), validating the CNN's ability to capture robust acoustic patterns. More modest but stable performance on
clinical datasets (CIRCOR, ICBHI, CoughVID) underscores the challenges of real-world medical audio, including noise, inter-patient variability, and subtle pathological cues.

Beyond classification accuracy, the main contribution lies in systematically assessing explanations. The three XAI methods, SHAP, LIME, and Grad-CAM, offer complementary perspectives, as summarized in Table 7, which presents a concise overview of the key findings. Numerical comparisons (Table 6) show that SHAP consistently achieved the highest fidelity (e.g., 0.85±0.02 on ICBHI multimodal vs. 0.83±0.02 for LIME and 0.81±0.03 for Grad-CAM) and the lowest sparsity, producing concise and stable explanations. LIME and Grad-CAM, while less precise, add interpretive richness and computational efficiency. A key insight from cross-domain analysis is that while the specific acoustic features highlighted by explanations are domain-dependent, such as prosodic cues in speech emotion recognition or systolic/diastolic segments in heart sounds, the interpretive role of XAI generalizes across contexts. In every task, explanations consistently surfaced the most task-relevant features, demonstrating that the framework is transferable even when the underlying signals vary.

Multimodal experiments further revealed how demographic variables interact with acoustic inputs. Explanations attributed predictions to both audio features and metadata (e.g., age, BMI), enabling richer and more patient-specific interpretations. Importantly, no single demographic factor disproportionately dominated attributions, supporting fairness in model behavior. Another practical consideration concerns computational feasibility. Perturbation-based methods such as LIME and SHAP are computationally demanding, making them best suited for offline auditing and validation. In contrast, Grad-CAM is lightweight and thus more viable for real-time or embedded systems. This distinction enables a division of labor: Grad-CAM for rapid deployment, SHAP for rigorous validation, and LIME as an accessible middle ground for interpretability checks.

Taken together, these findings motivate several practical guidelines. For healthcare applications, SHAP emerges as the preferred method due to its strong fidelity, stability, and theoretical grounding. For real-time or embedded systems, Grad-CAM offers the most feasible solution given its computational efficiency. In the context of general debugging and model development, a hybrid strategy is recommended: Grad-CAM provides quick focus visualizations, LIME enables intuitive local checks, and SHAP supplies robust global attributions. Finally, the choice of feature representation should also be task-driven: spectrograms are highly effective when human-centric interpretability is the priority, as Grad-CAM heatmaps align naturally with their time-frequency axes, while MFCCs are more appropriate when predictive accuracy and compactness are paramount.

Finally, we acknowledge a limitation regarding the inclusion of demographic data in our multimodal models. While feature attribution analyses suggest that predictions were not overly influenced by demographic factors, a formal quan-

titative fairness assessment such as evaluating demographic parity or other established metrics was not conducted. Conducting a rigorous fairness evaluation remains an important direction for future work. We note a dimensionality mismatch between MFCCs (40-D) and spectrograms (128×128), which may influence interpretability outcomes. While this choice follows common practice, it does not fully disentangle feature type from dimensionality. Matching feature dimensions or formally studying its impact on interpretability remains a potential avenue for future work.

Together, these insights offer a practical roadmap for deploying XAI in audio intelligence, ensuring transparency, robustness, and domain relevance across both clinical and non-clinical contexts.

## VI. CONCLUSION AND FUTURE WORK

This paper presented a comprehensive cross-domain analysis evaluating three prominent XAI techniques, LIME, SHAP, and Grad-CAM in diverse audio intelligence tasks. Moving beyond purely qualitative assessments, we introduced a rigorous quantitative framework to measure the quality of explanations. Our evaluation, based on fidelity, stability, and sparsity metrics, demonstrated that while all tested methods produce reliable explanations, SHAP consistently offers the highest fidelity and the most concise outputs. Furthermore, we identified a key practical trade-off for practitioners: spectrograms generally yield more intuitive explanations for human experts, whereas MFCCs can provide superior predictive accuracy in certain tasks. These findings, distilled into a set of practical guidelines, contribute to a deeper and more nuanced understanding of how to build transparent and trustworthy audio AI systems. Building on this work, several important avenues for future research are apparent. This framework could be extended to other advanced architectures, such as attention-based Transformers, which may yield different explainability patterns. A crucial next step is to conduct a formal quantitative fairness audit, expanding on the preliminary investigation presented here to ensure models are equitable across demographic subgroups. Other promising directions include developing interactive explanation interfaces for real-time clinical use and pursuing formal validation of these explanations with domain experts to confirm their practical utility.

## REFERENCES

[1] V. Bento, M. Kohler, P. Diaz, L. Mendoza, and M. A. Pacheco, "Improving deep learning performance by using explainable artificial intelligence (xai) approaches," *Discover Artificial Intelligence*, vol. 1, no. 1, p. 9, 2021.

[2] M. Swain, A. Routray, and P. Kabisatpathy, "Databases, features and classifiers for speech emotion recognition: a review," *International Journal of Speech Technology*, vol. 21, no. 1, pp. 93–120, 2018.

[3] K. Zaman, M. Sah, C. Direkoglu, and M. Unoki, "A survey of audio classification using deep learning," *IEEE access*, vol. 11, pp. 106620–106649, 2023.

[4] M. French and R. Handy, "Spectrograms: turning signals into pictures," *Journal of engineering technology*, vol. 24, no. 1, p. 32, 2007.

[5] Z. K. Abdul and A. K. Al-Talabani, "Mel frequency cepstral coefficient and its applications: A review," *IEEE Access*, vol. 10, pp. 122136–122158, 2022.
[6] G. Adomavicius and A. Tuzhilin, "Expert-driven validation of rule-based user models in personalization application," *Data Mining and Knowledge Discovery*, vol. 5, 10 2000.

[7] E. LaRosa and D. Danks, "Impacts on trust of healthcare ai," in *Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society*, 2018, pp. 210–215.

[8] R. Chatterjee, P. Bishwas, S. Chakrabarty, and T. Bandyopadhyay, "South asian sounds: Audio classification," in *2024 4th International Conference on Computer, Communication, Control & Information Technology (C3IT)*, 2024, pp. 1–6.

[9] D. Gunning, "Explainable artificial intelligence (xai)," *Defense advanced research projects agency (DARPA),* nd Web, vol. 2, no. 2, p. 1, 2017.

[10] M. T. Ribeiro, S. Singh, and C. Guestrin, "" why should i trust you?" explaining the predictions of any classifier," in *Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining*, 2016, pp. 1135–1144.

[11] S. M. Lundberg and S.-I. Lee, "A unified approach to interpreting model predictions," *Advances in neural information processing systems*, vol. 30, 2017.

[12] D. Vij, Y. Yogesh, D. Srivastava, and H. Shankar, "Detection of acoustic scenes and events using audio analysis – a survey," in *2023 3rd International Conference on Advance Computing and Innovative Technologies in Engineering (ICACITE)*, 2023, pp. 316–320.

[13] N. K. Chowdhury, M. A. Kabir, M. M. Rahman, and S. M. S. Islam, "Machine learning for detecting covid-19 from cough sounds: An ensemble-based mcdm method," *Computers in Biology and Medicine*, vol. 145, p. 105405, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0010482522001974

[14] D. Stowell, D. Giannoulis, E. Benetos, M. Lagrange, and M. D. Plumbley, "Detection and classification of acoustic scenes and events," *IEEE Transactions on Multimedia*, vol. 17, no. 10, pp. 1733–1746, 2015.

[15] W. C. Sleeman, R. Kapoor, and P. Ghosh, "Multimodal classification: Current landscape, taxonomy and future directions," *ACM Comput. Surv.*, vol. 55, no. 7, Dec. 2022. [Online]. Available: https://doi.org/10.1145/3543848

[16] R. Dwivedi, D. Dave, H. Naik, S. Singhal, R. Omer, P. Patel, B. Qian, Z. Wen, T. Shah, G. Morgan, and R. Ranjan, "Explainable ai (xai): Core ideas, techniques, and solutions," *ACM Comput. Surv.*, vol. 55, no. 9, Jan. 2023. [Online]. Available: https://doi.org/10.1145/3561048

[17] J. Van Der Waa, E. Nieuwburg, A. Cremers, and M. Neerincx, "Evaluating xai: A comparison of rule-based and example-based explanations," *Artificial intelligence*, vol. 291, p. 103404, 2021.

[18] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, "Grad-cam: Visual explanations from deep networks via gradient-based localization," in *Proceedings of the IEEE international conference on computer vision*, 2017, pp. 618–626.

[19] Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," nature, vol. 521, no. 7553, pp. 436–444, 2015.

[20] E. Glikson and A. W. Woolley, "Human trust in artificial intelligence: Review of empirical research," *Academy of management annals*, vol. 14, no. 2, pp. 627–660, 2020.

[21] W. Samek, T. Wiegand, and K.-R. Müller, "Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models," *arXiv preprint arXiv:1708.08296*, 2017.

[22] F. Doshi-Velez and B. Kim, "Towards a rigorous science of interpretable machine learning," *arXiv preprint arXiv:1702.08608*, 2017.

[23] E. Tjoa and C. Guan, "A survey on explainable artificial intelligence (xai): Toward medical xai," *IEEE Transactions on Neural Networks and Learning Systems*, vol. 32, no. 11, pp. 4793–4813, 2021.

[24] A. Holzinger, C. Biemann, C. Pattichis, and D. Kell, "What do we need to build explainable ai systems for the medical domain?" *arXiv preprint arXiv:1712.09923*, 12 2017.

[25] A. Barredo Arrieta, N. Díaz-Rodríguez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado, S. Garcia, S. Gil-Lopez, D. Molina, R. Benjamins, R. Chatila, and F. Herrera, "Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai," *Information Fusion*, vol. 58, pp. 82–115, 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1566253519308103

[26] A. Akman and B. Schuller, "Audio explainable artificial intelligence: A review," *Intelligent Computing*, vol. 2, 01 2024.

[27] C. O. Retzlaff, A. Angerschmid, A. Saranti, D. Schneeberger, R. Röttger, H. Müller, and A. Holzinger, "Post-hoc vs ante-hoc explanations: xai design guidelines for data scientists," *Cognitive Systems Research*,

vol. 86, p. 101243, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1389041724000378

[28] N. Sobahi, O. Atila, E. Deniz, A. Sengur, and U. R. Acharya, "Explainable covid-19 detection using fractal dimension and vision transformer with grad-cam on cough sounds," *Biocybernetics and Biomedical Engineering*, vol. 42, 2022.

[29] L. Orlandic, T. Teijeiro, and D. Atienza, "The COUGHVID crowdsourcing dataset, a corpus for the study of large-scale cough analysis algorithms," *Scientific Data*, vol. 8, no. 1, p. 156, jun 2021. [Online]. Available: https://doi.org/10.1038/s41597-021-00937-4

[30] G. Chaudhari, X. Jiang, A. Fakhry, A. Han, J. Xiao, S. Shen, and A. Khanzada, "Virufy: Global applicability of crowdsourced and clinical datasets for ai detection of covid-19 from cough," *arXiv preprint arXiv:2011.13320*, 2020.

[31] D. Bhattacharya, N. K. Sharma, D. Dutta, S. R. Chetupalli, P. Mote, S. Ganapathy, C. Chandrakiran, S. Nori, K. Suhail, S. Gonuguntla et al., "Coswara: A respiratory sounds and symptoms dataset for remote screening of sars-cov-2 infection," *Scientific data*, vol. 10, no. 1, p. 397, 2023.

[32] J. Ramakrishna, C. Sonagiri, K. Kumar, and P. Shreya, "Development of explainable machine intelligence models for heart sound abnormality detection," *Indonesian Journal of Electrical Engineering and Computer Science*, vol. 36, p. 846, 11 2024.

[33] E. Mahamud, N. Fahad, M. Assaduzzaman, S. Zain, K. O. M. Goh, and M. K. Morol, "An explainable artificial intelligence model for multiple lung diseases classification from chest x-ray images using fine-tuned transfer learning," *Decision Analytics Journal*, vol. 12, p. 100499, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2772662224001036

[34] A. Chattopadhay, A. Sarkar, P. Howlader, and V. N. Balasubramanian, "Grad-cam+: Generalized gradient-based visual explanations for deep convolutional networks," in *2018 IEEE Winter Conference on Applications of Computer Vision (WACV)*, 2018, pp. 839–847.

[35] D. Dang, N. T. Pham, and S. Nguyen, "Speech emotion recognition using overlapping sliding window and shapley additive explainable deep neural network," *Journal of Information and Telecommunication*, 03 2023.

[36] F. Prity, M. Hossain, M. Hossain, M. Uddin, M. Islam, M. Raquib, M. Ali, and K. Uddin, "Artificial intelligence-powered environmental sound recognition with explainable ai techniques," *Iran Journal of Computer Science*, pp. 1–34, 06 2025.

[37] M. Shen, P. Mortezaagha, and A. Rahgozar, "Explainable artificial intelligence to diagnose early parkinson's disease via voice analysis," *Scientific Reports*, vol. 15, no. 1, p. 11687, 2025.

[38] F. Prior, T. Virmani, A. Iyer, L. Larson-Prior, A. Kemp, Y. Rahmatallah, L. Pillai, and A. Glover, "Voice Samples for Patients with Parkinson's Disease and Healthy Controls," 8 2023. [Online]. Available: https://figshare.com/articles/dataset/Voice_Samples_for_Patients_with_Parkinson_s_Disease_and_Healthy_Controls/23849127

[39] L. Tang, Z. Zhang, F. Feng, L.-Z. Yang, and H. Li, "Explainable alzheimer's disease detection using linguistic features from automatic speech recognition," *Dementia and geriatric cognitive disorders*, vol. 52, 07 2023.

[40] A. Alyoubi and B. Alyoubi, "Interpretable multimodal emotion recognition using optimized transformer model with shap-based transparency," *The Journal of Supercomputing*, vol. 81, 06 2025.

[41] S. Wang, J. Hu, Y. Du, X. Yuan, Z. Xie, and P. Liang, "Wcformer: An interpretable deep learning framework for heart sound signal analysis and automated diagnosis of cardiovascular diseases," *Expert Systems with Applications*, vol. 276, p. 127238, 2025. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0957417425008607

[42] N. Das, N. Padhy, N. Dey, H. Paul, and S. Chowdhury, "Exploring explainable ai methods for bird sound-based species recognition systems," *Multimedia Tools and Applications*, vol. 83, no. 24, pp. 64 223–64 253, 2024.

[43] S. Becker, J. Vielhaben, M. Ackermann, K.-R. Müller, S. Lapuschkin, and W. Samek, "Audionnist: Exploring explainable artificial intelligence for audio analysis on a simple benchmark," *Journal of the Franklin Institute*, vol. 361, no. 1, pp. 418–428, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0016003223007536

[44] S. Sultana, A. B. M. A. Hossain, and J. Alam, "Covid-19 detection from optimized features of breathing audio signals using explainable ensemble machine learning," *Results in Control and Optimization*, vol. 18, p. 100538, 2025. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2666720725000244
[45] M. Norval and Z. Wang, "Explorable artificial intelligence techniques for speech emotion recognition: A focus on xai models," *Inteligencia Artificial*, vol. 28, pp. 85–123, 06 2025.

[46] Y. Choi and H. Lee, "Interpretation of lung disease classification with light attention connected module," *Biomedical Signal Processing and Control*, vol. 84, p. 104695, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1746809423001283

[47] T.-W. Kim and K.-C. Kwak, "Speech emotion recognition using deep learning transfer models and explainable techniques," *Applied Sciences*, vol. 14, p. 1553, 02 2024.

[48] S. Shokouhmand, M. M. Rahman, M. Faezipour, and S. Bhatt, "Ad-ventitious pulmonary sound detection: Leveraging shap explanations and gradient boosting insights," in *2024 46th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)*, 2024, pp. 1–4.

[49] I. D. Mienye, G. Obaido, N. Jere, E. Mienye, K. Aruleba, I. D. Emmanuel, and B. Ogbuokiri, "A survey of explainable artificial intelligence in healthcare: Concepts, applications, and challenges," *Informatics in Medicine Unlocked*, vol. 51, p. 101587, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2352914824001448

[50] P. Seth and V. K. Sankarapu, "Bridging the gap in xai-why reliable metrics matter for explainability and compliance," *arXiv preprint arXiv:2502.04695*, 2025.

[51] A. Frommholz, F. Seipel, S. Lapuschkin, W. Samek, and J. Vielhaben, "Xai-based comparison of audio event classifiers with different input representations," in *Proceedings of the 20th International Conference on Content-Based Multimedia Indexing*, ser. CBMI '23. New York, NY, USA: Association for Computing Machinery, 2023, p. 126–132. [Online]. Available: https://doi.org/10.1145/3617233.3617265

[52] D. Zhang, S. Yang, X. Yuan, and P. Zhang, "Interpretable deep learning for automatic diagnosis of 12-lead electrocardiogram," *Iscience*, vol. 24, no. 4, 2021.

[53] J. Ramakrishna, C. Sonagiri, K. Kumar, and P. Shreya, "Development of explainable machine intelligence models for heart sound abnormality detection," *Indonesian Journal of Electrical Engineering and Computer Science*, vol. 36, p. 846, 11 2024.

[54] A. Nfissi, W. Bouachir, N. Bouguila, and B. Mishara, "Unveiling hidden factors: explainable ai for feature boosting in speech emotion recognition," *Applied Intelligence*, vol. 54, no. 11, pp. 7046–7069, 2024.

[55] A. E. Jenifer, K. S. Abirami, and M. Rajeshwari, "Enhanced audio signal classification with explainable ai: Deep learning approach in time and frequency domain analysis," *Procedia Computer Science*, vol. 258, pp. 2372–2381, 2025, international Conference on Machine Learning and Data Engineering. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1877050925016047

[56] J. Oliveira, F. Renna, P. Costa, M. Nogueira, A. C. Oliveira, A. Elola, C. Ferreira, A. Jorge, A. Bahrami Rad, M. Reyna, R. Sameni, G. Clifford, and M. Coimbra, "The CirCor DigiScope Phonocardiogram Dataset," 2022. [Online]. Available: https://doi.org/10.13026/tshs-mw03

[57] W. Kester, "What the nyquist criterion means to your sampled data system design," *Analog Devices*, pp. 1–12, 2009.

[58] Z. Sun, "ICBHI 2017 challenge," 2023. [Online]. Available: https://doi.org/10.7910/DVN/HT6PKI

[59] H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova, and R. Verma, "Crema-d: Crowd-sourced emotional multimodal actors dataset," *IEEE Transactions on Affective Computing*, vol. 5, no. 4, pp. 377–390, 2014.

[60] R. A. Khalil, E. Jones, M. I. Babar, T. Jan, M. H. Zafar, and T. Alhussain, "Speech emotion recognition using deep learning techniques: A review," *IEEE Access*, vol. 7, pp. 117 327–117 345, 2019.

[61] S. Becker, J. Vielhaben, M. Ackermann, K.-R. Müller, S. Lapuschkin, and W. Samek, "Audiomnist: Exploring explainable artificial intelligence for audio analysis on a simple benchmark," *Journal of the Franklin Institute*, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0016003223007536

[62] K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," *arXiv preprint arXiv:1409.1556*, 2014.

[63] A. Botalb, M. Moinuddin, U. M. Al-Saggaf, and S. S. A. Ali, "Contrasting convolutional neural network (cnn) with multi-layer perceptron (mlp) for big data analysis," in *2018 International Conference on Intelligent and Advanced System (ICIAS)*, 2018, pp. 1–5.

[64] C. Goutte and E. Gaussier, "A probabilistic interpretation of precision, recall and f-score, with implication for evaluation," in *European conference on information retrieval*. Springer, 2005, pp. 345–359.

[65] A. Tharwat, "Classification assessment methods," *Applied Computing and Informatics*, vol. 17, no. 1, pp. 168–192, 07 2020. [Online]. Available: https://doi.org/10.1016/j.aci.2018.08.003

[66] K. Simonyan, A. Vedaldi, and A. Zisserman, "Deep inside convolutional networks: Visualising image classification models and saliency maps," *arXiv preprint arXiv:1312.6034*, 2013.

[67] V. Chamola, V. Hassija, A. R. Sulthana, D. Ghosh, D. Dhingra, and B. Sikdar, "A review of trustworthy and explainable artificial intelligence (xai)," *IEEE Access*, vol. 11, pp. 78 994–79 015, 2023.

<figure><img src="image_25.png" /></figure>

**SUDIP CHAKRABARTY** (M'24) received the B.Tech. degree in Computer Science and Engineering from the Kalinga Institute of Industrial Technology (KIIT), Bhubaneswar, India, in 2025, under a fully funded Government of India scholarship. He is currently a Research Scholar with the School of Computer Engineering, KIIT. His research interests include signal processing, explainable and trustworthy AI, computer vision, and medical AI applications. He has worked on projects involving

audio classification, speech emotion recognition, respiratory sound analysis, and multimodal explainability for deep learning. He has authored several conference papers and has served as a reviewer for multiple IEEE conferences. He was a recipient of the Best Paper Award at the IEEE ICICKE Conference in 2025.

<figure><img src="image_26.png" /></figure>

**PAPPU BISHWAS** received the B.Tech. degree in Computer Science and Engineering from the Kalinga Institute of Industrial Technology (KIIT), Bhubaneswar, India, in 2025, under a fully funded Government of India scholarship. He works as a Software Engineer at Chubb and as a Research Scholar at the School of Computer Engineering, KIIT, focusing on computer vision, signal processing, and explainable AI. His research encompasses

audio classification, speech emotion recognition, respiratory sound analysis, and multimodal interpretability in deep learning. He has contributed to several conference publications and was awarded the Best Paper Award at the IEEE ICICKE Conference in 2025.

<figure><img src="image_27.png" /></figure>

**DR. MAINAK BANDYOPADHYAY** received the Ph.D. degree in GIS and Remote Sensing from the Motilal Nehru National Institute of Technology -Allahabad, in 2018. He is currently serving as an Assistant Professor in the School of Computer Engineering, KIIT Deemed to be University, India. He is also the co-coordinator of the Cognitive Data Science Group in the School of Computer Engineering, KIIT University. His research interests include computer vision, audio processing, and deep

learning applications. He is the reviewer of many reputed Springer, Elsevier, and Taylor & Francis journals.

<figure><img src="image_28.png" /></figure>

**DR. JÉRÉMIE SUBLIME** received the Ph.D. degree in Applied Computer Science from the University Paris-Saclay, France, in 2016. Since 2016, he has been an Associate Professor with ISEP, Paris, France, where he is a member of the DaSIP Team (Data Science, Signal and Image Processing) and currently serves as the Head of the Data Science and Mathematics department. His research interests include machine learning and data science, with a focus on unsupervised learning,

image processing (satellite and medical), clustering paradigms (multi-view, collaborative, and alternative clustering), change detection, and unsupervised neural networks.

<figure><img src="image_29.png" /></figure>