The Omni-Expert: A Computationally Efficient Approach to Achieve a Mixture of Experts in a Single Expert Model
Authors: Sohini Saha
Mezisashe S Ojuba
Leslie M. Collins
Boyla Mainsah
Venue: NeurIPS 2025 poster ()
Keywords: speech enhancement
mixture-of-experts
omni-expert
feature transformation
cochlear implants
Primary area: applications

Abstract: Mixture-of-Experts (MoE) models have become popular in machine learning, boosting performance by partitioning tasks across multiple experts. However, the need for several experts often results in high computational costs, limiting their application on resource-constrained devices with stringent real-time requirements, such as cochlear implants (CIs). We introduce the Omni-Expert (OE) – a simple and efficient solution that leverages feature transformations to achieve the 'divide-and-conquer' functionality of a full MoE ensemble in a single expert model. We demonstrate the effectiveness of the OE using phoneme-specific time-frequency masking for speech dereverberation in a CI. Empirical results show that the OE delivers statistically significant improvements in objective intelligibility measures of CI vocoded speech at different levels of reverberation across various speech datasets at a much reduced computational cost relative to a counterpart MoE.

The Omni-Expert: A Computationally Efficient Approach to Achieve a Mixture of Experts in a Single
Expert Model
Sohini Saha?
Mezisashe S. Ojuba²
Leslie M. Collins"
Boyla O. Mainsah 1+
Duke University, Durham, NC, USA; Howard University, Washington, DC, USA
Abstract
Mixture-of- of-Experts (MoE) models have become popular in machine learning, boosting performance by partitioning tasks across multiple experts. However, the need for several experts often results high computational costs, limiting their application resource-constrained devices with stringent real-time requirements, such as cochlear implants (Cls). We introduce the Omni- Ammi-EEpert (OE) simple and efficient solution that leverages feature transformations to achieve the 'divide- und-conquer' functionality of a full MoE ensemble single expert model. We demonstrate the effectiveness of the OE using phoneme-specific time- free-fryyy masking for speech dereverberation in a CI. Empirical results show that the OE delivers statistically significant improvements in objective intelligibility measures CI vocoded speech at different levels reverberation across various speech datasets a much reduced computational cost relative a counterpart MoE.
Introduction
Lixture-of-Experre (MoH) models 1 2 have emerged powerful and flexible architectures for machine learning (ML) tasks that require specialized fine-tuning of complex tasks, such as language modeling and computer vision. However, MoE models pose computational challenges, in terms of power consumption, processing capability, latency and memory capacity, as the number of experts increases. Hardware considerations devices, especially for applications with real-time inference constraints, such as auditory prostheses. Thus, there is a need for lightweight adaptations of MoE models or alternative strategies that preserve performance while meeting resource limitations.
highly relevant for deployment in resource-constrained edge
In this work, we focus on the cochlear implant (CI), a medical device that restores hearing to individuals with profound hearing loss by converting sound to electrical pulses that directly stimulate an impaired cochlea. Most CI users generally have good speech understanding in quiet conditions; however, CI users struggle to understand speech in challenging acoustic environments with noise and reverberation [3] Some modern hearing aids now incorporate deep neural networks (DNNs) either for acoustic scene selection to (de)activate specific features, to control channel-specific gains for noise reduction and for separating noise and speech, enabled by more powerful on-chip processors, e.g., 4... DNNs have been investigated mostly for speech denoising in CIs [7 however, to our knowledge, no DNNs have been deployed in current commercially available CI sound processors. Unlike hearing aids that primarily function to amplify sound, CIs have higher functional burden of cewnustic-to-lletttr signal conversion, wircless signal transmission and electrical stimulation. In addition, real-time CI sound processing must causal with time delays that are within tolerable limits of audiovisual asynchrony for CI users; preferably <10 [1]
Conresponding author: boyla.mainsahe @ duke.edu
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Real-time speech enhancement models in CI systems must strike a balance between computational efficiency, performance and latency, making scaling with traditional MoE-based solutions potentially impractical. We propose novel network architecture that eliminates the need for multiple experts while retaining the specialization benefits of MoE models. Our contributions are as follows:
We introduce the Omni-Experi (OE), a computationally efficient alternative to MoE. The OE model achieves the effectiveness of full MoE ensemble in single network by using subtask-specific transformations to partition the feature space into distinct regions that correspond to a specialized expertise. This allows single expert network to maintain subtask specialization while operating with much reduced computational overhead.
We demonstrate the effectiveness of the OE in achieving superior performance relative to a counterpart MoE in task of phoneme-based speech dereverberation in Cls.
We conduct additional experiments to analyze the effect of feature transformation compo- nents on the performance of the OE model in the speech dereverberation task.
2 Related Work
Speech Enhancement in Cochlear Implants. Currently, CIs incorporate several signal processing solutions for noise management, such as beam-forming and signal-to-noise ratio (SNR)-basee noise reduction [3315;; however, there is no solution that directly addresses reverberation. Even in the absence of noise, individuals with auditory prostheses often struggle understand speech in reverberation [16]. which can negatively impact their quality of life; for example, learning experiences typically involve single talker scenarios in classroom/lecture hall. Strategies for noise reduction are not as effective in fully addressing reverberation due to differences in how noise and reverberation distort speech signals. Noise distortions additive and do not depend on the target speech, while reverberant distortions are delayed and attenuated copies of the target speech.
11 [8]
A common approach for speech enhancement is time-frequency (T-F) masking, where gain matrix (or mask) is applied to a T-F representation of the degraded speech to separate segments dominated by speech and acoustic distortions. An ideal mask is calculated based on measure of distortion of degraded speech signal relative to its clean counterpart. A typical ideal ratio mask (IRM) with mask values ranging from 0 to is computed according [19]
[s f)l²
0.5
ss f(t, S(t + (N(t.
S
(1)
Stt. () - M(, ((tt.)
where M(t, f) represents the ratio mask; S(t, and X (t, represent the clean and degraded speech signals, respectively; N represents the noise; and S(t, f) represents the enhanced signal.
Studies have shown ideal T-F masks for dereverberation improve speech intelligibility for hearing- impaired listeners 20 In real-world settings, the ideal mask needs to be estimated using only the reverberant speech signal. Traditional mask estimation algorithms for CI applications have relied on statistical features, such as kurtosis (25, linear prediction residuals (26), or estimated signal-to-distortion ratios (27), typically require room-specific tuning and not generalize well across varying acoustic conditions. Research on speech enhancement in CIs now utilizes DNNs that offer better robustness and generalization across diverse acoustic conditions. However, improvements with advanced ML models come the cost of increased computational complexity and latency.
main challenge with mitigating reverberation is how distinguish between wanted vs. unwanted speech with similar characteristics (i.e., reverberant reflections) from the same target speaker. In general, the performance of ML algorithms for speech dereverberation is frequency dependent, Mid- to high-frequency speech regions often include relatively long gaps in between phonemes, which allows late reverberant reflections to exponentially decay over time; thus, ML models can leverage features that capture this decaying pattern to better estimate and suppress unwanted reverberant speech. However, lower-frequency speech regions have more energy and relatively short gaps between phonemes, so late reverberant reflections are often interrupted by the next phoneme before

---

Subtask-specife transformedd feature, En
Transformation
b
Single Expert
b
Gating
Expert
Expert
Expert N
Gating network
Scale factor
Shift
nelwwrk
Feature Transformation Blors
Encoder,
Input feature,
(a) Mfxture-of-Expee Experts
(b) Omni-
Omni-Expert Expert
Figure 1: Illustration of (a) the Mixture-of- (MoE) model and (b) our Omni- Eopi-Expert (OE) model given an input feature, x. The MoE model uses probabilities (p..) from gating network to weight outputs (fn)) from multiple expert networks. In contrast, the OE model uses a single expert network and subtask-specife feature transformations (z,) achieve the functionality of multiple experts. We train separate models to predict the scale and shift parameters for subtask-specific feature transformations and use lookup table based the subtask label during inference.
they can fully decay. Thus, it is more challenging to differentiate reverberant reflections from target speech in lower frequency speech regions because of more persistent, higher energy reverberation.
While speech signals exhibit high spectro-tempee variabilities, speech structure is generally pre- dictable, and this predictability can useful inform speech enhancement. The energy of phonemes tends to be concentrated in specific frequency regions (28). The presence of energy in other frequency regions during a phoneme utterance likely indicative of an acoustic artifact; thus, knowledge of the current phoneme can be leveraged during mask estimation better identify and remove acoustic distortions. Phoneme-based mask estimation models, where separate models are trained for different phonemes, have shown improved performance in speech denoising for automated speech processing applications [2 Chu et al. 33] developed phoneme- based mask estimation model for speech deneverberation in CIs based on a dense MoE (Figure Ta). Typically, speech enhancement models applied as preprocessing step prior to acoustic signal delivery to the CL. However, the model in [33] relies causal features extracted from within the CI processing framework, design that is feasible for real-time deployment in Cls.
Mixture-of Expperss. A key limitation of MoE approaches is the computational costs as the number of experts increases: in [33], the number of experts depends on the number of phonemes. In sparse MoE models 384, only subset of experts are activated during inference. In (39 40), the number of computations at inference in the MoE is reduced by merging experts, aggregating multiple expert parameters at inference time using gating weights. In sparse and merging of experts variants of the MoE model, multiple experts must still be trained, stored and maintained. To address this limitation, have developed novel technique that achieves multi-expert functionality in single network.
Conditional Computation. Prior works 41 42 have used conditional computation to improve model performance in multi-task problems. The MTFormer framework [41] employs multi-task learning architecture consisting of shared transformer- based feature extractor (encoder and decoder), followed task-specific branches for specialization. In conditional batch normalization [42), multilayer perceptrons are trained to learn additive adjustments to batch normalization scaling and shifting parameters of a pretrained convolutional neural network based on an external conditioning vector (e.g., a language embedding). Our approach applies conditional feature transformations directly to input features to achieve subtask specialization in single network, and feature transformation parameters are learned jointly with the expert network.
3 The Omni-Expert (OE)
Our goal is to enable a single expert model exhibit subtask-specific expertise based on the input features of the associated subtask. In sparse MoE models, routing mechanism directs inputs to

---

the appropriate expert or set of experts. In contrast, our approach is to encode subtask selection for specialization implicitly in the feature space. To achieve this, we apply learned subtask-specific transformations that create homogenous features within a specific subtask and distinct features across subtasks. The OE model architecture, illustrated Figure 1b, consists of three core components:
A feature transformation block apply subtask specific feature transformations based on the subtask label.
A single expert network, which processes the transformed input features and performs the target task (in this case, mask estimation for speech dereverberation).
A gating/routing network, which is used to weight the outputs produced by applying the single-expert network to the transformed features:
0 Ú- Σ (x)
(3)
I:N
where x is the input feature; Pn (x), is the gating network probability of subtask n; E(z,) is the subtask-specific output from the single expert model E' based on the transformed feature Zn.
Instead of a simple linear transformation, an affine transformation offers greater flexibility by allowing shift in the origin to better align the feature distribution for each subtask while preserving discrim- inability across subtasks. For a sparse affine transformation matrix, we restrict linear operations to scale transformations. The subtask-specinf transformed feature Zn is defined as:
Z A b,
(4)
where x is the input feature; A. and b, represent scale and shift transformations, respectively, for subtask n. For scale, A, is diagonal matrix, which simplifies element-wise multiplication.
4 Methods
We demonstrate the efficiency of our OE over MoE in the task of real-time speech dereverberation in Cls. The CI processing pipeline was implemented using the Nucleus MATLAB Toolbox [43). All models were implemented in Py Torch [4] on an NVIDIA Titan V GPU.
4.1 Reverberation Model
The reverberant signal is modeled as the convolution of the clean, anechoic speech signal with room impulse response (RIR). For proper time alignment with the delayed reverberant signal, the direct path component of the reverberant signal is used as the reference clean signal; see Appendix A.
4.2 Experimental Settings
Datasets. Speech utterances used for training were a randomly selected 8000-utterance subset (approximately 28 hours) of the 100-hour LibriSpeech corpus [45]. Recorded RIRs used for training
were from the Brno University of Technology @ FIT Reverberation Database [46]. Speech utterances
used for testing were from speech datasets that commonly used in listening studies: Hearing In
Noise Test (HINT) [47]; and the City University of New York (CUNY) Male and Female datasets 48 Recorded RIRs used for testing were selected from the Aachen Impulse Response database (49) to represent diverse rooms: office, stairway, lecture hall and church (Appendix A.
Feature Extraction. Causal T-F features were extracted from speech signals following the Advanced
Combination Encoder (ACE) strategy based on the Nucleus CI system [50] 43 The acoustic signal is segmented into 8 ms frames with a 2 overlap, processed via short-time Fourier transform yielding 65 frequency features, and log-compressed to reduce dynamic range. The log-compressed spectral features were used as inputs to both the phoneme classification and mask estimation models. Feature normalization was applied using the global mean and variance calculated from the training set.
Phoneme Label Extraction. The phoneme classes are 39 standard American English phonemes 28 and nonphoneme class for silent gaps. Phoneme labels were generated from the direct path signal using forced alignment based the LibriSpeech recipe in the Kaldi speech recognition toolkit [5]

---

and phoneme time stamps were converted into Cl-based time units; example framewise phoneme labels are provided in Appendix A.2 A one-hot vector was used encode phoneme labels.
Model Architecture. For real-time feasibility in Cls, we use lightweight recurrent neural networks. The models include: i) single layer of 123 unidirectional long short-term memory (LSTM) units [52]: and ii) a single layer of 1 17 unidirectional gated recurrent units (GRU) [53] with a multi-head attention (A) layer [54]. The outputs of the GRU model following layer normalization are fed into the multi-head attention module (A), a residual connection around the attention block, followed by layer normalization (to stabilize gradients and accelerate convergence). The attention outputs are fused with the original GRU hidden state outputs via element-win multiplication so that the final representation retains both the sequential encoding and the learned contextual weighting.
Phoneme Classifier. Each model is followed by a fully connected output layer with 40 sigmoidal units, corresponding to the 40 phoneme class labels (55) Model parameters were randomly initialized from a uniform distribution with range [-0.1, 0. Training was performed using stochastic gradient descent with a learning rate of le-5 and momentum of 0.9, Each training batch consisted of single speech utterance split into 2-second non-overlapping segments to accommodate memory constraints and maintain sequence continuity. The model was optimized using cross-entropy loss. Training was terminated when the validation loss was unchanged for 10 consecutive epochs.
Mask Estimation Models. Models were trained to minimize the T-F signal loss function:
1 -F L = TF
(5)
where X(t, f) represents the magnitude spectrum of the reverberant signal: M (t,, represents the estimated mask; (t,, f) represents the ideal mask; and T and F represent the number of time and frequency bins, respectively.
Phoneme Independent. Each model is followed by fully connected layer, followed by an output layer with 65 sigmoidal units, representing the 65-dimensional IRM values 33 The phoneme independent model was trained on the entire training dataset.
Phoneme-based MoE. The model consists of the phoneme classifier as the gating network and 40 expert networks for phoneme-specific mask estimation. Each expert network is structurally identical to the phoneme independent model but is trained only data corresponding to its associated phoneme group. The phoneme classifier probabilities are used to weight the predictions of the phoneme-spen specific mask estimation models to get the final estimated mask.
Phoneme-basee OF The single expert model structurally identical to the phoneme independent model. Input features are transformed based on a phoneme-specific transformation:
Zn Ø x + b..
where zn is the transformed feature vector; x is the input feature vector; a, and b, are the scale and shift factor vectors, respectively, for phoneme n; and ( represents element-wise multiplication.
The transformation parameters are predicted by two separate multilayer perceptrons (MLPs) with the one-hot phoneme encoding as input. Both MLPs have an input size of 40 (number of phoneme classes) and an output size of 65 (matching the input feature size). The scale MLP uses ReLU activation, while the shift MLP uses LeakyReLU introduce nonlinearities [56]. The transformation parameters are precomputed after training and stored for all 40 phonemes, effectively reducing the process during inference to a lookup table.
Model Training. The phoneme independent model was initialized using weights drawn from a uniform [-0.1, 0. distribution
[57]. The phoneme-specific mask estimation networks of the MoE and OE models were initialized using pre-trained weights from the phoneme independent model. Models were trained using the Adam optimizer
(58) with learning rate of le-3 and momentum coefficients of 0.9 and 0.999. Training batches consisted of 16 speech utterances segmented into 2 second chunks. Training was terminated when no improvement was observed for 10 consecutive epochs.
4.3 Performance Evaluation
CI vocoded speech was generated by acoustic signal resynthesis from CI electrodograms (CI electrical stimuli patterns) with sine wave vocoder and used to predict speech intelligibility in CI users.
5

---

1.0
HINT
1.0
CUNYFemale
Tue dass probalility Frediited class proburiiity
Fiu cldss probutiiity
0.8
cless protabillty
Prbbabiity
Probabbitty
0.6
0.4
0.4
0.2
0.2
0.0
0.0
(a) the boy broke the wooden wencenfence
(b) have you yet
Figure 2: Framewise probabilities of true and predicted phoneme classes of reverberant speech utterances from (a) HINT and (b) CUNY-Female datasets using the GRU+A classifier. True class labels are annotated on the a-axis. Shaded regions indicate incorrect predictions.
Objective Intelligibility Metrics. Intelligibilityy of CI vocoded speech predicted using objective metrics that have been shown to be predictive of speech intelligibillity in CI listening studies [59] the speech-to-revenee rreerrrrrteverrriiionnn modulation energy ratio for CI users (SRMR- CI) [60] 61;; and the short-time objective intelligibility (STOI) metric [62] with the direct path signal used as the reference signal [63] The STOI metric developed for evaluating speech intelligibility in noise and excludes frames with silent gaps in clean speech. Since reverberation occurs in silent speech gaps, the removal of reverberant reflections in the same silent speech gap will not be captured by the STOI metric. The SRMR metric was originally developed for evaluating speech intelligibility in reverberation for normal hearing listeners [60], and later adapted for CI users (SRMR-CL by using the CI filterbanks 61]. Thus, the SRMR-CI metric provides more reliable speech intelligibility predictor for CIs in reverberation. We include STOI for completeness and to facilitate comparison with prior literature.
Benchmarks. Chu et al. 33 showed improvements in SRMR-CI and STOI scores, as well as intelligibiliit of CI vocoded speech in normal hearing listeners, with phoneme-based MoE model relative to a (conventional) phoneme independent mask estimation model. The objective here is to demonstrate that the OE model achieves minimum similar performance the MoE model with much reduced complexity. Oracle conditions include mask estimation with ideal phoneme knowledge (i.e., perfect phoneme classification), the IRM and the direct path signal.
Statistical Analysis. Statistical tests were performed R. two-way repeated measures ANOVA performed with within-utteranee fixed factors of room and mask type, interaction between mask type and room, and a random factor of speech utterance. Mauchly's ' test [64] was used to adjusted using the Greenhouse-Geri
assess the sphericity assumption; degrees of freedom correction where necessary. For statistically significant main effects or interactions, post-hoe pairwise comparisons were performed using estimated marginal means with Tukey's multiple comparisons correction, All tests were conducted at a significance level of 0.05.
Results
5.1
Phoneme Classification
Figure 2 show frame-wise phoneme classifier predictions using the GRU+A classifier applied to sample reverberant speech utterances. Frame-wise class balanced phoneme classification accuracies summarized in Table The phoneme distributions, classification confusion matrices and additional framewise classification results are provided in Appendix A.5
Table 1:
Phoneme Classification Accuracies (%) Across Test Datasets
Long Short-term Memory
Gated Recurrent Unit Attention
Dataset
Church
Office
Lecture
Stairway
Church
Office
Lecture
Stairway
CUNY-Female
20.98
26.26
25.41
27.65
24.30
35.29
32.89
35.91
CUNY-Mals
20.11
26.63
23.41
25.36
27.61
39,52
34.11
36.85
HINT
22.44
31.39
28.38
31.2
33.02
47.82
43.39
46.78

---

PI-GRU+A
MOEP-GRU+A
MOE--GRU+A
OER-GRU+A
OE*
Church
Office
Lecture
Stairway
Loss Values
1250 HE
40
60
1250 H2
40
60
1250 H
40
60
1250
H
40
60
Frequency Bin
Figure 3: Average signal loss across frequency bins with phoneme independent mask estimation and mask estimation with mixture-of-expert (MoE) and Omni-expert (OE) models with predicted (p) and ideal (k) phoneme knowledge using gated recurrent unit + attention (GRU+A) models.
5.2 Mask Estimation
Figure 3 shows the mean signal loss across frequency bins to visualize the frequency-dependent impact of mask estimation. In general, signal loss is highest in lower frequency regions (< 1250 Hz), reflecting the difficulty in mitigating low frequency reverberation. Phonemes typically range from 70-200 [65] 66]. making phoneme classification based
frame hard task. Even low accuracies Table 1). predicted phoneme knowledge is still beneficial to mask estimation, with a progressive increase in performance from phoneme independent to MoE OE models. Phonemes with similar fime-fre characteristics are likely to be confused with each other (Appendix (A.5). The weighting of phoneme-specific masks reduces the impact of phoneme misclassifications. With known phonemes, the OE provides a higher performance upperbound than the MoF. This indicates that encoding subtask-specific cues via feature transformations is more effective for specialization vs. specialized experts with subtask partitioning of the original feature space.
5.3
Objective Speech Intelligibility
Sample electrodograms are shown in Figure 4a with annotations of target speech and (late) reverberant reflections; corresponding spectrograms are shown Appendix [A.3) Room-specific statistical results of SRMR-CI and STOI scores are shown in Figure 4b) summary statistics are provided in Appendix Aggregate statistical results are summarized in [Table 2 Performance trends of objective speech intelligibilig measures are generally consistent with those of mask estimation. The higher performance bound with the OE provides more robustness to the impact of phoneme prediction errors.
A.6
Table 2: Objective intelligibility scores (estimated marginal mean 95% confidence interval) for the reverberant signal (Rev), direct path signal (DP), and across different mask estimation methods: ideal ratio mask (IRM), phoneme independent model (PI), phoneme- based mask predicted by mixture-of- experts/Ommii-Exper with ideal phoneme knowledge (MoF*/ OF*), and using phoneme classifier probabilities (MoP'/ OF?). Bold indicates the highest performance among the non-oracle models.
Long Short- Memory (LSTM)
Metric
Rev
PI
MoE
MoF.
OFP
OF
IRM
DP
SRMR
1,302
1.733
1.744
1.841
1.794
1.938
2.187
2.447
-CI
+0.007
-0.009
#0.009
+0.009
+0.010
+0.009
=0.008
+0.009
0.719
0.797
0.807
0.822
0.807
0.836
0.972
1.000
STOI
+0.001
=0.001
10.01
+0.001
+0.001
+0.001
=0.000
+0.000
Gated Recurrent Attenttno (GRU+A)
Metrie
Rev
P1
MoE"
MoE
OF
OE
IRM
DP
SRMR
1.302
1.873
1.948
1.945
2.014
2,113
2.187
2.447
-CI
10.007
10.011
10.010
10.009
100011
10.010
10.008
10.009
0.719
0.812
0.829
0.833
0.829
0.850
0.972
1.000
STOI
10.001
10.001
10.001
10.001
10.001
10.001
10.000
10.000

---

Direct Path
Rev
IRM
PL-GRU-A
Rewer fhe ation Speech
Biorede
Bletrode
8
05
500
1030
1500
SCD
1000
1500
500
1000
1500
500
100M
1500
Time (md)
Time (m))
Time (ms)
Time (ms)
MoEP-GRUM
MoE"-GRU~A
OEP-GRU-A
OEN-GRUHA
Brerodt
Ueerode
UP
Destrdde
1903
520
18000
1500
1500
1000
1900
Time
Time
(a) Electrodograms the speech utterance "the boy broke the wooden fence." "
Rev
McEP.LSTM
-REP-GRUU+ MoE*-GRU+A
OEP-LSTM DE
OFP-GRU+A OE"-GRU-A
IRM
PI-LSTM
MoEXLLSMM
DP
PI-GRU+A
1.0
2.0
0.9
SRMR-CI
1TOV
1.5
0.8
1.0
0.7
0.6
church
office
lecture
stalrway
church
office
lecture
stairway
(b) Room-specific estimatedd marginal means and 95% confidence intervals of reeerbto-reverrerratin modulation energy ratio for cochlear implant users (SRMR-CI) and short-time objective intelligibility (STOI) scores.
Figure 4: (a) Example clectrodograms of a speech utterance and (b) room-specific statistical results of objective speech intelligibility measures of cochlear implant vocoded speech generated for direct path (DP), reverberant speech (Rev), enhanced reverberant speech after applying the ideal ratio mask (IRM) and estimated masks with the phoneme independent (PI) model, mixture-of-expeee model with predicted and known phonemes (MoFP/k) and Omni-Expert model with predicted and known phonemes (OEP/k) for long short-term memory (LSTM) and gated recurrent unit attention (GRU+A) networks.
No Transform
Scale Only
Shift Only
Scale + Shift
100
Component
-20
-00
SNE Component
Figure 5: Visualization of serooeem--peeifiie features from a subset of randomly selected phoneme frames (N 1000) of reverberant speech from the CUNY Female speech dataset in the stairway room. Column panels represent features: before applying transformations; with scale-only; shift-only; and scale shift transformations. Arrows indicate example of a visually discernible impact of a shift transformation on a phoneme cluster. t-distributed stochastic neighbor embedding (t-SNE) was used.

---

5.4 Ablation Analysis
The rest of the paper presents aggregate results. Room- specific results are provided in the Appendix.
Feature Transformation Type. The contribution of each feature transformation was assessed with isolated (i.c., shift only scale only) and combined (i.e., shift and scale) transformations, |Figure 5 shows visualizations of features with respective transformations, The scale transformation enhances the separability of phoneme-specific feature clusters, while the shift transformations adjusts the feature offsets for better alignment. Aggregate statistical results are summarized in Taale 3) Overall, applying scaling or shifting has significant impact on the objective speech intelligibility metries to similar extent relative the non-transformed features. However, the combined transformation yields the highest improvements SRMR-CI and STOI scores, Table
Table 3: Objective intelligibility scores (estimated marginal mean 95% confidence interval) with the Omni-Expert model with predicted phonemes (OEP) across different types of feature transformations, scale (Sc) only, shift (Sh) only, scale shift (default) and no transformation. Bold indicates the highest performance. LSTM, long short-term memory; GRU+A, gated recurrent unit + attention.
OFT-LSTM
OE OEP-GRU+A
Metric
None
Sh Only
Sc Only
Sc Sh
None
Sh Only
Se Only
Se + Sh
SRMR -CI
1.683
1.711
1.706
1.794
1.923
1.987
2.000
2.014
+0.009
+0.010
+0.009
=0.010
+0.011
#0.011
+0.011
=0.011
0.793
0.792
0.793
0.807
0.819
0.826
0.829
0.829
STOI
10.001
1.0.001
+.00011
1..001
100.001
10.001
10.001
1..001
Feature Transformation Position. We also investigated the impact of applying the feature transfor- mation different layer positions during mask estimation: prior the input of the model (default), after the hidden layer, and both the input and hidden layers. Aggregate results are summarized in Table 4) Overall, applying feature transformation least the input layer is more effective than applying the transformation only at the hidden layer.
Table 4: Objective intelligibility scores (estimated marginal mean +95% confidence interval) with the Omni-Expert model with predicted and known phonemes (OE*/k) across different feature transformation positions: after the hidden layer (H), prior to the input the model (I) (default) and both the input and hidden layers (I + H). Bold indicates the highest performance among the non-oracle models. LSTM, long short-term memory; GRU+A, gated recurrent unit attention.
OE"-LSTM
OE* -LSTM
Metric
H
I
I+ H
H
I H
1.764
1.794
1.805
1.863
1.938
1.947
SRMR-CI
+0.009
10.010
10.010
+0.009
10.009
-0.009
0.803
0.807
0.805
0.824
0.836
0.835
STOI
#0.001
10.001
10.001
10.001
#0.001
=0.001
-RU-GR
OE GGR+AA
Metric
H
I
I+ H
H
I
I+ H
1.367
2.014
2.004
1.387
2.113
2.073
SRMR-CI
+0.008
+0.011
+0.010
+0.006
+0.010
0.010
0.693
0.829
0,822
0.621
0.850
0.842
STOI
10.001
1.0.001
+0.001
10.002
10.001
10.001
5.5 Model Complexity
Model size and training times are shown in Figure 6) detailed values are provided in Ap- pendix A.4. The number of parameters and the computation load are obtained using the open- source piflops package 667] The OE model achieves comparable to superior performance with a much smaller model size and faster training time relative the MoE model. Each ex- pert in the MoE model is trained only phoneme-specifie data and the reduced amount of

---

training data per expert model results in a longer training time. In contrast, the OE model uses the full training dataset while still benefiting from sub-task specialization via the feature transformations. Note that the models for shift and scale factor estimation are only used dur- ing OF training: in this case, the mapping from phoneme label to feature transformation is deterministic, so only the subtask-specific transformation factors are needed during inference.
PI-LSTM
MoE-LSTM MAE-GRU+A
OE-LSTM OE-GRU+A
PI-GRU+A
6 Conclusion
Total #Parameters
1e6
Training Time (Hours)
12.5
We introduced the Omni-Expers high- performing and compute-efficient alternative to achieving mixture of expertise in the same model. By conditioning feature transformations on subtask, the OE model learns to partition the input space into subtask-specific regions, ef- fectively scaling the functionality of multiple ex- perts without incurring the added computational costs. The current OE configuration assumes the subtasks are known to determine the number of experts, as is the case with phonemes. Alterna- tive variants of the OE may be needed in other applications where the specialized subtasks are not as well-defined.
10.0
7.5
5.0
PC
nutd
Model
Model
Figure 6: Complexity of phoneme independent (PI), mixture-of-experts (MoE, N = 40 experts) and Omni-Expert (OE) models using long short- term memory (LSTM) and gated recurrent unit + attention (GRU+A) networks for speech derever- beration in cochlear implants. MoE and OE model training times are marked by the phoneme classi- fier (PC, same for MoE and ОЕ) and expert (E) networks. Training times based on Titan GPU.
While this work focused speech dereverber- ation in cochlear implants, real-world listening scenarios will also include combinations of reverberation and ambient noise (e.g., multi-talker babble, equipment noise, etc). Results with the current mask estimation models trained only on reverberant speech and applied speech in a variety of noisy reverberant settings are presented in Appendix [A.8] As expected, there is significant drop in performance across all models as the signal-to-nois ratio increases. The OE with known phonemes still outperforms the counterpart MoE, indicating that the learned subtask feature transformations are more robust to training/test data mismatch.
Limitations and Future Work. Performance trends observed with objective speech intelligibillity measures may not necessarily translate to speech understanding in real-world settings; the test datasets may not fully represent the variety in speech patterns and reverberant room conditions. Additional performance improvements can be obtained with alternative OE model architectures, more diverse datasets for training and advanced training techniques. A relatively lightweight OE model for speech enhancement is more practical for CI deployment and improvements in CI processor chip technology are expected over time. Further work needed evaluate real-time time feasibility and real-world utility in clinical studies with CI users.
Societal Impact. speech enhancement models to increase speech understanding for CI users in challenging listening environments, which can improve the quality of life of CI users. Real-time speech enhancement is also relevant for individuals with hearing aids and automatic speech processing applications that require minimal latency, such as live transcription. More broadly, the computational efficiency and scalability of the OE technique have the potential to address the computational bottleneck associated with MoE in other applications. The core architectural design of substituting multi-expert inference with mubtask-spccific transformations applied to single expert is in principle domain- In other applications, there is often latent structure among tokens, tasks, or embeddings (c.g., syntactic roles, semantic types, or class labels) and MoE have been applied at the embedding, token/sequence or task level. These can be used to define subtask- specific transformations or input conditioning, which suggests that our Omni- Expert approach could be applicable.
The OE provides promising solution for developing compact, performinf
Acknowledgments and Disclosure of Funding
This work was supported by a grant from the National Institutes of Health (R56DDC000267001A The NVIDIA Titan V GPU was donated by the NVIDIA Corporation. Ojuba was supported by
10

---

the Duke ECE Research Experience for Undergraduates Program. Mainsah received a Microsoft Research Faculty Fellowship.
References
[1] Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computation, 2.79-87, 1991.
[2] Michael Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computation, 6:181 1--114, 1993.
[3] Javier Badajoz- Davila, Jörg M Buchholz, and Richard Van-Hoesel. Effect of noise and rever- beration on speech intelligibility for cochlear implant recipients in realistic sound environments. The Journal of the Acoustical Society of America, 147(5):3538-33 1-1333333999 2020.
[4] S Raufer, P Kohlhauer, F Jehle, V Kühnel, M Preuss, and S Hobi. Spheric Speech Clarity proven to outperform three key competitors for clear speech in noise. Whitepaper, Phonak Field Study News, Sonova AG, retrieved from https://www. phonak. com/evidence, 2024.
[5] Charlotte T Jespersen, Lena Dieu, and Thipiha Rubachandran. Organic Hearing drives user preference for Intelligent Focus. Whitepaper, GN Hearing A/S, 2025.
[6] Sébastien Santurette and Thomas Behrens. The audiology of Oticon Mo Whitepaper, Centre for Applied Audiology Research, Oticon A/S, 2020.
[7] Enoch Hsin-Ho Huang, Rong Chao, Yu Tsao, and Chao-Min Wu. Electrodenet-a deep- learning-based sound coding strategy for cochlear implants. IEEE Transactions on Cognitive and Developmental Systems, 16(1):346 357, 2024. doi: 10.11097TDDS
[8] Agudemu Borjigin, Kostas Kokkinakis, Hari M Bharadwaj. and Joshua S Stohl. Deep learning restores speech intelligibility in multi-talker interference for cochlear implant users. Scientific Reports, 14(1): 13241, 2024.
[9] Nursadul Mamun and John H Hansen. Speech enhancement for cochlear implant recipients using deep complex convolution transformer with frequency transformation. /EEE/ACM Transactions Audio, Speech, and Language Processing, 2024.
[10] Tom Gajecki, Yichi Zhang, and Waldo Nogueira. A deep denoising sound coding strategy for cochlear implants. IEEE Transactions on Biomedical Engineering, 70(9):2000-279,, 2023.
[11] Clément Gaultier and Tobias Goehring- Recovering speech intelligibility with deep learning and multiple microphones in noisy-reverberant situations for people using cochlear implants. The Journal of the Acoustical Society of America, 155(6):3833-388 2024.
[12] Marcia J Hay-MeCut,heon, David B Pisoni, and Kristopher K Hunt. Audiovisual asynchrony detection and speech perception in hearing-impairee listeners with cochlear implants: prelimi- nary analysis. International Journal of Audiology, 48(6):321-333, 2009.
[13] Marian Jones, Chris Warren, Marjan Mashal, Paula Greenham, and Josie Wyss. Speech understanding in noise for cochlear implant recipients using spatial noise reduction setting in an off the ear sound processor with directional microphones. Cochlear Implants International, 24(6):311-324 2023.
[14] Jace Wolfe, Mila Morais, Erin Schafer, Smita Agrawal, and Dawn Koch. Evaluation of speech recognition of cochlear implant recipients using adaptive, digital remote microphone technology and speech enhancement sound processing algorithm. Journal of the American Academy of Audiology, 26(05):502-500 2015.
[15] Anja Kurz, Kristen Rak, and Rudolf Hagen. Improved performance with automatic sound management 3 in the MED-EL SONNET 2 cochlear implant audio processor. PLOS One, 17 (9)c00244466, 2022.
[16] Pavel Zahorik. Spatial hearing in rooms and effects of reverberation. In Binaural Hearing: With 93 Illustrations, pages 243-280. Springer, 2021.

---

[17] Frank Iglehart. Speech perception in classroom acoustics by children with cochlear implants and with typical hearing. American Journal of Audiology. 25(2):100-109, 2016. doi: 10.1044/
2016LAJA-155 15-00440
[18] Frank Iglchart. Speech perception in classroom acoustics by children with hearing loss and wearing hearing aids. American Journal of Audiology, 29(1):6-17, 2020. doi: 10.1044/2019
1MAA-19-0111
[19] Norbert Wiener. Extrapolation, Interpolation, and Smoothing of Stationary Time Series: With Engineering Applications. The MIT Press, 1949. doi: 10.7555//mippress/246..000 .0001.
[20] Nicoleta Roman and John Woodruff. Speech intelligibilityy in reverberation with ideal binary masking: Effects of early reflections and signal-to-noiss ratio threshold. The Journal of the Acoustical Society of America, 133(3), 2013. ISSN 0001-4966. doi: 10.1121/1
[21] Kostas Kokkinakis and Joshua S. Stohl. Optimized gain functions in ideal time- freeffeyyy masks and their application dereverberation for cochlear implants. JASA Express Letters, 1(8). 2021. ISSN 2691-1191. doi: 10.1 10.1121/1000005540.
[22] Eric W. Healy, Jordan L. Vasko, and DeLiang Wang. The optimal threshold for removing noise from speech is similar across normal and impaired hearing- a time-frequency masking study. The Journal of the Acoustical Society of America, 145(6), 2019. ISSN 0001-4966. doi: 10.1121/1.511228.
[23] Koning, Madhu, and Wouters. Ideal time-frequency masking algorithms lead different speech intelligibility and quality in normal-hearing and cochlear implant listeners. IEEE Transactions on Biomedical Engineering, 62(1), 2015. ISSN 1558-2531. doi: 10. D0.109//BME.20142235
[24] DeLiang Wang. On ideal binary mask the computational goal of auditory scene analysis. In Speech Separation by Humans and Machines, pages 181-197. Springer, 2005.
[25] Oldooz Hazrati, Jaewook Lee, and Philipos C. Loizou. Blind binary masking for reverberation suppression in cochlear implants. The Journal of the Acoustical Society of America, 133(3),
2013/03/01. ISSN 0001-4966. doi: 10. (0.1121/1.478989 47789911.
[26] Oldooz Hazrati and Philipos C. Loizou. Reverberation suppression in cochlear implants using a blind channel-selectinn strategy. The Journal of the Acoustical Society of America, 133(6),
2013. ISSN 0001-4966. doi: 10.1121/1.4804133
[27] Oldooz Hazrati, Seyed Omid Sadjadi, Philipos C Loizou, and John H L Hansen. Simultaneous suppression of noise and reverberation in cochlear implants using a ratio masking strategy. The Journal of the Acoustical Society of America, 134(5), 2013. ISSN 0001-4966. doi: 10.1 00.1121/14483888 4823839.
[28] P. Ladefoged and K. Johnson. A Course in Phonetics. Cengage Learning. 2014. ISBN 97813 13051771885
[29] Zhong-Qiu Wang, Yan Zhao, and DeLiang Wang. Phoneme-specific speech separation. 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2016. doi: 10.1009ICASPPPP2017777716
[30] Shlomo E. Chazan, S. Gannot, and J. Goldberger. A phoneme-based pre-training approach for deep neural network with application to speech enhancement. In IEEE Internationnal Workshop on Acoustic Signal Enhancement (IWAENC), 2016. doi: 10.109//WAEN.2221667700009
[31] Shlomo E Chazan, Jacob Goldberger, and Sharon Gannot. Speech enhancement with mixture of deep experts with clean clustering pre-training. In 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 716-720. IEEE, 2021.
[32] Aswin Sivaraman and Minje Kim. Sparse mixture of local experts for efficient speech enhance- ment. In Interspeech 2020, pages 4526 4530, 2020. doi: 10.21437/nntesppeech. 2aw.ch...222999000 2989.
[33] Kevin Chu, Leslie Collins, and Boyla Mainsah. Suppressing reverberation in cochlear implant stimulus patterns using time-frequency masks based on phoneme groups Proceedings of Meetings Acoustics, volume 50, 2022. doi: 10.1121/2.0001998.
12

---

[34] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, An- dré Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. In Advances in Neural Information Processing Systems, volume 34, pages 8583-8595. Curran Associates Inc., 2021. ISBN 9781713845393.
[35] Barret Zoph. Designing effective sparse expert models. IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pages 1044-104 1044. IEEE, 2022.
[36] Nan Du Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. GLaM: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pages 5547-5569. PMLR, 2022.
[37] Bo Li*, Yifei Shen, Jingkang Yang. Yezhen Wang, Jiawei Ren, Tong Che, Jun Zhang, and Ziwei Liu. Sparse mixture-of-experts are domain generalizable learners. In International Conference on Learning Representations, 2023.
[38] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017.
[39] Shwai He, Run-Ze Fan, Liang Ding, Li Shen, Tianyi Zhou, and Dacheng Tao. Merging experts into one: Improving computational efficiency of mixture of experts. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 14685-14691. Association for Computational Linguistics, 2023.
doi: 10. /6533
[40] Mohammed Muqeeth, Haokun Liu, and Colin Raffel. Soft merging of experts with adaptive routing. Transactions Machine Learning Research, 2024. ISSN 2835-8856. Featured Certification.
[41] Xiaogang Xu, Hengshuang Zhao, Vibhav Vineet, Ser-Nam Lim, and Antonio Torralba. MTFormer: Multi-task learning via transformer and cross-task reasoning. In European Conference on Computer Vision (ECCV), pages 304-321. Springer- Opringer-Verlag, 2022. doi: 00.1007797--3-01-19988222
[42] Harm De Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. Modulating early visual processing by language. Advances in Neural Information Processing Systems, 30, 2017.
[43] B Swanson and H Mauch. Nucleus Matlab Toolbox 4.20 software user manual. Cochlear Ltd, 2006.
[44] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-perfornance deep learning library. In Advances in Neural Information Processing Systems, volume 32, 2019.
[45] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR corpus based on public domain audio books. In JEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015. doi: 10. 10.119//CAS99:2115..77998
[46] Igor Szöke, Miroslav Skácel, Ladislav Mošner, Jakub Paliesek, and J. Černocký. Building and evaluation of a real room impulse response dataset. IEEE Journal on Selected Topics in Signal Processing, 13(4), 2018. ISSN 1932-4553. doi: 10. 10.1109//SSSP220192200 17522.
[47] Nilsson M. Soli SD, and Sullivan JA. Development of the Hearing in Noise Test for the measurement of speech reception thresholds in quiet and in noise. The Journal of the Acoustical Society of America, 95(2), 1994. ISSN 0001-4966. doi:
[48] Arthur Boothroyd, Laurie Hanin, and Theresa Hnath. A sentence test of speech perception: Reliability, set equivalence, and short term learning. Internal Report RCT10, City University of New York, 1985.
13

---

[49] Marco Jeub, Magnus Schafer, and Peter Vary. A binaural room impulse response database for the evaluation of dereverberation algorithms. 2009 16th International Conference on Digital Signal Processing, 2009. doi: 10.1 109//CDSP220995222122
[50] Andrew E. Vandali, Lesley A. Whitford, Kerrie Plant, Clark, and Graeme M. Speech perception as a function of electrical stimulation rate: Using the Nucleus 24 cochlear implant system. Ear and Hearing, 11:608-624, 2000.
[51] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukáš Burget, Ondrej Glembek, Nagendra Ku- mar Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, Jan Silovský. Georg Stemmer, and Karel Vesely. The kaldi speech recognition toolkit. In IEEE Workshop on Automatic Speech Recognition and Understanding, 2011.
[52] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, (8):1735-1780, 1997.
[53] Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. In Proceedings of SSST-8, Eighth Workshop Syntax, Semantics and Structure in Statistical Translation, pages 103-111, Association for Computational Linguistics, 2014.
[54] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[55] Kevin Chu, Leslie Collins, and Boyla Mainsah. A causal deep learning framework for classifying phonemes in cochlear implants. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6498-6502, 2021. 9413986.
doi: 10. D0.10/ASSSS339722..000
[56] Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted Boltzmann machines. In International Conference on Machine Learning, pages 807-814, 2010.
[57] Alex Graves and Jürgen Schmidhuber. Framewise phoneme classification with bidirectional Istm networks. In IEEE International Joint Conference on Neural Networks, volume 4, pages 2047-2052. IEEE, 2005.
[58) Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference for Learning Representations, 2015.
[59] Tiago H Falk, Vijay Parsa, Joao F Santos, Kathryn Arehart, Oldooz Hazrati, Rainer Huber, James M Kates, and Susan Scollie. Objective quality and intelligibility prediction for users of assistive listening devices: Advantages and limitations of existing tools. IEEE Signal Processing Magazine, 32(2):114-12) 2015.
[60] Tiago H. Falk, Chenxi Zheng, and Wai- Yip Chan. A non non-intrusive quality and intelligibility measure of reverberant and dereverberated speech. IEEE Transactions on Audio, Speech, and Language Processing, 18(7):1766-111 1774, 2010. doi: 10. 10.1109/AASL.220022522222
[61] João F Santos and Tiago H Falk. Updating the ERMR-CI metric for improved intelligibility prediction for cochlear implant users. /EEE/ACM Transactions on Audio, Speech, and Language Processing. 22(12):117-22206 2014.
[62] Cees H Taal, Richard C Hendriks, Richard Heusdens, and Jesper Jensen. An algorithm for intelligibilltt prediction of frequency weighted noisy speech. IEEE Transactions on Audio, Speech, and Language Processing. 19(7):2125-2)) 2136, 2011.
[63] Lidea K. Shahidi, Leslie M. Collins, and Boyla O. Mainsah. Objective intelligibility mea- surement of reverberant vocoded speech for normal-hearing listeners: Towards facilitating the development of speech enhancement algorithms for cochlear implants. The Journal of the Acous- tical Society of America, (55(3)221112215 2024. ISSN 0001-4966. doi: 10.1121/100022585..
[64] John W. Mauchly. Significance test for sphericity of a normal n-variate distribution. Annals of Mathematical Statistics, 1 11:204-009, 1940.
14

---

[65] Gordon E Peterson and Ilse Lehiste. Duration of syllable nuclei in English. The Journal of the Acoustical Society of America, 32(6):693-703, 1960.
[66] Noriko Umeda. Consonant duration American English. The Journal of the Acoustical Society of America, 61(3)8866-858 1977.
[67] Vladislav Sovrasov. ptflops: A flops counting tool for neural networks in Pytorch framework, com com////orrra////gggs-ccooeeeee pytorch
2018. URL https:
[68] Manfred R Schroeder. New method of measuring reverberation time. The Journal of the Acoustical Society of America, 37(3):409-41,, 1965.
[69] Heiner Löllmann, Emre Yilmaz, Marco Jeub, and Peter Vary. An improved algorithm for blind reverberation time estimation. In Proceedings of International Workshop on Acoustic Echo and Noise Control (IWAENC), pages 1-4, 2010.
[70] Patrick A Naylor and Nikolay Gaubitch. Speech dereverberation. Business Media, 2010.
Springer Science
[71] Camegii Mellon University. The CMU Pronouncing Dictionary, 1993. URL http://www speech. cs. cmu www.eee.....i. citp/in/gii///////// bww.ddiiiimmmcittctccccccm.c cmudict
Version 0.7b.
[72] Joachim Thiemann, Nobutaka Ito, and Emmanuel Vincent. The diverse environments multi- channel acoustic noise database (DEMAND): A database of multichannel environmental noise recordings. In Proceedings of Meetings on Acoustics, volume 19, AIP Publishing, 2013.
[73] Tim Fischer, Marco Caversaccio, and Wilhelm Wimmer. Multichannel acoustic source and image dataset for the cocktail party effect hearing aid and implant users. Scientifie Data," (1):440, 2020.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly describe the main contributions of the paper, proposing a novel lightweight alternative to achieving the functionality a Mixture-of-Expere (MoE). The primary contributions Expert (OE) model which overcomes the computational overhead of MoE using a simple feature transformations. We provide experimental results from the task of real-time speech dereverberation in cochlear implants to support these claims.
follows: We introduce Omni-
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: the conclusion section of the paper, we have provided detailed explanation of the method's 's limitations, section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
16

---

3. Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer[[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer:[ Yes]
Justification: The OE architecture is fully described in section 3 Details of methods and experiment settings to reproduce results are provided in subsection 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
Depending on the contribution, reproducibility
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
17

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [No]
Justification: The code is proprictary and not publicly released. All the datasets and software packages used in the paper are properly cited. The experimental settings, model configurations and model complexities needed to reproduce all experimental results have been described in subsection 4.2 Audio files of vocoded speech signals cannot be shared due to copyright restriction of source speech datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the training and testing dataset used, feature extraction process, phoneme alignment tools, model architectures, optimizer, learning rate, loss functions and activations used have been described in Iubseectio 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Statistical tests are detailed Figure 4.3
We report estimated marginal means and 95% confidence intervals of performance measures in subsectiio 5.2) Statistical significance is indicated if confidence bars do not overlap. [Tabl 2 shows mean and confidence intervals of the results that support the main claims.
18

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All models implemented in PyTorch and trained NVIDIA Ti- tan V GPU. The paper clearly specifies the compute resources required to reproduce the experiments, including the training time and model size shown in subsection 5.5)
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:p//...earppp.p.o/ppp.//stta
Answer: [Yes]
Justification: The research is consistent with the NeurIPS Code of Ethics. The paper provided comprehensive details for reproducibility. discussed potential societal impacts, limitations and future directions.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
19

---

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses positive societal impacts and limitations in section 6 particularly how the Omni- Expi-t model could benefit CI users by making the technology suitable for edge devices.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All existing resources used this paper, including the comparative experiments and the datasets, have been properly cited, subsection 4.2
20

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:[Yes [es]
Justification: The model architecture of our Omni Omni-Expert for the task of speech dereverbera- tion and the training details are presented in subsection 4.2
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
21

---

Answer:
[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
The paper does not use LLMs for the core method development.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy htttg:///aaarpssscc////ooeeeeeee for what should or should not be described.
22

---

Appendix
Table of Contents
A Technical Appendices
24
A.1
Reverberation Model and Room Impulse Response (RIR) Characteristics
24
A.2
Frame-wiss wise Phoneme Labels
25
Example Spectrograms and Electrodograms LST models
26
Complexity Analysis
28
A.5
Phoneme Analysis
28
A.6
Room-specific Mask Estimation Results
31
A.7
Additional Ablation Analysis-I.ST LTM models
33
A88
Robustness in Noise
38
23

---

A Technical Appendices
A.1 Reverberation Model and Room Impulse Response (RIR) Characteristics
The reverberant signal Teev(t) (t) is modeled as the convolution of the clean, anechoic speech signal, (t), with the room impulse response (RIR), h(t):
Zzev (t) s(t) h(t)
To isolate the direct-path component, the RIR is decomposed into two parts:
Trev (t) s(t) haireat (t) s(t) Prevert (t)
Tairect (t) + s(t) * (rrever(())
(8)
I(t) Frev (t) Jedieea (1)
(9)
where (trea(t) represents the RIR function for direct path (and early reflections); (tever( is the RIR of the remaining late reverberation; Tdirear represents the direct path signal; and ((t) represents the late reverberant reflections, i.e., the difference between the reverberant and direct path signals.
Table AT lists characteristies recorded RIRs of four rooms the Aachen Impulse Response database 49] used for testing. RIRs were selected from office, lecture, stairway, and a church. For the stairway and the church, RIRs were selected at an azimuth of 90 degrees, where the source and receiver are directly facing each other. RIRs were filtered using anti-aliss anti-aliasing filter and then downsampled from 48 to 16 kHz before convolution with anechoic speech stimuli. Reverberation times (RT608) were calculated using the Schroeder method 68 using the code provided by [69]. The torecttt--rvveberantt ratios (DRRs) of the recorded RIRs were calculated using [70].
Table A1: Room impulse response characteristics of test room conditions. (T) reverberation time; DRR; Direct-to-reverberant ratio.
Source Receiver Distance (m)
Dimensions (L x W H) (m)
Dataset
Room
(8)
DRR (dB)
Office
5.0 6.4 x 2,9
3.0
0.6
0.4
Aachen Impulse Response (AIR)
Lecture
10.8 x 10.9 3.15
5.56
0.9
-0.1
Stairway
7.0 x 5.2
3.0
0.9
1.6
Church
19.0 30.0
5.0
6.5
-0.6
24

---

A.2 Frame-wise Phoneme Labels
Direct Path
-40
-60
Frequency (kHz)
-80
-100
-120
-140
0y
0.5
1..5
Time (s)
Direct Path
Electrode
15
17
18
19
21
22
6lah
0y
dhah
0
500
1000
1500
Time (ms)
Figure A2.1: Example annotations of phoneme labels aligned cochlear implant time bins.
25

---

A.3
Example Spectrograms and Electrodograms LSTM models
Direct Path
Reverberant
40
Freqanny (eez)
100
-40
Time (S)
Time
Ideal Ratio Mask
Phoneme Independent
40
40
Frequency (dh1
100
100
-20
120
Time (S)
Time
Phoneme-rredicee-too
40
Frequency (8440
40
120
120
-40
140
0.5
1.5
Phoneme-Prediceed
Phoneme Known-OE
60
Frequency (4h1)
Frequency (ahh)
80
100
0.5
1.5
Time (5)
Time
Figure A3.1: Spectrograms of the speech utterance "the boy broke the wooden fence" generated for direct path speech, reverberant speech, enhanced reverberant speech after applying the ideal ratio mask and estimated masks with the phoneme independent model, mixture-of-experts (MoE) model with predicted and known phonemes, and Omni Amni-Exxert (OE) model with predicted and known phonemes.
26

---

Direct Path
Rev
0
Reverberation
Target Speech
Electrode
Electrode
10
10
15
15
20
500
1000
1500
0
500
1000
1500
Time (ms)
Time (ms)
IRM
PI-LSTM
5
Electrode
Electrode
10
10
15
20
20
500
1000
1500
500
1000
1500
Time (ms)
Time (ms)
MoEP-LSTM
MoE
0
5
Electrode
Electrode
10
10
15
20
20
500
1000
1500
0
500
1000
1500
Time (ms)
Time (ms)
OEP-LSTM
OEK-LSTM
5
5
Electrode
Electrode
10
15
15
20
500
1000
1500
0
500
1000
1500
Time (ms)
Time (ms)
Figure A3.2: Electrodograms of the speech utterance "the boy broke the wooden generated for direct path speech, reverberant speech (Rev), enhanced reverberant speech after applying the ideal ratio mask (IRM) and estimated masks with the phoneme independent model (PI), mixture-of - model with predicted and known phonemes (MoE*/k), and Omni-Expert model with predicted and known phonemes (OEp/A).
27

---

A.4
Complexity Analysis
Table A4.1: Summary of complexity of long short-term memory (LSTM) models used for speech
dereverberation in cochlear implants.
Model
Parameters
Training Time'
MACs (M)
Size (MB)
Phoneme Independent
108,225
2 hrs 58 mins
109.44
0.43
Phoneme Classifier (PC)
98440
3 hrs
99.63
0.39
Mixture of Experts (MoE)
40 108,225 PC
5 hrs 22 mins hr 57 mins
4377.6 + PC
16.51 PC
Omni-Expert (OE)
113555
109.45 + PC
0.45 PC
Expert
108.225
0.43
Shift Scale Factors
2,665 2,665
0.1 + 0.1
Shiff scale multilayer perceptrons are not deployed during inference: *NVIDIA Titan GPU
Table A4.2: Summary of complexity of gated recurrent unit + attention (GRU+A) models used for
speech dereverberation in cochlear implants.
Model
Parameters
Training Time+
MACs (M)
Size (MB)
Phoneme Independent (PI)
127946
hrs 43 mins
127.76
0.51
Phoneme Classifier (PC)
124996
hrs 15 mins
124.84
0.5
Mixture of Experts (MoE)
40 107947 + PC
10 hrs 47 mins
5110.58 + PC
19.52 + PC
Omni Omni-Expert (OE)
133276 PC
hr 21 mins
127.77 PC
0.53 PC
-Expert
127946
Shift Scale Factors
2,665 2,665
0.1 + 0.1
Shift scale multilayer perceptrons are not deployed during inference; !NVIDIA Titan GPU
A.5
Phoneme Analysis
A.5.1
Phoneme Distribution
Phoneme Distribution
40%
Distribution: Bottom-10
35%
CUNYFemale (N 720)
Phoneme Classes
30%
Distribution
CUNYMale (N -880))
2.0%
25%
1.0%
20%
IHINT (N =240)
0.0%
15%
Librispeech (Training
36
oy
20
Reference, N 8000)
10%
5%
881 mid
ib
ng
ch
Phonemes
Figure A5.1: Distribution of phoneme classes in the training and test speech datasets sorted by the frequency of phonemes in the training dataset. The silence class (sil) has been divided into two categories: silences occurring at the beginning and end of an utterance, denoted by sil sil_edee; and silences occurring in the middle of an utterance, denoted by sil mid. N is the number of speech files in each dataset.
28

---

A.5.2 Phoneme Classifier LSTM
HINT
probability
Prodicted class p probobility
Probability
(a) the boy broke the wooden fence
CUNYFemale
probabsiity
Predicted class probebility
probabiitty
(b) have you eaten yet
Figure A5.2.1: Framewise probabilities of true and predicted phoneme classes of reverberant speech utterances from (a) HINT and (b) CUNYFemale datasets using the long short-term memory (LSTM) classifier. True class labels are annotated on the r-axis. Shaded regions indicate incorrect predictions.
29

---

CUNYFomale
CUMtHole
1.0
0.6
0.6
Thu
Roold
-0.2
0.2
DACRSNG
PINT
1.0
Toue Labil
0.6
0.2
0.0
82588
Figure A5.2.2: Confusion matrices of phoneme predictions in datasets using long short-term
memory (LSTM) classifier. Phonemes are annotated by manner of articulation 71
A.5.3 Phoneme Classifier GRU+A
30

---

CUNYFomale
CUMrMale
0.6
0.6
Thon
-0.2
0.2
ren-ptnal
HINT
1.0
0.6
Tru LABaT
0.2
0.0
835
Figure A5.3.1: Confusion matrices of phoneme predictions with GRU+A phoneme classifier in test datasets across room conditions. Phonemes are annotated by manner of articulation 71]
A.6
Room-specific Mask Estimation Results
A.6.1
Signal Loss
Phoneme Independent
Phoneme-Predictu--ee
Phoneme- Prrretee-redicted--e
Phoneme-Known-EE
Stairway
1250
40
60
1250
20
60
12:5
20 40
50
1250
20
4.0
60
Frequency
Figure 16.1.1: Average signal loss values across frequency bins with phoneme independent mask estimation and mask estimation with mixture mixture-of-expere -ffeeperts (MoE) and Omni- Omni-Expert (OE) models with predicted and ideal phoneme knowledge using long short-term memory (LSTM) models.
A.6.2 Mask Estimation-LSTYy models
31

---

Table A6.2.1: Long short-term memory (LSTM) models. Mean 95% confidence interval of objective speech intelligibility scores for the reverberant signal (Rev), direct path signal (DP), and across different mask estimation methods: ideal ratio mask (IRM), phoneme independent model (PI), phoneme-based mask predicted by (MoE*/ OE*), and using predicted phoneme classifier probabilities (MoE% OE?). Bold indicates the highest performance among the non-oracle models.
with ideal phoneme knowledge
SRMR-CI
Church
Office
Lecture
Stairway
0.923
1,495
1.333
1.456
Rev
#0.005
+0.010
#0.008
+0.010
1.327
1.984
1.694
1.928
PI
#0.010
10.015
10.012
#0.015
1.351
1,956
1.731
1.939
MoE?
+0.010
+0.014
+0.012
+0.015
1.547
1.988
1.826
2.004
MoR
#0.012
10.014
10.014
#0.016
1.370
2.029
1.787
1.990
OF
10.010
10.015
10.013
10.016
1.616
2.077
1.956
2.105
OE
#0.013
+0.015
#0.015
#0.017
2.115
2.210
2.220
2.200
IRM
+0.014
#0.016
+0.016
+0.016
2.438
2.423
2.452
2.473
DP
+0.017
+0.018
+0.018
+0.017
STOI
Church
Office
Lecture
Stairway
0.684
0.746
0.700
0.746
Rev
+0.002
+0.002
=0.002
#0.002
0.771
0.814
0.779
0.823
PI
+0.002
+0.002
+0.002
+0.002
0.781
0.820
0.792
0.833
MoFP
0.0002
+0.002
+0.002
+0.002
0.806
0.831
0.804
0.845
MoE
#0.002
#0.002
+0.002
#0.002
0.780
0.823
0.795
0.831
OE
#0.002
10.002
10.002
#0.002
0.819
0.845
0.827
0.855
OE
+0.002
#0.022
#0.002
#0.001
0.975
0.970
0.974
0.969
IRM
+.0000
+0.001
+0.000
+0.001
1.000
1.000
1.000
1.000
DP
+-0.000
+0.000
+0.000
+00000
A.6.3 Mask Estimation-GRU44 models
32

---

Reverberant
Phoneme Predicted
Phaneme Phoneme-Known--O
independent
Phoneme-Known-MoE
Direct
CuNaFemale
CUNYMale
SRMM-CC Scores
(a)
1.0
0.6
church
a fffice
lecture
stairway
church
office
lecture
stairway
church
office
Tecture
stairway
Conditions
(b)
Figure A6.2.1: Objective intelligibility scores of speech from HINT, CUNYFemale, and Male datasets church, office, lecture, and stairway rooms. Results are shown for enhanced reverberant speech after applying estimated masks with the phoneme independent model, mixture of experts (MoE) model with predicted and known phonemes, Omni-Expert (OE) model with predicted and known phonemes, the ideal ratio mask, and the direct path signal. SRMR-CI, Speech-oorrevererratio modulation energy ratio for CI users; STOI, short-time objective intelligibility.
A.7 Additional Ablation Analysis-LSTM models
A.7.1 Transformation Type
33

---

Table A6.3.1: Mean (# 95% confidence interval) of objective speech intelligibility scores across different mask estimation methods: phoneme independent model (PI), phoneme- based mask predicted by mixture-of mfxuure-of-epperts///n experts/Omnii-Exppppe
with ideal phoneme knowledge MOE"), and using phoneme classifier probabilities (MoF//OEF) Results are shown for the + (GRU+A) model architecture aggregated across three test datasets in four room conditions. Bold indicates the highest performance among the non-oracle models.
SRMR-CI
Church
Office
Lecture
Stairway
PI- GRU+A
1.377 #0.014
2.113
#0.016
1.895 #0.016
2.016 10.018
MoEP GRU+A
+0.015
2.133 #0.016
1.987 #0.016
2.145 +0.021
OE GRU+A
1,500 +0.013
2.268 =0.017
2.059 #0.017
2.228 +0.019
MoE GRU+A
1.559 #0.017
1.971 10.015
2.117 #0.021
OE GRU+A
1.747 +0.014
2.252 +0.017
2.173 +0.017
2.280 +000199
STOI
Church
Office
Lecture
Stairway
PI GRU+A
0.771 #0.003
0.832 +0.003
0.804 #0.003
0.843
#0.002
MoD - GRU+A
0.774 +:0002
0.842 +0.002
0.801 #0.002
0.846 +0.002
OEP GRU+A
0.783 #0.002
0.847 =0.002
0.825 #0.002
0.860 =0.002
MoE* GRU+A
0.793 +-0.00
0.843 +0.001
0.804 +:0000
0.848 +0.000
OK -GRUA
0.826 +0.002
0.858 +0.002
0.845 +0.002
0.873 =0.001
Table A7.1: Room-specific Objective intelligibility scores (estimated marginal mean (+: 95% con- fidence interval)) with the Omni-Expert model with predicted phonemes across different types of feature transformations. Bold indicates the highest performance.
Specch-oo-reverberatioo modulation energy ratio for CI users (SRMR-CI)
Church
Office
Lecture
Stairway
None
1.302 (#00.00)
1.903 (#0.014)
1.647 (#0.012)
1.881 (+0.014)
Shift Only
1.278 (+000000)
1.968 (+0.015)
1.695 (+0.012)
1.903 (+0.015)
Scale Only
1.288 (+00009)
1.925 (+00014)
1.706 (+0.012)
1.906 (+000155
Scale + Shift
1.370 (+.0.010
2.029 (+.00015)
1.787 (=00))
1.990 (=001
Short-time objective intelligibility (STOI)
Church
Office
Lecture
Stairway
None
0.767 (+0.002)
0.811 (#0.002)
0.774 (=0.003)
0.821 (=0.002)
Shift Only
0.767 (+0.002)
0.809 (#0.002)
0.774 (=0.002)
0.815 (=0.002)
Scale Only
0.766 (+0.002)
0.810 (=0.002)
0.778 (=0.002)
0.819 (=0.002)
Scale + Shift
0.780 (+-0.002)
0.823 (+000002)
0.795 0t.00)
0.831 (+ 0.002)
34

---

Reverberant
Phoneme-Pred Predicted-OE (GRU+ (GRU+Attention)
Phoneme Independent
Phoneme-Knowe -noonn--E (GRU+Attention)
Phoneme Independent Attenttttn
Ideal Mask
Phoneme- Piieemee-redicte---ee
Direct Pathh
Phoneme-Known-OE
CUNYFemale
CUNYFemale
4.0
1.0
3.5
0.9
3.0
SRMR.
2.5
0.8
2.0
0.7
0.6
1.0
0.5
0.5
CUNYMale
CUNYMale
4.0
1.0
3.5
0.9
3.0
SRMR-
2.5
0.8
2.0
0.7
1.5
0.6
1.0
0.5
0.5
HINT
HINT
1.0
3.5
0.9
3.0
SRMR-
0.8
2.0
0.7
1.5
0.6
1.0
0.5
0.5
church
office
lecture
stairway
church
office
lecture
stairway
Figure A6.3.1: Boxplots objective speech intelligibility scores cochlear implant vocoded speech evaluated for three test datasets in all four room conditions using ratio masks with baseline LSTM and a GRU+Attention networks. Objective speech intellibility measures include speech-to-ree rorerttt-rrevrrattioonnn modulation energy ratio for CI users (SRMR-CI) and short-time objective intelligibility (STOI). Results are shown for direct path, reverberant speech, enhanced reverberant speech after applying the ideal ratio mask and estimated masks with the Phoneme Independent model and Omni-Ea Omni-Expert model (OE) with predicted and known phonemes.
35

---

No Transform
Scale Only
Shift Only
Scale + Shift
Companent 2
-00
USE 108
(a) CUNY Male
Transform
Scale Only
Shift Only
Scale Shift
Component
24
25
25
105
330
LNE componiet
-300
HINT
Figure A7.11: Visualization of phoneme-speciiee features from subset of randomly selected phoneme frames (N = 1000) of reverberant speech from the CUNY Male and HINT speech datasets in the stairway room. Column panels represent features: before applying transformations; with scale-only; shift-only: and scale shift transformations. Arrows indicate an example of visually discernable impact of shift transformation phoneme cluster. t-distributed stochastic neighbor embedding (t-SNE) was used.
none
shift only
scale only
scale + shift (ours)
CUNYFemale
CUNYMale
HINT
SRMR-CI Scores
2
(a)
1.0
STOI Scores
0.8
0.6
church
office
lecture
stainway
church
office
lecture
stainway
church
office
lecture
stairway
(b)
Figure 17.122: Boxplots of (a) SRMR- CI and (b) STOI scores evaluated for three test datasets in all four room conditions without any feature modulation and using three different feature modulation techniques: shift only, scale only, and scale+shift (default). Results are shown for Omni- Mani-EExert model with predicted phonemes.
36

---

A.7.2 Position
DE-I-Predicee (ours)
Preeiccdd
OE-H OE-H-Known
OE-I-Known (ours)
HE-lwH
CUNYFemale
CUNYMale
HINT
SRMR-CI Scores
(a)
1.0
STO Scores
0.8
0.6
church
office
lecture
stairway
church
office
lecture
stairway
church
office
lecture
stairway
(b)
Figure A7.2.1: Boxplots of (a) SRMR-CI and (b) STOI scores evaluated for three test datasets in all four room conditions using ratio masks for an LSTM network of layer. Results are shown for the Omni-Exper model with predicted phonemes.
37

---

Table A7.2: Performance across different feature transformation locations, Estimated Marginal Mean (# 95% Confidence interval). Bold indicates the highest performance among the feature transformation locations.
SRMR-CI
InsertionPoint
Church
Office
Lecture
Stairway
Hidden
1.364
1.988
1.750
1.952
Phoneme Preedicted -OE
Layer (H)
(+0.010)
(+00.014)
(+0.013)
(+0.015)
Input
1.370
2.029
1.787
1.990
Layer (I)
(=0.010)
(=0.015)
(#0.013)
(#00006)
1.385
2.039
1.801
1.996
I+H
(#00.100
(#0.011)
(#0.013)
(#000055)
1.544
2.030
1.849
2.028
Phoneme -nownn
H
(+00013)
(1:0.015)
(+0.014)
(+0.016)
1.616
2.077
1.956
2.105
-OF
(#0.013)
(=0.015)
(#0.015)
(#0.017)
1.643
2,076
1.960
2.109
I+H
(=0.013)
(=0.015)
(+0.0015
(=0.017)
STOI
Insertion Point
Church
Office
Lecture
Stairway
0.778
0.818
0.789
0.829
Phoneme -Predicted
H
(+0.002)
(+0.002)
(#0.002)
(#0.002)
0.780
0.823
0.795
0.831
-OE
I
(+-0.002)
(++0.002)
(+-0.002)
(+00002))
0.778
0.818
0.793
0.833
I ++ H
(+00002))
(+0.0022
(+00.002)
(+0.002)
0.808
0.833
0.810
0.845
Phoneme -nnwnn
H
(+0.002)
(+0.002)
(+0.002)
(=0.002)
0.819
0.845
0.827
0.855
-OE
I
(E0.000)
(+.0.002)
(+000002)
(+:00011
0.819
0.841
0.825
0.857
I+H
(#:0.002)
(=0.002)
(#:0.02)
(+0.001)
A.8 Robustness in Noise
Noisy- Russy-eeerberant Testing Conditions The test datasets were developed by adding noise from DEMAND (72) and Cocktail Party [73] noise datasets. Two different noise conditions were chosen from DEMAND Domestic and Public. Domestic noises include kitchen, living room, and washing machine noise environments, and Public noises include the interiors of cafeteria, restaurant, and busy subway station. Two-talker Babble (TTB) was selected from Cocktail Party dataset. We used speech from the HINT dataset. Noise added signal-to-noise (SNR) levels: -5, 0, 5, 10, 15, 20, and noisy speech was convolved with room impulse responses (RIRs) from office, stairway, lecture, and church room conditions.
A.8.1 Roomwise model performance L.STM
38

---

Noisy Reverberant
Phaneme Predicted-McE Phoneme-Krown-oo
Phoneme Predicted-Os Thowe---Koww-OE
Ideal Mask
Phonime Independent
Direct Path-Noisy Signal
TTB-Church
Domestic-Office
2.4
2.4
2.2
2.2
2.0
2.0
1.4
1.4
1.2
1.2
1.0
1.0
5
10
20
noise
5
15
20
No noise
TTB-Lecture
Public-Stairway
2.4
2.4
2.2
2.2
2.0
2.0
3RMR-C
1.2
1.2
1.0
1.0
0.8
0.8
-5
5
10
20
noise
5
15
20
No noise
SNR
SNR
(a) SRMR-CI
Noisy Reverberant
Ceeneme--redicted-Mae
Denneme-Predicte-OEE Phaneme-Knowee Known-OE
Ideal Mask
Phoneme Independent
Phoname-Knawn-Mr
Direct Path (Noley Signal)
TTB-Church
Domestic-Office
1.0
1.0
0.8
0.8
0.7
0.7
0.4
0.4
10
20
noise
15
20
No noise
TTB-Lecture
Public-Stairway
1.0
1.0
0.7
0.7
0.6
0.6
0.4
0.4
10 20
20 No moise
(b) STOI
Figure A8.1: SRMR-CI and STOI scores for HINT speech with noise conditions, Domestic and Public noises from DEMAND dataset 722 and Two-Talker Babble (TTB) from Cocktail Party dataset 73 convolved with office, stairway, lecture, and church room conditions (RIRs), respectively. Results are shown for unenhanced noisy reverberant speech, mask estimated using phoneme independent models, Omni-Expert model (OE), ideal
phoneme-spectne mixture-of-expert model (MoE), choneme--peciffe ratio mask (IRM), and the direct path (DP) of the noisy reverberant signal. Noise was added at SNR levels: -5, 0, 5, 10, 15, 20. Additionally, results are shown for no noise (only RIR) condition.
39

---

Table A8.1: Performance across different mask estimation methods. Estimated Marginal Mcan (# 95% Confidence interval) for unenhanced noisy reverherant (Noisy Rev) speech, mask estimated using phoneme independent (PI) model, phoneme-specific mixture-of-expert model (MoEP/<) and thoneme-speciff Omni-Expert model (OE"/K) with predicted/known phonemes, ideal ratio mask (IRM), and the direct path of the noisy reverberant signal ( (oiss))) across noise conditions (SNR in dB). Bold indicates the highest performance among the non-oracle models.
SRMR-CI
Model
-5
0
10
15
20
No noise
1.007
1,060
1.153
1.239
1.290
1.313
1.327
Noisy Rev
#0.015
#0.014
#0.015
#0.017
#0.018
=0.018
#0.018
1.329
1.431
1.555
1.649
1.702
1.725
1.812
PI
+0.022
+0.022
+0.023
+0.024
0..24
+0.024
#0.025
1.294
1.388
1.518
1.621
1.680
1.708
1.825
MoF
10.021
+0.020
+0.021
+0.023
10.024
10.024
#0.024
1.367
1.478
1.608
1.706
1.760
1.788
1.930
MoE
10.019
+0.020
+0.021
+0.023
+0.023
10.024
+0.023
1.295
1.387
1.522
1.634
1.701
1.734
1.848
OF
10.021
+0.020
#0.021
#0.023
#0.24
10.025
#0.025
1.434
1.547
1.686
1.790
1.850
1.881
2.011
OE
+0.021
10.021
#0.022
+0.023
#0.024
10.024
10.024
2.065
2.127
2.163
2.185
2.198
2.204
2.203
IRM
+0.021
+0.021
#0.021
+0.021
+0.021
+0.021
#0.021
1.487
1.637
1.856
2.053
2.181
2.249
2.303
DP noisy
10.031
#0.026
+0.023
+0.021
+0.021
+0.021
#0.022
STOI
Model
-5
0
10
15
20
No noise
0.681
0.686
0.691
0.697
0.707
0.718
0.738
Noisy Rev
+0.002
+0.003
+0.003
+0.003
#0.003
10.003
#0.003
0.531
0.614
0.685
0.734
0.763
0.778
0.801
PI
+0.009
#0.008
+0.006
+0.005
+0.004
10.003
#0.003
0.545
0.629
0.701
0.752
0.780
0.794
0.814
MoE
+0.009
+0.008
10.006
#0.005
+0.003
0.003
#0.003
0.608
0.677
0.734
0.773
0.796
0.807
0.829
MoB
+0.007
#0.006
+0.005
#0.004
+0.003
00.03
#0.022
0.546
0.629
0.700
0.750
0.779
0.793
0.812
OFP
0.009
0.008
+0.006
+0.004
+0.003
+0.003
+0.003
0.620
0.687
0.744
0.784
0.808
0.819
0.840
OE
+0.007
+0.006
+0.005
+0.004
+0.003
+0.002
10.00
0.973
0.980
0.982
0.982
0.982
0.982
0.981
IRM
10.001
10.001
+0.000
10.000
+0,00
10.000
10,00
0.642
0.759
0.859
0.928
0.968
0.987
1.000
DP noisy
10.000
+0.008
+0.005
+0.003
+0.002
10.001
10.00
40

---

A.8.2 Roomwise model performance GRU+A
Objective speech intelligibility scores (estimated marginal mean (# 95% confidence interval) for mask estimated using phoneme independent (PI) model, and phoneme-specific Omni- Expert model with predicted/known phonemes (OEP/k) across varying noise types and signal-to-noise ratio (SNR in dB). Results are aggregated for HINT speech with domestic noise + office, public noise + stairway, two-talker babble (TTB) noise + lecture and TTB church. Results shown for the base LSTM model and the GRU+Attention (GRU+A) model. Bold indicates the highest performance among the non-oracle models.
Table A8.2:
SRMR-CI
Model
-5
5
10
15
20
No noise
1,329
1,431
1,555
1.649
1.702
1.725
1.812
PS-M
+0.022
+0.022
+0.023
#0.024
+0.024
10.024
+0.025
1.353
1.486
1.656
1.784
1.849
1.878
1.986
PI -R-UA+A
#0.024
= 0.022
#0.033
#0.025
#
#0.026
#0.028
1.295
1.387
1.522
1.634
1.701
1.734
1.848
OE LSTM
+0.021
+0.020
#0.02
+0.023
+0.024
0.025
+0.025
1.368
1.499
1.685
1.826
1.902
1.936
2.119
OEP
GRU+A
#0.026
+0.024
0.026
#0.027
+0.028
0.029
10.030
1.434
1.547
1.686
1.790
1.850
1.881
2.011
OE
LSTM
#0.021
#0.021
#0.022
#0.023
#0.024
#0024
+0.024
1.555
1.706
1.864
1.972
2.027
2.052
2.217
OE GRU+A
#0.023
#0.023
0.025
#
+0.027
=0.277
+0.027
STOI
Model
-5
10
15
20
No noise
0.531
0.614
0.685
0.734
0.763
0.778
0.801
PI PI-MSTM
+0.009
+0.008
#0.006
+0.005
+0.004
+0.003
#0.003
0.558
0.642
0.716
0.767
0.795
0.808
0.823
PI- GRU+A
+0.009
+0.007
#0.006
#0.004
+0.003
+0.003
+0.003
0.546
0.629
0.700
0.750
0.779
0.793
0.812
OF LSTM
+0.009
+0.008
+0.006
+.0004
+00.03
+0.003
+0.003
0.559
0.645
0.720
0.773
0.804
0.818
0.836
OF
GRU+A
+..009
+.000
0.006
0.005
+ 0.004
+0.003
0.003
0.620
0.687
0.744
0.784
0.808
0.819
0.840
OE
LSTM
+0.007
+0.006
+0.005
+0.004
+0.003
+0.002
+0.002
0.629
0.701
0.760
0.801
0.824
0.835
0.855
OE GRU+A
+0.007
+0.006
+0.005
+0.004
+0.003
+0.002
+0.002
41

---

Nolty Reverberant
Donnmm-Pradicted-OF
Cennime-Predicted-Oe (GRU
Ideal Mask
Independent
Phoneme Known-DE
Phoneme-Knowwe
-atssy Signal
Phoneme eme Independent (GRU+A)
TTB- Church
Comestic-Oficc
2.4
2.0
2.0
1.8
SRMR
2.6
2.4
144
1.0
1.0
0.8
10
15
20
No noise
5
15
20
No nolse
TTB-Lacture
Public-Stairway
2.4
2.4
2.2
SRMR-CI
1.8
3.8
2.6
1.4
1.2
3.2
1.0
0.8
20
SNR
SMR
(a) SRMR-CI
Nolsy Reverberant
Phoneme-Prodicted-0 Prodicted-DE Phoneme Known-DE
Phoneme-Predictdd.oo A)
Mask
Phoneme Indepen dent Independent,
Phoneme- Known-OE (GRU+A)
Direct Path (Noisy Signal)
1.0
1.0
0.8
0.8
0.6
0.6
0.5
0.5
TTB-Lecture
Public-tairwway
0.9
0.9
0.7
0.4
0.4
10
15
20
No noise
-5
0
10
15
No nake
SNR
SNR
(b) STOI
Figure A8.2: SRMR-CI and STOI scores for HINT speech with noise conditions, Domestic and Public noises from DEMAND dataset and Two-Talker Babble (TTB) from Cocktail Party dataset convolved with office, stairway, lecture, and church room conditions (RIRs), respectively. Results are shown for unenhanced noisy reverberant speech, mask estimated using phoneme independent models, thoneme-specitee Omni- Omni-Expert models (OE) LSTM and GRU+A, ideal ratio mask (IRM), and the direct path (DP) of the noisy reverberant signal. Noise was added signal-to-noise (SNR) levels: -5, 0. 5, 10, 15, 20. Additionally, results are shown for noise (only RIR) condition.
42

---

A.8.3 Room-specific Phoneme Classifier Performance in Noisy Reverberant Conditions
Table A8.3.1:
Phoneme classification accuracies in noisy reverberant test conditions using long short-
term memory (LSTM) model architecture (%). Models are trained in reverberant only conditions.
Dataset
-5
5
10
15
20
16.4
19.87
23.43
26.04
28.18
29.76
HINT-Public-Stairre
6.08
9.31
14.32
19.71
24.25
27.31
HINT-TTB-Leauie
7.59
10.56
14.15
18.28
22.06
24.81
HINT-TTB-Chunl
7.57
9.93
12.70
15.78
18.43
20.23
Table A8.3.2: Phoneme Classification Accuracies in noisy reverberant test conditions using gated recurrent unit attention (GRU+A) model architecture (%). Models are trained in reverberant only conditions.
Dataset
-5
10
15
20
24.04
29.46
34.97
40.03
43.26
45.14
HINT--ublic- Public-ttarrwwy
6.50
11.65
19.75
28.16
35.36
40.25
MINT-TTB-Lectuure
7.91
12.13
18.20
25.15
32.18
37.34
HINT-TTB-Chun
7.96
11.50
16.36
21.73
26.67
29.89
PC-LSTM
PC-GRU+A
TTB-Church
Domestic-Office
50
50
40
40
30
Accuracy
10
Signal to Noise Ratio (JB)
Signal to Noise Ratio (dB)
Public-Stairway
350
30
30
20
100
155
20
36
1.
20
Signal to Noise Ratio (dB)
Signal-to Noice Ratio (dB)
Figure A8.3.1: Phoneme classifier (PC) performance in noisy reverberant room conditions using the long short-term memory (LSTM) model and the gated recurrent unit + attention (GRU+A) architecture. TTB, two-talker babble.
43

---

44


Reviews:
[{"review_id": "EFjO9im16L", "reviewer": "Reviewer_YCmV", "summary": "The authors propose a variant or alternative to mixture of experts (MoEs) but constrain their analysis to dereverbration of audio signals for cochlear implants and the associated compute and memory constraints. The authors address dereverberation by estimating a time frequency mask which then gets multiplied with the spectrogram representation. The authors propose to have task-specific scale and shift operations and a shared backbone model instead of multiple experts. This idea is validated by comparing first a single model against MoE and then against the proposed task-specific feature transformation with a single model. Experiments are thorough and contain intelligibility metrics obtained after synthesis from the cochlear implant output electrode signals.", "questions": "Please clarify if the scale matrix A_k is a dense matrix or just a diagonal matrix. Was not fully clear.\n\nPlease clarify why you need a network to predict A_k and b_k from a one-hot vector. It seems wasteful. If you have one hot input and an MLP, the MLP is not required. You can just look up the end result because there are anyway only 40 possible vectors. (line 177). A frequently used term for this is \"embedding layer\".\n\nFigure 2: Why are true labels with probabilities 0<p<1 instead of either 0 or 1?\n\nConfidence intervals in Table3/4 are odd. Why are they symmetric? Bootstrap-based confidence intervals should be shown as separate columns or smaller numbers but not relative with +/-. Further, are the confidence intervals characterizing the test example variance or the variance between multiple repeated training runs?", "limitations": "The biggest limitation of this work is the evaluation on such a niche application. The reader can hardly infer if this Omni-Expert approach applies to their domains and can replace their expensive MoE.\n\nIt is also highly likely that the single model capacity and consequentially the MoE model capacity is just much too high for this application. Thus, compressing it into only task-specific feature transformation works. Imagine a more complex task like language modeling where really huge MoEs are used. Given your experimental results, would you advise them to exchange their MoEs with Omni-Expert?", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The paper title suggests an alternative to MoEs widely applicable and studied on various domains. However, the abstract then disappoints a little by reducing the scope to audio, to be precise, dereverberation, to be precise, under limited compute, to be precise, on cochlear implants. The contribution is meaningful nonetheless but the title should be toned down or contain \"dereverberation\".\n\nThe experimental validation is very well done. Laudable that you use recorded room impulse responses (RIRs). Even more so that you take test RIRs from a different database than the train RIRs. Good experimental study with three different oracle conditions, good use of direct path signal to get, e.g., alignments and oracle. Often enough I had seen papers where people use the source signal as reference instead of direct path signal (and then wonder why timing is not right).\n\nThe references for alternatives to MoEs are a bit sparse. I suggest the conditional scale and shift parameters should cite https://arxiv.org/abs/1707.00683, in particular conditional batch normalization in Section 3. Also https://www.ecva.net/papers/eccv_2022/papers_ECCV/papers/136870299.pdf is a paper in which the backbone is shared and then only task specific (output) branches are used to make this behave like MoE.", "was_revised": false, "final_justification": "", "rebuttal": "Thank  you for spending your time reviewing our paper and providing valuable feedback. We are particularly encouraged by the reviewer’s commendation of the key strengths of our work that we were indeed most excited about in our paper. \n\n# Comments:\n\n> **C1: The paper title suggests an alternative to MoEs widely applicable and studied on various domains. However, the abstract then disappoints a little by reducing the scope to audio, to be precise, dereverberation, to be precise, under limited compute, to be precise, on cochlear implants. The contribution is meaningful nonetheless but the title should be toned down or contain \"dereverberation\".**\n\nWe thank the reviewer for highlighting the meaningful contribution of our work. The NeurIPS conference covers a diversity of research topics, and we submitted our paper under the primary area of *Applications*. We believe that researchers from other fields could benefit from learning about practical considerations when deploying algorithms for end-users in real-world conditions. In this case, the current paradigm of scaling with a MoE is not practical given constraints imposed by real-time processing and limited compute-resources. We note that real-time speech enhancement is applicable to cochlear implants (acoustic-to-electric stimuli) and hearing aids (acoustic-to-acoustic), with the former representing the harder case. While our work focuses on speech dereverberation, the core architectural idea of substituting multi-expert inference with subtask-specific transformations applied to a single expert is in principle domain-agnostic. As researchers have developed innovative ways to extend the MoE technique since the original paper, we believe other researchers could advance or adapt the OE technique if it is a suitable alternative.  We will clarify this in the revised paper. \n\n> **C2: The reader can hardly infer if this Omni-Expert approach applies to their domains and can replace their expensive MoE.**\n\nWe broadly discussed the societal impact of our work in the *Discussion* section (**lines 284-289**) and will revise for better clarity. As we do not necessarily have expertise in other domains, our intent is not to indicate a universal replacement of MoE but offer guidance to determine if the Omni-Expert approach is a suitable alternative. MoE have been applied at the embedding, token/sequence or task level. Our main goal here is estimating a phoneme-specific mask based on the phoneme in the current time frame and phoneme features exhibit inherent structure and homogeneity; hence the subtasks are defined/known. Similarly, in domains like large language models (LLMs) or computer vision, there is often latent structure among tokens, tasks, or embeddings (e.g., syntactic roles, semantic types, or class labels). These can be used to define subtask-specific transformations or input conditioning, which suggests that our Omni-Expert approach could be applicable. Other variants of the OE may be needed when specialized subtasks are not as well-defined.\n\n> **C3: Imagine a more complex task like language modeling where really huge MoEs are used. Given your experimental results, would you advise them to exchange their MoEs with Omni-Expert?**\n\nIn more complex domains, we postulate that our OE approach could offer value where expert specialization is correlated with known input attributes (e.g., task tokens, domain IDs, or modality types). We do not claim OE would match the performance of large-scale MoEs in language modeling, but it may serve as a scalable, cost-effective solution in specific parts of a larger system or in resource-constrained deployments of LLMs.\n\n>**C4: The references for alternatives to MoEs are a bit sparse. I suggest the conditional scale and shift parameters should cite ..., in particular conditional batch normalization in Section 3. Also ... is a paper in which the backbone is shared and then only task specific (output) branches are used to make this behave like MoE.**\n\nWe thank the reviewer for highlighting these relevant works and we will reference them in our revised paper. Please note that we removed the links in accordance with the current policy of not including links in the rebuttal response. \n\n*[1] De Vries, Harm, et al. \"Modulating early visual processing by language.\" Advances in neural information processing systems 30 (2017).*\n\nIn Conditional Batch Normalization (CBN) [1], a pretrained convolutional neural network is implemented with the idea of predicting additive adjustments to the batch normalization scaling and shifting parameters based on an external conditioning vector (e.g., a language embedding). Instead of directly predicting the full parameters, CBN predicts small residual changes to pretrained batch normalization parameters, which helps stabilize training and preserve pretrained knowledge. This approach maintains pretrained performance and allows scalable adaptation with minimal additional parameters.\n\nThus, our proposed method shares the concept of conditional scaling and shifting with the following differences:\n\n•\tInstead of modulating batch normalization parameters in each block of the residual network, we apply conditional scale and shift transformations directly to the input features before passing them to a single expert network.\n\n•\tUnlike CBN’s residual updates to frozen parameters of a pretrained residual network, our transformations are learned jointly end-to-end with the rest of the expert network.\n\n*[2] Xu, Xiaogang, et al. \"Mtformer: Multi-task learning via transformer and cross-task reasoning.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.*\n\nMTFormer framework [2] employs a transformer-only multi-task learning (MTL) architecture consisting of a shared feature extractor (encoder and decoder) and lightweight task-specific branches. MTFormer uses self-task attention within each task-specific branch and introduces cross-task attention modules to enable feature propagation among tasks, achieving efficient parameter sharing and superior performance on multi-task problems. \n\n•\tWhile MTFormer separates specialization through task-specific output branches after a shared transformer backbone, our Omni-Expert model realizes expert specialization within a single expert network by applying task-specific feature transformations. This design avoids multiple output heads and keeps the model size minimal.\n\n> **C5: Please check how you write the scale and shift terms. Sometimes they are capital letters, sometimes not.**\n\nWe thank the reviewer for noticing the inconsistency. Our intention was to use capital letters for the general case of the linear transformation matrix ($\\mathbf{A}_k$) and lowercase notation for the specific case of scale transformation ($\\textit{\\textbf{a}} _k$). The shift transformation has the same dimension as the feature space; we will correct the typo in line 113 to ($\\mathbf{b}_k$). For clarity and consistency, we will revise Equation (3) and the surrounding text to align with this convention and make the transition between the general and specific notations more explicit (see continuation of explanation in later section).\n\n> **C6: Text in figures (e.g. Figure 3) borderline too small.**\n\nWe will increase font size in the revised manuscript.\n\n# Questions:\n\n>**Q1: Please clarify if the scale matrix A_k is a dense matrix or just a diagonal matrix. Was not fully clear.**\n\nWe appreciate the question regarding the structure of linear transformation $\\mathbf{A}_k$ and complete our explanation from the previous section. $\\mathbf{A}_k \\in \\mathbb{R}^n$ ($n$ is the length of the feature vector and different linear transformations (scale, shear, reflection, etc.) have different matrix forms; a scale transformation is a diagonal matrix, which simplifies to element-wise multiplication. To reduce the number of $\\mathbf{A}_k$ parameters that need to be estimated, we use a scale transformation.\n\n>**Q2: Please clarify why you need a network to predict A_k and b_k from a one-hot vector. It seems wasteful. If you have one hot input and an MLP, the MLP is not required. You can just look up the end result because there are anyway only 40 possible vectors. (line 177). A frequently used term for this is \"embedding layer\".**\n\nWe appreciate the reviewer’s insightful comment. Since the mapping from phoneme label to phoneme-specific transformation is fixed, we use MLPs only during training to learn the feature transformation parameters. We precompute and store the final transformation vectors for all 40 phonemes, effectively reducing the process during inference to a lookup table. While we discuss this in the *Model Complexity* section of our paper (***Section 5.5, lines 257-260 and Table 5***), we recognize the need for a different placement of this information and a better description of the model architecture (including figure) and training for better clarity. We will revise the description of the OE model architecture and training in Section 3 of the paper.  \n\n>**Q3: Figure 2: Why are true labels with probabilities 0<p<1 instead of either 0 or 1?**\n\nFigure 2 shows soft posterior probability outputs from the phoneme classifier, rather than hard classifier decisions. \n\n> **Q4: Confidence intervals in Table3/4 are odd. Why are they symmetric? Bootstrap-based confidence intervals should be shown as separate columns or smaller numbers, but not relative with +/-. Further, are the confidence intervals characterizing the test example variance or the variance between multiple repeated training runs?**\n\nThe confidence intervals in Table 3/4 are derived from the standard error (hence symmetric) and characterize the variability across the test examples."}, {"review_id": "218gCcVG3Z", "reviewer": "Reviewer_q66g", "summary": "The paper proposes a speech dereverberation framework targeting Cochlear Implant(CI) device. It attempts to improve upon an existing MoE approach by proposing a single expert model and a phoneme-based feature transformation block. The feature transformation block learns a scale-shift operation to transform the input speech features based on the predicted phoneme label and the expert model works on these features to estimate the masks for speech dereverberation task.", "questions": "I have concerns about the proposed scale-shift feature transformation. As I expected, the shift operation would try to learn shift vectors for each phoneme so that the transformed feature clusters are pushed far apart and it gets easier for the model to reduce the training loss (Fig. 5). I suspect this will lead to poor generalization as the clusters grow apart with enough training. Was there any constraint/regularization on the shift vectors? Would it also help to use some kind of contrastive/regularization loss to keep the clusters close by and to maintain continuity in the feature space?\n\nIs the phoneme classifier well trained? Why is the phoneme classification accuracy (Table 1) so low? In the HINT dataset, for some RIRs, accuracy is lower than 30% (whereas the phoneme /sil/ itself is around 31% of the test data). \n\nIs the Fig A6.1.1correct? Are the shift and scale images swapped? It’s counter intuitive to me  that shift is not able to change the cluster locations but scale alone can.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\nThe paper is well motivated and easy to follow. \nThe paper targets a very important application of Cochlear Implant for hearing impaired individuals.\nSubstantial memory and computational complexity reduction compared to baselines. \nAblation experiments are performed to establish understanding of the proposed method. \n\nMajor Weaknesses:\nPerformance improvement is minor from provided baselines specially in more realistic noisy conditions. The much simpler PI model outperforms the proposed method in most of the SNR conditions (Table A7.1). \nLimited performance comparisons. Comparisons with existing dereverberation baselines such as [1], [2] not provided. \nSince the model is built for practical purposes,I feel a user subjective quality test under real conditions is necessary. \n\n\n[1] Zhao et al.  A deep learning based segregation algorithm to increase speech intelligibility for hearing-impaired listeners in reverberant-noisy conditions. J Acoust Soc Am. 2018 Sep\n[2] Healy et al. A deep learning algorithm to increase intelligibility for hearing-impaired listeners in the presence of a competing talker and reverberation. J Acoust Soc Am. 2019 Mar\n\nMinor weaknesses:\nNo demo samples are provided. In the absence of subjective quality tests from human users, I would have appreciated some samples of the model's produced output and baselines to better judge the performance.", "was_revised": true, "final_justification": "The paper has some flaws but overall a reasonably good submission, especially due to it's thorough evaluation.\n\nAdding this comment on Aug 12:\nThe issue is that paper takes on a very niche/narrow application and the contributions are not revealing new insights into machine learning, deep learning, or audio. That said, this narrow application is an important one, and the paper has systematically done a suite of thorough experiments. I am truly on the fence about this paper and wouldn't object if the paper gets rejected.", "rebuttal": "Thank you for spending your time to review our paper and providing valuable feedback. Please find below responses to your comments.\n\nOur work on speech dereverberation is *significant for individuals with hearing impairments* as there is currently no solution in cochlear implants (and hearing aids) that directly addresses reverberation (***lines 50-55***). Even in the absence of noise, individuals with auditory prostheses often struggle to understand speech in reverberation [1], which can negatively impact their quality of life. For example, the learning experience of individuals with hearing impairments [1, 2], which typically involves single talker scenarios in a classroom/lecture hall. We will discuss the impact of reverberation in the revised paper. \n[1] P. Zahorik, \"Spatial hearing in rooms and effects of reverberation,\" in Binaural Hearing, Cham: Springer Int. Publ., 2021, pp. 243–280.\n[2] F. Iglehart, \"Speech perception in classroom acoustics by children with cochlear implants and with typical hearing,\" Am. J. Audiol., 2016;25(2):100-9.\n# C1: Performance in Noisy-reverberant Conditions \n- While our work focused on dereverberation, we tested models in noisy reverberant conditions to evaluate generalizability to unseen conditions. We applied the models *trained only on reverberant speech* on noisy reverberant conditions, creating a train/test data mismatch. In ***Figure A7.1*** (Appendix), the PI model outperforms the OE particularly at lower SNRs. Nonetheless, the OE with known phonemes provides the best overall performance, indicating that the limiting factor is the phoneme classifier. \n- The OE model in the paper used a LSTM for the phoneme classifier. We updated the phoneme classifier with GRU+Attention (GRU+A) network (***Section A.9 and Table A9.1.1***, Supplementary file). New results in Table R1 below show improvements with OE with the new phoneme classifier, outperforming the PI model in reverberant-only (***Table R1 (a)***) and a wider range of SNR levels in noisy-reverberant conditions (***Table R1(b)***). \n\n*Table R1. Mean (± 95% confidence interval) of objective speech intelligibility scores for reverberant (Rev) or noisy Rev signal, direct path (DP) signal and mask estimation methods: phoneme independent (PI), phoneme-based mask estimated by OE with ideal phoneme knowledge ($OE^k$) and phoneme probabilities ($OE^p$) from LSTM (PC-1) and GRU+A (PC-2) classifiers. Bold indicates the highest value among non-oracle models.*\n\n**(a)  Reverberant-only conditions**\n||$Rev$|$PI$|$OE^p$ (PC-1)|$OE^p$ (PC-2)|$OE^k$|\n| ----------- | ------------- | ------------- | ------------- | ------------- | ------------- | \n|SRMR-CI|1.302$\\pm$0.007|1.733$\\pm$0.009|1.794$\\pm$0.010|$\\textbf1.874\\pm0.010$|1.938 $\\pm$0.009|\n|STOI|0.719$\\pm$0.001|0.797$\\pm$0.001|0.807$\\pm$0.001|$\\textbf0.816\\pm0.001$|0.836$\\pm$0.001|\n\n**(b)\tNoisy Reverberant conditions**\n| | -5 | 0 | 5| 10 | 15 | 20 | No noise |\n| ----------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |\n| | | | |***SRMR-CI***| | | |\n|Noisy Rev|1.007$\\pm$0.015|1.060$\\pm$0.014|1.153$\\pm$0.015|1.239$\\pm$0.017|1.290$\\pm$0.018|1.313$\\pm$0.018|1.327$\\pm$0.018|\n|$PI$|$\\textbf1.329\\pm0.022$|$\\textbf1.431\\pm0.022$|$\\textbf1.555\\pm0.023$|1.649$\\pm$0.024|1.702$\\pm$0.024|1.725$\\pm$0.024|1.812$\\pm$0.025|\n|$OE^p$ (PC-1)|1.295$\\pm$0.021|1.387$\\pm$0.020|1.522$\\pm$0.021|1.634$\\pm$0.023|1.701$\\pm$0.024|1.734$\\pm$0.025|1.848$\\pm$0.025|\n|$OE^p$(PC-2)|$\\textbf1.328\\pm0.023$|$\\textbf1.428\\pm0.021$|$\\textbf1.577\\pm0.023$|$\\textbf1.699\\pm0.025$|$\\textbf1.769\\pm0.026$|$\\textbf1.804\\pm0.026$|$\\textbf1.934\\pm0.027$|\n|$OE^k$|1.434$\\pm$0.021|1.547$\\pm$0.021|1.686$\\pm$0.022|1.790$\\pm$0.023|1.850$\\pm$0.024|1.881$\\pm$0.024|2.011$\\pm$0.024|\n| | | | |***STOI***| | | |\n|Noisy Rev|0.533$\\pm$0.007|0.600$\\pm$0.006|0.657$\\pm$0.005|0.697$\\pm$0.004|0.721$\\pm$0.004|0.732$\\pm$0.003|0.738$\\pm$0.003|\n|$PI$|0.531$\\pm$0.009|0.614$\\pm$0.008|0.685$\\pm$0.006|0.734$\\pm$0.005|0.763$\\pm$0.004|0.778$\\pm$0.003|0.801$\\pm$0.003|\n|$OE^p$ (PC-1)|0.546$\\pm$0.009|0.629$\\pm$0.008|0.700$\\pm$0.006|0.750$\\pm$0.004|0.779$\\pm$0.003|0.793$\\pm$0.003|0.812$\\pm$0.003|\n|$OE^p$ (PC-2)|$\\textbf0.552\\pm0.009$|$\\textbf0.636\\pm0.008$|$\\textbf0.708\\pm0.006$|$\\textbf0.758\\pm0.005$|$\\textbf0.787\\pm0.004$|$\\textbf0.801\\pm0.003$|$\\textbf0.819\\pm0.003$|\n|$OE^k$|0.620$\\pm$0.007|0.687$\\pm$0.006|0.744$\\pm$0.005|0.784$\\pm$0.004|0.808$\\pm$0.003|0.819$\\pm$0.002|0.840$\\pm$0.002|\n- We included results using GRU+A for PI and OE (mask estimation and phoneme classification) models in the supplement, which show significantly higher performance for OE in reverberation and noisy-reverberant conditions vs. the PI model (***Figures A9.2.3, A9.2.4, A9.3.2, and Table A9.3.1***, Supplementary file). \nWe will consolidate the results in the revised paper.\n\n# C2: Comparison with Prior Work\n> **Comparisons with existing dereverberation baselines such as [1], [2] not provided.**\n\nThanks for bringing [1-2] to our attention. Our goal is to demonstrate that the OE achieves at least equivalent performance to MoE with reduced model complexity. We present key differences with [1-2] based on our objective within the context of real-time processing in low-resource settings. Both studies do speech enhancement in the acoustic-to-acoustic domain, applicable to hearing aids and as a pre-processing step in cochlear implants (CIs). In our work, speech enhancement for CIs occurs within the CI sound processing pipeline (i.e., acoustic-to-electric stimuli) to reduce delay. \n\n*[1] Zhao et al. A deep learning-based segregation algorithm to increase speech intelligibility for hearing-impaired listeners in reverberant-noisy conditions. J Acoust Soc Am. 2018* \n\n- The work uses the same *synthesized* room impulse responses (RIRs) generated in the same simulated room for training and testing. Our work uses **recorded RIRs** from diverse real-world rooms (***Table A1*** in Appendix of our paper) from different dataset for training and testing (**Section 4.2 in our paper**).\n- Different partitions from the *same speech* dataset with the same speaker were used for training and testing, whereas our training and testing use different speech datasets with no overlap in speech material (**Section 4.2 in our paper**).\n- The DNN uses a context window of 9 frames on each side of the current frame, thus *non-causal*, which introduces delay and infeasible for real-time applications. Our approach is *causal* as it relies only on features from the *current* time frame, thus real-time feasible. The paper lacks details about the DNN architecture. \n\n*[2] Healy et al. A deep learning algorithm to increase intelligibility for hearing-impaired listeners in the presence of a competing talker and reverberation. J Acoust Soc Am. 2019*\n- Similar to [1], there is overlap of train and test material: synthesized RIRs generated from the same simulated room, different partitions of the same speech dataset with the same speaker. \n- The DNN is a bidirectional LSTM network, so non-causal.\n\nThus, a direct comparison of our work with [1-2] is not possible. \n# C3: Sample Speech Files\nWe agree on the need for a subjective test to further validate results. We will include CI vocoded speech files in the supplement file of the revised paper. Note that CI vocoded speech sounds very different from natural speech.\n\n# Q1: Constraint/Regularization on Feature Transformations\nWe appreciate the reviewer’s observation about feature transformations. To clarify, the size of clusters and distance between clusters with t-SNE are not necessarily informative about the relationship between clusters.  \nWe apply feature transformations only in the input feature space and the OE training loss is based on the difference between the signal after applying estimated and ideal masks (***equation 1***, main text). Since transformed features are passed through a single expert network and optimized end-to-end with respect to signal loss, the OE balances cluster separation with speech enhancement performance. \n\nWe did not apply explicit regularization or contrastive loss on shift or scale parameters. Contrastive and regularized learning may offer potential benefits and will include as future work in the revision. \n# Q2: Phoneme Classifier Accuracy\nPhoneme classifier accuracy is low, particularly for HINT dataset, for these reasons:\n- For real-time speech enhancement, the phoneme classifier is intentionally *lightweight*. Features are extracted at the CI sound processing time frame of 8ms. Phonemes typically range from 70-200 ms [1], making phoneme classification based on an 8ms frame a relatively hard task. Even at low accuracies, noisy phoneme probabilities are still useful to the OE model. Phonemes with similar time-frequency characteristics are likely to be confused with each other (***Figure A4.2***, Appendix). The soft weighting of phoneme-specific masks reduces the impact of phoneme misclassifications. We also show benefits with improved phoneme classification (response C1 above).\n- The silence class includes silent regions at the start and end of the speech file, and silent gaps between phonemes. CUNY datasets have long periods of silence at the start and end of the speech file, which are easy to detect (***Figure A4.2***, Appendix). In contrast, the HINT dataset has minimal silent regions at the start and end of speech (***Figure 2*** in main paper), so the silence class is mostly silent gaps between phonemes, which are harder to detect when filled with reverberant reflections (***Figure A4.2***, Appendix). \n[1] Fant, Gunnar. Speech acoustics and phonetics. Dordrecht: Springer Netherlands, 2005.\n\n# Q3: Figure A6.1.1\nThe transformation labels in Figure A6.1.1 are correct. Shift transformation changes cluster origin. Scale transformation changes the cluster size; the scaling is non-uniform as each feature has a different scale parameter and a negative scale value yields a reflection."}, {"review_id": "61PfTbnyeD", "reviewer": "Reviewer_5JAh", "summary": "This submission provided a new omni-expert (OE) model, a more efficient alternative to the mixture-of-experts model (MoE) by creating a subtask-specific feature transformation block so that all the features are fed to a single model. The OE model is evaluated against MOE model in the context of cochlear implant (CI), where multiple tasks are performed. The OE model is found to have a better performance and is more computationally effective.", "questions": "What are the reasons that the MOE and OE models perform similarly in terms of the STOI metric?\n\nAre the relatively small improvements from MOE and OE models meaningful from using a PI model?\n\nHow would other strategies of improving MOE perform in this task, such as merging the expert models (i.e. from existing work \"Merging Experts into One: Improving Computational Efficiency of Mixture of Experts\")?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Quality: The submission presents an OE model architecture, and its performance is rigorously compared to the MOE model variant in the context of phoneme prediction of CI. However, in some of the results, i.e. table 2, where the OE did not show any performance improvement in terms of the STOI metric, or in figure 4, where both the MOE and OE models show minimal improvements to the phoneme independent (PI) model. The submission could benefit from the authors justifying that the minimal improvement is meaningful and discussing the result more, where STOI is not improved by using OE compared to MOE.\n\nClarity: The article is well-written and organized, although some of the specific terms could use a bit more explanation to be clearer for people outside the field of audio processing field, such as the difference between a phenome-known model and a phenome-predicted model. \n\nSignificance: The submission is proposing a new model for CI data processing that is less computationally expensive and performs better. It can be very beneficial for CI manufacturing. \n\nOriginality: This submission proposes a new OE model that attempts to improve upon MOE by using a feature transformation block. However, existing work exists that attempts to reduce the computational cost of MOE systems, albeit using a different strategy. (See the paper titled \"Merging Experts into One: Improving Computational Efficiency of Mixture of Experts\") The authors should mention previous work and consider explaining why this new strategy could be a better approach.", "was_revised": true, "final_justification": "The authors' rebuttal sufficiently addressed my previous concerns. The authors clarified that the main selling point of the OE model is reduced computational cost and justified why the STOI metric might fail at capturing improvements. The authors also made the effort to compare with prior work mentioned in my original review. As such, I maintain my original recommendation of acceptance.\n\nWhile this work is primarily focused on signal processing for cochlear implants, the OE methodology could be adapted to similar types of signals, albeit it would be an entirely new study. I lean towards publishing for the new methodology, even with a potentially limited application case.", "rebuttal": "Thank you for the feedback and spending your time reviewing our paper. Please find below responses to your comments and the changes we will make to the paper.\n## C1: Relative Performance Improvements with Objective Speech Intelligibility Metrics.\n> **In some of the results, i.e. table 2, where the OE did not show any performance improvement in terms of the STOI metric, or in figure 4, where both the MOE and OE models show minimal improvements to the phoneme independent (PI) model. The submission could benefit from the authors justifying that the minimal improvement is meaningful and discussing the result more, where STOI is not improved by using OE compared to MOE.**\n\n> **What are the reasons that the MOE and OE models perform similarly in terms of the STOI metric? Are the relatively small improvements from MOE and OE models meaningful from using a PI model?**\n\n- ***STOI excludes frames with silent gaps in clean speech, whereas reverberation occurs in silent speech gaps***. The STOI metric [1] was developed for evaluating intelligibility of speech *in noise*, not reverberation. From the metric authors, the STOI computation does not include silent gaps in the speech signal and focuses only on speech segments. From page 2126, Section II. [1]:\n\n      Before evaluation, silent regions which do not contribute to speech intelligibility are removed. This is done by first finding the frame with the maximum energy of the clean speech signal. Both signals are then reconstructed, excluding all the frames where the clean speech energy is lower than 40 dB with respect to the maximum clean speech energy frame. \n\n  The rationale the authors provide is a metric to reflect intelligibility of actual speech content rather than being influenced by non-informative silent regions. However, reverberant reflections persist during silent gaps. Removal of reverberant speech reflections typically occurs from the tail of decay inward (see electrodograms in ***Figure 4(a)***). Thus, removal of reverberant reflections in the same silent speech gap will not be captured by the STOI metric. \n\n- ***Speech-to-Reverberation Modulation Energy Ratio (SRMR) metric [2,3] was developed specifically for evaluating speech intelligibility in reverberation.*** The original SRMR metric was proposed for normal hearing listeners [2] and later adapted for CI users (SRMR-CI) by using the CI filterbanks [3]. Thus, the SRMR-CI metric provides a more reliable speech intelligibility predictor for cochlear implants in reverberation. Nonetheless, we included STOI for completeness and to facilitate comparison with prior literature. We will include the clarification of the STOI and SRMR metrics in the Performance Evaluation portion (***Section 4.3***) in the revised paper.\n\n- ***Relative performance improvements from PI to OE to MoE.*** We implemented the MoE from [4] who showed improvements in SRMR-CI and STOI scores, as well as intelligibility of CI vocoded speech in normal hearing listeners with the MoE model relative to a PI model (see ***194-196*** in the main paper). As we stated in the paper, our goal with the OE is to achieve at a minimum the same performance at a reduced computational cost relative to a counterpart MoE model (***lines 196-198***, main paper). The OE model offers a computationally efficient alternative to the MOE model. The OE has a much faster training time relative to the MoE model (***Table 5***, main paper). We observe statistically significant improvements in SRMR-CI scores in aggregate (***Table 2***, SRMR-CI) and room-specific (***Figure 4(b)***, SRMR-CI) results from PI to MoE to OE. We have discussed the limitations of the STOI metric not capturing the impact of removal of reverberant reflections in the same silent speech gaps, which likely explains relatively small differences in STOI between PI vs. {MoE, OE} and similar STOI between MoE and OE models (STOI results in ***Table 2 and Figure 4(b)***). Note that we also included results with an advanced model architecture for the PI and OE models in the supplement file (***Section A.9***), where we obtain similar performance trends with wider margins of improvement from PI to OE models (***Figures A9.2.3, A9.2.4 and A9.3.2***). We will consolidate results from both models in the revised paper to show that our OE approach is applicable to other models.\n\n- OE provides a higher performance upperbound vs. MoE. We included performance with perfect sub-task selection with the gating network, i.e., perfect phoneme classification, in the MoE and OE configurations; this provides a benchmark of additional improvements that could be obtained by just improving the phoneme classifier performance with the same mask estimation model. We have additional results in the supplement file that shows increased phoneme classification rates with an advanced phoneme classifier (GRU+Attention) vs. the LSTM phoneme classifier used in the main paper.  \n\n## C2: Definition of Terms.\n\n> **Some of the specific terms could use a bit more explanation to be clearer for people outside the field of audio processing field, such as the difference between a phenome-known model and a phenome-predicted model.** \n\nWe will clarify the definitions of “phoneme-known” and “phoneme-predicted” models in the *Performance Evaluation* section of the paper (***section 4.3, lines 198-199***). The phoneme-known model refers to a MoE/OE mask estimation with perfect phoneme classification (i.e., oracle condition with ideal expert/subtask selection). The phoneme-predicted model refers to the MoE/OE mask estimation with phoneme classifier predictions that are used to probabilistic weight the expert/subtask-specific outputs during inference.\n\n## C3: Relationship to Prior Work\n\n> **Existing work exists that attempts to reduce the computational cost of MOE systems, albeit using a different strategy. (See the paper titled \"Merging Experts into One: Improving Computational Efficiency of Mixture of Experts\") The authors should mention previous work and consider explaining why this new strategy could be a better approach.** \n\n> **How would other strategies of improving MOE perform in this task, such as merging the expert models (i.e. from existing work \"Merging Experts into One: Improving Computational Efficiency of Mixture of Experts\")?**\n\nWe thank the reviewer for highlighting the related work, “Merging Experts into One: Improving Computational Efficiency of Mixture of Experts” [5], which we will cite and discuss in the revised manuscript in the Related Work section. Merging of Experts into One (MEO) addresses computational efficiency in MoE by aggregating multiple expert parameters at inference time using gating weights accordingly: \n\n$\\hat{W}_i = \\sum\\_ {k \\in T} G_k(x_i ) \\cdot W_k, \\hat{\\beta} _i  = \\sum\\_ {k \\in T} G_k(x_i ) \\cdot \\beta_k$   (1)\n\n$y_i = \\sigma(\\hat{W}_ix_i + \\hat{\\beta} _i)$  (2)\n\nwhere $x_i$ is the input; $G_k(x_i)$, is the gating network probability for input $x_i$ of the $k$-th expert; $W_k$ and $\\beta_k$ represent the weight and bias terms of the $k$-th expert; $\\hat{W}_i $ ,$\\hat{\\beta}_i$ are the aggregate weight and bias terms, respectively, for $x_i$; $T$ is the total number of selected experts. The final output for MEO is given by $y_i$, where $\\sigma$ represents the activation function. \n\nWhile the MEO reduces the computation cost at inference when compared to evaluating all selected experts, it still requires training and storing ***multiple expert*** networks, as well as performing parameter aggregation dynamically during inference. In contrast, our proposed Omni-Expert (OE) model utilizes ***a single expert*** network, $E$, throughout both training and inference accordingly: \n\n$z_k = a_k⊙x_i+b_k$ (3)\n\n$y_i =\\sum\\_ {k \\in T} G_k(x_i )\\cdot E(z_k)$ (4)\n\nwhere $x_i$ is the input feature vector; $z_k$ is the transformed feature vector; $a_k$ and $b_k$ are the scale and shifting factor vectors, respectively, for the $k$-th subtask; and ⊙ is element-wise multiplication.  Instead of selecting or merging experts, we apply subtask-specific (e.g., in this case, phoneme-based) transformations to the input features (3), followed by a gating network that softly weights the output of each transformed input (4). This allows our model to retain the representational flexibility of MoE systems while avoiding the cost of loading or merging multiple experts at inference, making it significantly more suitable for deployment on resource-constrained platforms.\n\nAlso, the training time of the OE model is much reduced when compared to the counterpart MoE configuration (***Table 5*** in the main paper; ***Table A9.2.1*** in the supplement file). As we discussed in the Model Complexity section of the paper (***page 9, lines 250-260***), each expert in MoE is trained only on the sub-task specific data whereas the OE is trained on the full training dataset. So, the OE model not only benefits from training on a larger dataset, but also from sub-task specialization via the feature transformations. \n\n[1] Taal, Cees H., et al. \"An algorithm for intelligibility prediction of time–frequency weighted noisy speech.\" IEEE Transactions on audio, speech, and language processing 19.7 (2011): 2125-2136.\n\n[2] T. H. Falk et al., A non-intrusive quality and intelligibility measure of reverberant and dereverberated speech. IEEE Trans. Audio Speech Lang. Process. 2010;18(7):1766–1774.\n\n[3] J. F. Santos and T. H. Falk. Updating the srmr-ci metric for improved intelligibility prediction for cochlear implant users. IEEE Trans. on Audio, Speech, and Lang. Process. 22(12):2197–2206, 2014.\n\n[4] K. Chu et al., Suppressing reverberation in cochlear implant stimulus patterns using time-frequency masks based on phoneme groups. Proc. Meet. Acoust., 50(1), 2022.\n\n[5] He, Shwai, et al., Merging Experts into One: Improving Computational Efficiency of Mixture of Experts. In Proc. Conf. Empir. Methods Nat. Lang. Process. (EMNLP), 2023, pages 14685–14691."}]

Author rebuttal:

Meta-review: The paper proposes an omni-expert (OE) model as a more efficient alternative to mixture-of-experts (MoE) for speech dereverberation in cochlear implant devices. Instead of training multiple experts, the approach uses a single backbone model combined with a task-specific feature transformation block that applies learned scale-and-shift operations based on phoneme information. Experiments show that the OE model achieves better intelligibility and computational efficiency compared to both single-model and MoE baselines.

The paper is well-motivated, easy to follow and targets an important real-world application (cochlear implants for hearing-impaired users). Methodology-wise, it yields a significant reduction in memory and computational complexity compared to MoE. The experimental evaluation is thorough. However, the title and scope are sold broader than the actual contribution and the paper misses some related work. Also the improvements to the baselines are rather slim at times.

All reviewers were quite positive about the paper. In the rebuttal and the discussion phase, this opinion was confirmed.
Decision: Accept (poster)