Self-supervised Speech Enhancement using Multi-Modal Data
Authors: Yu-Lin Wei
Bashima Islam
RAJALAXMI RAJAGOPALAN
romit choudhury
Venue: Submitted to ICLR 2023 (2023)
Keywords: multi-modal
selfsupervise
denoising
iterative algorithm
attention map
expectation maximization
IMU

Abstract: Modern earphones come equipped with microphones and inertial measurement units (IMU). When a user wears the earphone, the IMU can serve as a second modality for detecting speech signals. Specifically, as humans speak to their earphones (e.g., during phone calls), the throat’s vibrations propagate through the skull to ultimately induce a vibration in the IMU. The IMU data is heavily distorted
(compared to the microphone’s recordings), but IMUs offer a critical advantage — they are not interfered by ambient sounds. This presents an opportunity in multi-modal speech enhancement, i.e., can the distorted but uninterfered IMU signal enhance the user’s speech when the microphone’s signal suffers from strong ambient interference?
We combine the best of both modalities (microphone and IMU) by designing a cooperative and self-supervised network architecture that does not rely on clean speech data from the user. Instead, using only noisy speech recordings, the IMU learns to give hints on where the target speech is likely located. The microphone uses this hint to enrich the speech signal, which then trains the IMU to improve subsequent hints. This iterative approach yields promising results, comparable to a supervised denoiser trained on clean speech signals. When clean signals are also available to our architecture, we observe promising SI-SNR improvement. We believe this result can aid speech-related applications in earphones and hearing aids, and potentially generalize to others, like audio-visual denoising.

Under review as a conference paper at ICLR 2023
SELF- SELF-SUPERVISED SPEECH ENHANCEMENT USING MULTI-MODAL DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Modern earphones come equipped with microphones and inertial measurement units (IMU). When user wears the carphone, the IMU can serve as a second modality for detecting speech signals. Specifically, humans speak their earphones (e.g., during phone calls), the throat's vibrations propagate through the skull to ultimately induce vibration in the IMU. The IMU data is heavily distorted (compared to the microphone's 's recordings), but IMUs offer a critical advantage they are not interfered by ambient sounds. This presents an opportunity multi-modal speech enhancement, i.e., can the distorted but uninterfered IMU signal enhance the user's 's speech when the microphone's signal suffers from strong ambient interference and mitigate the need of labeled data for model development?
We combine the best of both modalities (microphone and IMU) by designing a cooperative and self-superre supervised network architecture that does not rely on clean speech data from the user. Instead, using only noisy speech recordings, the IMU learns to give hints on where the target speech is likely located. The microphone uses this hint to enrich the speech signal, which then trains the IMU to improve subsequent hints. This iterative approach yiclds promising results, comparable to supervised denoiser trained on clean speech signals. When clean signals are also available to architecture, we observe promising SI-SNR improvement. We believe this result can aid speech-related applications in earphones and hearing aids, and potentially generalize to others, like audio- visuall denoising.
INTRODUCTION
Speech enhancement/denoisingg are long-standing problems in audio analysis. The recent deep learning approaches have successfully broken through the performance walls, the extent that even pre-trained voice assistants like Siri, Alexa, Google are remarkably successful (Tulsiani et al., 2020). not surprising that the bar on speech enhancement is getting raised, with newer form-factors and more challenging use-cases in the horizon. A growing domain of interest is in the context of "carables" (c.g., earphones, hearing aids, and glasses). Even though the user speaks from close to the earphone, the problem is particularly challenging because: (1) the background interference can be high in real-world public environments (e.g., restaurants, airports, busses, trains) (Schwartz) (022). (2) Users tend to speak softly, lest they disturb others around them. Finally, (3) the relatively fewer microphones on earable devices must forgo some of the array processing gains (compared to, say, table-top devices such as Amazon Alexa or teleconferencing systems). In sum, the SINR (Signal to Interference Noise Ratio) of the target speech signal can be very low in real-world scenarios.
Although challenging, unique opportunities emerge as well. Modern earphones are equipped with IMUs that sense the vibrations due to human speech (Jabra (002). Of course, the IMU's sampling rate R 400 H2, hence the recording of the human speech is heavily aliased and distorted by the non-linear human bone-channel (Blue et al.) 2013). However, ambient sounds do not induce vibrations the IMU, implying that the IMU signal remains immune to external interference. The microphone on the other hand records high quality signal from the user's mouth (44 k)d but can be heavily polluted by ambient interference. Non-stationary interference, such as voices other people, are difficult to denoise; even today's 's best denoisers (Wang& & Chen 2018), that perform remarkably well on stationary noise or on pre-trained distributions, falter against speech and music. Moreover, existing techniques mostly require clean speech for training the models. With multi-modal

---

Under review as a conference paper at ICLR 2023
Clean Audio
Noley Audio
2000
1750
1150
1750
1500
1300
1500
00
1750
1730
1750
1000
1000
1060
00
750
150
150
230
230
230
05
Tome 54
Time
1ime 34
Figure 1: Earphone measurements: (a) Audio signal recorded without interference, (b) Audio recording with interference. (c) IMU recording from earphone. (d) Zoomed in view of IMU signal.
data from both the microphone and IMU (see spectrograms in Fig. we gaps in speech enhancement and remove the necessity of collecting clean and labeled speech.
opportunity to close
We propose self-supervisee architecture that does not rely on clean speech data to train the network. Instead, we utilize the everyday, noisy recordings from the carphone that the user can record on the fly. The key idea is develop a cooperation between the IMU and the microphone, each modality can teach and learn from the other. To this end, our architecture, called IMU-to-Voice (IMUV), is composed of two separate models Translator and Denoiser.
Briefly, the Translator translates the distorted IMU signal to higher-resolution audio, and then constructs a time-frequency mask crudely identify the locations of user's speech. The Denoiser, which only has noisy speech signals, uses this crude mask to slightly enrich the user's speech signal. The Denoiser's output - the slightly enriched speech signal now offers a reference to the Translator to learn better mask, which is in turn fed back to the Denoiser to further enrich the speech signal. The iterations converge to an SNR-enhanced voice signal at the output of the Denoiser, even in the presence of strong interference. Importantly, no clean speech is needed to bootstrap or train this network; the noisy voice signal can even be at dB SINR.
Zooming out to a higher level, the results are not surprising. Given that the IMU is completely unaffected by strong interference, should be able to guide the audio Denoiser down the correct path of gradient descent. The only risk emerges from the fact that the IMU has no way validate whether its guidance is correct, and given the IMU is heavily distorted, it is easy to make mistakes. However, this is where we find that even noisy voice recording gives the needed validation the IMU, so the Translator and Denoiser can teach each other and independently descend in the right direction.
We show that our proposed two-step model actually inherits the structure of expectation maximization (EM) (?), with the likelihood and posterior functions estimated by neural networks. EM is known to be sensitive to its initial condition, similar to how the initial mask from the Translator is crucial for downstream convergence. Although our Translator is able to provide one acceptable mask, the question around the optimal mask (upper bound), or the minimally adequate mask ensure convergence (lower bound), remains open. We leave this to future research.
Summary of Results:
With help from 4 volunteers, we gathered IMU and microphone data from earphones, and injected interference from a public audio dataset into the microphone data stream. The self-supervised IMUV model is trained on this unclean dataset (at varying SINR levels). We evaluate the final denoised signal using two metrics: scale invariant signal to noise ratio (SI-SNR), and word error rate (WER) from an automatic speech recognizer (ASR) (Yu & Deng 2016). Results show that in terms of WER, self-supervised IMUV is comparable with the supervised audio Denoiser (trained with clean voice data), achicving less than 1% difference. When we allow IMUV to also train on clean signals, supervised IMUV exceeds self.supervised IMUV by 5%. Finally, when using SI-SNR as the metric, the gains are higher and more consistent with both supervised and self-supervised IMUV. In closing, we find that IMU extends only one of two advantages improve denoising performance, or relieve the user from collecting clean voice data.
can either choose to
2
MULTI-MODAL SELF-SUPERVISION
2.1 PROBLEM STATEMENT
We consider two input streams: a high-resolution audio signal H from the microphone, and a low-resolution surfcc-vibration signal I. from the IMU. Since all recordings are from everyday
2

---

Under review as a conference paper at ICLR 2023
environments, H. is composed of three parts: the speech signal from the target user, Hr, the interfering signals from nearby people, H1, and the hardwaree/ambiet noises, HN. Thus, H = HT + H1 + HN. We assume knowledge of Hr, H, H
The IMU signal L consists of only two parts: the target vibration from the user, Lr, and the hardware noise, Lx. We assume knowledge of either Lr or Lv- Also, since the vibrations are essentially an outcome of the user's speech, LT non-linear projection of Hr. i.e., Ly = f(H)). This projection is expected be different across users, depending on each user's bone, muscle, and tissue conduction properties.
The final output of our model is expected to be a denoised high-resolution audio signal HT,, containing only the speech of the target user, T'.
2.2 NETWORK ARCHITECTURE
Translator design: Figure 2 shows the proposed network architecture, with Translator top and a Denoiser below it. For self-supervision, the Translator needs to supply reference signal to the Denoiser. This means the vibration signal L at 400 Hz needs to be translated to an audio signal Hr that approximates the target speech signal. Since Hy needs to be at, say, 16 kH z, the Translator's task that super-resolution. This large up-sampling factor from 400 H to 16 KH2 is prone to overfitting with conventional nuto-encoder. Hence, we design the network as guided autoencoder to inherit earlier successes in (Lai et al. 2017). The idea to up-sample the signal in multiple stages, each stage with small up-scaling factor and corresponding stage loss. Using 3-stage decoder, we up-sample I., represented in the time-frequency (TF) domain, from 400 H 2 to 800, 3200, and finally to 16 HHz. The final loss is regularized by the individual stage losses curb the decoder from overfitting. Of course, the reference signal for computing loss is the noisy audio signal H from the microphone (but in subsequent rounds, becomes the output of the Denoiser). Though we train the translator with noisy audio as ground truth, we observed that the translated audio (mask) does not contain high energy for noises. As the noises are absent the IMU data and are random, i.e, they have different sources and are at various places for different samples, the translator can not learn the noise signals. On the other hand, the translator can identify the target audio as its aliased version present in the IMU data.
Translator: J. Hy
Guided auto-encoder
Mask M
Masked noisy audio Hy'
IMU spectrogram
IMU vibrations I,
Lerans
Denoiser: (H, L) By
L denotser
Denoised audio f7
jenoissrr
Noisy audio
Audio & IMU spectrogram
& IMU vibrations
Deep auto-encoder
Figure 2: Proposed architecture composed of Translator top, that attempts super-resolution from L to H. and generates a mask. The model below is a multimodal Denoiser that uses the mask to enhance the target audio. The enhanced target audio is returned to the Translator to improve the super-resolution, resulting in an improved mask. This iteration continues.
Mask generation: The heavy up-sampling, and that too with a noisy reference signal, cannot be expected yield fine-grate fine-grained speech (with accurate amplitude and phase in each time- tmmefrequency (TF) bin). Instead, the spectrogram can only expected to have marked TF bins where the target audio dominates. Hence, we treat the output spectrogram of the guided auto-encoder as mask (Vaswani
3

---

Under review as a conference paper at ICLR 2023
et al. 2017) for Hr this map M gives us p(Hr (Hr) for each TF bin. To derive an approximate audio signal, Hy pass the mask through a Sigmoid function first generate a mask, and then multiply the mask element-wise with the denoiser's output Hr to obtain H - HT @ (M). Of course, in the initial round of mask generation, HiT H.
Denoiser design: The Denoiser's imput is both H and and the output should be the denoised signal Hr. One option is train a network to learn the end-to-end mapping from the input data (H, L) to the desired output Hy. However, without clean audio data, this mapping does not converge well, or is slow and data hungry. More importantly, we know that a consistent mapping exists between audio and IMU, dictated by the bone channel that conducts the throat's vibration. Our network architecture must incorporate this knowledge.
We design an auto- auto-rn (AE) using only the audio H as input, however, we force part of the latent space to match the IMU signal L. Specifically, we design the AE's latent space as Z = {Zz. Zx (see Fig. (3 and force Z to match the IMU data L (we detail the loss terms in the next section). Since the IMU only contains low frequencies (s 400 22) and contains interference,
intend the remaining latent space ZH = Z \ Zr to capture the gap between audio and IMU. Hence, we model (lu) 7 where Z.(hd) is representation of the target's s high frequency components, and Z is a very compressed representation of all the interfering signals.
Zu (Z]
Assuming the interference is uncorrelated the target user's speech, we add a loss term between (all) and the IMU signal I, to enforce contrast between them. We also add another loss term between Z/haa) and L to enforce their correlation. Finally, the decoder uses only {Z z((hi) reconstruct the denoised audio signal, Hr, and trains it against the Translator's 's output, Hy.
IMU input (20,211
Append as 2dd channel
Encode in=32, Im), pn2]
220.211
# [800,21]
input
Masponllp)
ZeI [30,21]
In-2. 1-9 5-1)
Decade
Audio output Ay (800,21)
Camidiiin
Inded Ind:dr/dpe4)
Latent space Z
Skip connection
Figure 3: Denoiser architecture: The audio is encoded into a latent space, one part of which mimics the IMU and the other parts are representations of high frequency speech signals and interference.
The Denoiser is almost ready, except for one small detail. To utilize the IMU data J. during test time as well, we concatenate as a second channel, alongside Zr. The 2 channels serve as the first layer of the decoder. To match the dimensions, ZH progresses through one additional layer of decoding. Although we design Z,, to match L, it's important to concatenate because it does not contain any ambient noises. Since Zr. and L both represents the low-frequency target signal, subsequent layers will learn the weights from both modalities.
The Translator begins by training against the noisy audio H. After N, 25 epochs, freeze the Translator and use its output (i.e., the masked audio Hr) to train the Denoiser for the next N₂ = 75 epochs. We denote (N, + N2) epochs as a training cycle. We then start the next cycle by freezing the Denoiser and using the denoised signal Hr from the previous cycle to train the Translator. The iteration is performed for C 3 cycles.
Training:
Fig. shows snapshots from the start and end of the training process. The first column in Fig. 4 plots the spectrogram of clean target speech Hy on top, and the interfered audio H at the bottom. The top of the second column shows the Translator's 's mask after the first training cycle; evidently, IMU offers a crude map M at this time. The bottom of the second column plots the Denoiser's 's output when has been trained using the masked audio, H = H @ a (M). The top of the third column shows the mask after the last training cycle- the improvement is visible. We observe that the Translator converges reasonably well because the interference varies over time, preventing the Translator from overfitting to the interference. Finally, the bottom of column 3 shows the denoised audio Ay using the final mask; this is our final output.
4

---

Under review as a conference paper at ICLR 2023
1500
1500
Erequetty (1ke
1000
1000
1000
500
500
500
Time
Tirre
2000
2000
2000
(ts)
(F2))
(to)
1000
1000
500
500
500
Time
Time
Time
Figure 4: (Column 1) Spectrogram of clean target signal HT on top, and the noisy microphone signal H at the bottom. (Column 2) Translator's mask after the first cycle on top, and the Denoiser's s output after the first cycle at the bottom. (Column 3) Translator's mask after the last cycle top, and the Denoiser's output after the last cycle at the bottom.
Figure 5: IMU placement for data collection
2.3 LOSS FUNCTIONS
Translator's loss function:
Aggressive up-sampling is prone overfitting, the Translator incorpo- rates a loss function at each stage of the guided auto-encoder. The final loss is a convex combination of Mean Absolute Error (MAE):
Disn w. (1ID1(()) - T(x)
Lirons Expp(()
(1)
Σi=1
where n is the number scale-up stages; wi the weight for stage i; D-1(z); the Denoiser's output, down-sampled to match stage i; and T(x); is the Translator's output after stage i.
The Denoiser's loss function is composed of three terms follows:
Ldenoiser = LH + ' * Lr, + X2 Lo
where L1 denotes the audio reconstruction loss; L.. is the IMU loss from the latent space; Lo is the correlation loss, and À the weighing scalar. The loss functions are defined as:
LH E Pemp(w))l - D(x)lla
L1 R - Z
(3)
Lc = Bawp(u)
Σ abs( abs(correoe 2
(hi)) Σ abs(correce ((, TT,
(4)
2,5
The Correlation loss Le aims to capture the uncorrelated relationship between the IMU signal I, and the interference Z(all), well the correlation between the IMU L and the high frequency components of the speech, T The negative sign for the second term indicates that higher correlation reduces the loss function (and vice versa for the first term). In the equation, i, are the indices of the dimensions of L, 2 and z(hh) We calculate the absolute value of correlation coefficients to account for harmonic behaviors in the speech signals. Algorithm T shows the pseudo- code for training self- seffssuper IMUV.
2.4 ITERATIVE BEHAVIOR
Our iterative training process inherently mimics the Expectation-Maxinn Maximization framework (Dempster al. 1977). For ease of explanation, consider L and H denoting the spectrogram of IMU and audio (instead of the time domain signals). Observe that the Translator poses as the first E-step; it calculates the distribution of the latent variable, which is the mask M {m,) for each TF bin i. Given the low-resolution vibration data L as well as Hr from the Denoiser, outputs p( {m²} L, Hrr, where m, p( Hr]HH) is the mask of the target signal. By injecting the denoiser loss into the distribution, we get the E-step function:
Q Q 18, =E L.y [denniser]]
=E EMIL [F (flr Ar, M)]
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Self-supervised IMUV Training (H,L, Translator, Denoiser)
1: Hr t- H
2: for e Ooo
3:
Translator t- Ltrans Hr)
4:
Translator
M Translator.train((ppttt (iain &- L,, epoch +- No)
5:
poonslatoo-prreiiclee HT' + & (M)
6:
7:
// Freeze the translator and train the denviser
8:
Denotser t- Ldenosser (HT')
9:
Denoiser.train
(H, [H, epoch t- Na)
HT +- Denoiser Denoiser.predicc L)
11: // Freeze denoiser and train translator in next cycle
end for
where Hr,eew is the Denoiser's output in the new cycle, and F F(HT,eew Hr, M) = || new Hr @ MlII. Lastly, the Denoiser training be viewed the M-step, where find the best target signal estimation nee minimizing the expected loss Q.
EXPERIMENTS AND RESULTS
3.1 EXPERIMENT SETUP:
Dataset Construction. We recruit volunteers and ask them to wear a normal earphone and a separate IMU (Fem 2022) near their ears Figure shows the set-up. separate IMU is needed since today's 's earphones do not make the recorded IMU data accessible. Each volunteer speaks 39 different keywords prescribed by the Google's Speech Command dataset (Warden 2018). as well as wakewords like Google, Siri, Bixby, and Alexa each word is repeated 10 times. The measurements are performed in a quiet room, and serve as HT. The earphone's microphone samples the audio at 44.1 K = and we sub-sample to 4 K H to mimic phone calls over earphones (Lee et al. (016). The IMU sampled at 400 Hz. We have currently published the dataset on GitHub anonymously (IMU) 2022b). To the best of our knowledge, this is the first speech dataset composed of synchronized audio and IMU vibrations from the car-location.
To synthesize noisy signals H. randomly draw audio samples from Google's speech command dataset (Warden 2018), containing voices of 2, 618 human speakers. These samples serve as Hj- Unless specified otherwise, we synthesize H at dB SIR. The IMU signals, on the other hand, need no synthesis, so we automatically have L. The total dataset (H, L) is now ready and extends over 990 hours.
Performance Metrics.
(i) We use scale invariant signal to noise ratio (SI-SNR) as the main evaluation metric (Le Roux ct al. 2019). SI-SNR is computed as the ratio between the correlation our output signal with the target signal, as follows:
AG |
H1
SI-SNR 10log10
Hr H 1: ||
(ii) We also report the word recognition accuracy of the denoised signal, using Google's Key Word Spotting Classifier (KWS) with 10 and 35 classes, denoted as KWS10 and KWS35, respectively We re-trained the KWS models using 4 K KH2 audio data.
(Rybakov et al. 2020 Goo 2022).
Models for Comparison.
(1) Unprocessed:
The raw audio without denoising A recent speech enhancement model (Park & Lee 2016) trained on clean speech; 216K parameters.
(2) Supervised Denoiser:
(3) Supervised IMUV:
Our proposed model trained clean speech; 60K parameters.
(4) relf-Supervised IMUV:
Our proposed iterative model in Figure 180K parameters.

---

Under review as a conference paper at ICLR 2023
For fairesss, train all the above models using the same dataset. We employ "no overlapped words between training and testing set" policy. Around 16% of the dataset set aside for testing and the rest for training. We also ensure that Hr and H, are different words. We publish the codes in the GitHub Repository (IMU) 2022b), and denoised audio samples on the GitHub site (IMU) 2022a).
3.2 RESULTS
Overall performance: Table reports comparative results across all metrics and models. Unsurpris- ingly, supervised IMUV substantively outperforms all models across all metries. On the other hand, self-supervised IMUV is comparable to supervised Denoiser, outperforming marginally in SI-SNR but weaker in KWS metrics. This distills the contribution IMUV to speech enhancement as follows: we can either choose to obtain 0.6 I (personal) to 0.2 3.1 (general) SI-SNR gain while requiring the user to provide clean speech data or relieve the user from the data collection burden at the cost of sacrificing that same performance gain. Both not possible yet.
Table 1: Performance comparison across models and metrics.
Personal Model
General Model
SI-SNR
Accuracy(%)
Acc.(%) KWS35
SI-SNR
Acc. (%)
Acc.
Models
(dB)
KWS10
(dB)
KWS10
KWS35
Unprocessed
5.3
49.24
48.92
5.3
49.24
48.92
Supervised Denoiser
8.3
65.72
64.02
2.9
51.37
46.22
Supervised IMUV
9.9
69.02
66.78
6.0
55.27
52.96
Self-supervised IMUV
9.3
64.45
62.83
5.8
48.67
45.19
Personalized versus generalized models: Table I compares the accuracy of personalized model (where training data is drawn from the target user) and a generalized model (where the training data drawn from users other than the target user). Understandably, the accuracy improves across all models when the training data is personalized, however, the self-supe IMUV performs slightly better than the supervised Denoiser for generalized models. This implies that the marginal gain from the IMU is higher in generalized models, since with personalized models, the supervised Denoiser may able to learn user-specific voice patterns (e.g., base frequency, pitch, prosody, harmonies, etc.) Observe that the generalized models are valuable during the phase when on-the-fly -ly data is being gathered for self-supervision. In the case carphones, for example, the user would buy an carphone with the pre-installed generalized model and personalize it over time by training with recorded data from everyday life.
Variation across users: Figure both the SI-SNR and KWS accuracy across 4 users. supervised IMUV consistently benefits from both the IMU and the clean-data supervision, and as expected, the mostly comparable. With users who tend to produce high frequency voices/sounds, the gain from IMU is relatively less since the IMU does not capture these frequencies; the supervised Denoiser avails an advantage for these voice characteristics.
supervised Denoiser and self-supervised IMUV
15
100%
100%
KWS10 Accuracy
KWS35 Accuracy
SI-SNR (aB)
80%
80%
60%
60%
40%
40%
20%
20%
0%
0%
User
User
User
User 4
User
User 2
User
User 4
User
User 2
User
User 4
Supervioed Denoleer
Belr-Bupevised IMUV
Supervised IMUV]
Figure 6: Performance across different users for (a) SI-SNR. (b) KWS 10, (c) KWS 35 accuracy.
Varying mix of clean and interfered data: self-supervised IMUV has trained entirely interfered data. Figure 7 shows the effect of some fraction the training data being clean; the testing data also follows the same training data distribution. This is likely to be the average application, where the user may sometimes speak in a silent environment. In such settings, the gain of self-supervised IMUV over supervised Denoiser is not affected by the fraction of clean data.
in our earphone
7

---

Under review as a conference paper at ICLR 2023
Varying interference: Figure plots SI-SNR against varying SIR (Signal Interference Ratio) of the training/testing data. The contribution IMU grows as the SIR drops since the additional IMU modality becomes more valuable under more noisy environments. This explains why self-supervised IMUV outperforms supervised Denoiser at low SIRs, but worse at higher SNRs where the gain from IMU's guidance is offset by the penalty of self-supervision.
20
20
SI-SNR (dB)
15
SI-SNR (dB)
15
10
5
Self Supervised IMUV
Supervised Denoiser
5
Self- SuperSuper IMUV
Supervised Denoiser
Supervised IMUV
0%
25%
50%
75%
10
Percentage of Clean Samples
Noisy audio SIR (dB)
Figure 7:
IMUV offers gain with increasing percentage of clean-data in training samples.
Figure Performance versus SIR regimes.
Effects of the skip connection in the Denoiser Architecture: Although our network is configured by several hyper-parameters, we report the performance variation against one of them - the weights assigned to skip connections. In a conventional auto-encoder (AE), skip connections are used to mitigate vanishing gradients in the deep network (Tong et al., 2017). However, in this work, we are injecting the IMU imput L in the latent space, which will be "skipped" by the skip connections. Hence, we analyze the effect of regulating the skip connection in IMUV. Table for different skip connection weights in the Denoiser network. Since the skip connection carries some fine-grained speech patterns (that may not be captured in the latent space), the SI-SNR drops if we remove the skip connection (0%). Here, 0% skip means that we do not change the hierarchy but just change the weights of the skip connections to 0%, when it joins the latent space. On the other hand, increasing the skip connection weights (100%) reduces the gain from clean IMU in the latent space. Consequently, we use a moderate skip connection weight in IMUV.
the performance
Table 2: Performance for varying weights on the skip connection in the Denoiser. The metrics are (SI-SNR(dB) KWS 10 accuracy(%)) KWS 35 aceuracy($))] P% denotes the skip connection contributes P% weight when joining the decoder latent space. 100% setting ignores all inputs from the latent space, and only takes the results from the skip connection.
0%
10%
50%
100%
Self-S. IMUV
11.0/ 68.4 70
13.9 /3.9/74.2/73.
/5.1/653//83
11.1 / 70.1
Super. IMUV
12.1 / 80.2 79.8
14/814/77.8
12.5
13.6 83.0 83.0
RELATED WORK
Multi-modal speech enhancement: The closest work this paper is SEANet (Tagliasacchi et al.) 2020) that uses both audio and IMU through wave-to- wave convolutional generator and discrimina- for architecture. The core architecture builds on (Kumar et al. 2019) and achieves promising SI-SDRi (Scale Invariant Signal to Distortion Ratio Improvement). SEANet uses accelerometer data which is unaffected by ambient noise to partially reconstruct user speech. The key difference between SEANet and our work is that, SEANet assumes the availability of clean speech reference which places considerable onus on the user to acquire which is impractical. Our work leverages the self-supervision provided by the IMU to create the clean reference without additional burden on the user.
Another work that is closely related to ours is [Wang et al. 2021). This work uses "alias unfolding" to reconstruct user speech from low resolution IMU motion signals. This work is not multi-modal as the core idea of this work is reconstruction of spoken words from IMU signal. This different from work which uses IMU to denoise noisy audio. However, one commonality is that, translator does the job of alnti-aliasing" to upsample a low resolution signal (IMU) to a high resolution signal (speech) which is similar to "alias unfolding" of (Wang al. 2021
8

---

Under review as a conference paper at ICLR 2023
Multi-modality learning: In recent years, many ideas have used synchronized audio and visual modalities et al.] 2018: Gabbay et al. 2018 Ephrat et al. 2018 Afouras et al. 2018: Lu et al. 2019) for speech related applications (e.g.. enhancement and source separation). IMOV targets different scenario (e.g., outdoor, mobile) where a camera is unavailable to record the user's face/lips.
Kashyap et al. 2021 have incorporated self- supervision in audio processing. Authors in Waan et al.] 2020) learn a latent representation of a limited set of clean speech and uses noisy speech to share a latent representation with the clean examples, reducing the burden of clean data. This bears similarity to IMUV, however, we face the challenge of not having initial clean information to descend down the correct gradient. The work done by (Kashyap et al.) (02T) uses 20 layered Deep Complex U-Net to perform self-supervised speech denoising with noisy speech targets. The authors leverage two key conditions that the noise distribution must adhere to: (1) The input and target noises are sampled from zero-mean distributions and are uncorrelated to the speech input. (2) The correlation between the noise the input and in the target is close to zero. In contrast, in IMUV make no such assumptions about the noise distribution as the availability of another modality (IMU) provides more information.
Self-supervision:
Several work (Chen et al. 2021:
The performance of any denoiser is impacted by the representation of the speech signal. Authors in (Nossier et al. 2020) compare the impact of TFF representations on noise reduction. Most deep learning approaches can be broadly classified based on whether they operate on raw audio waveforms (Kolbæk et al. 2020 Pandey & Wang 2019) and WaveNets for raw audio waveforms (Germain al. 20018). STFT spectrograms (Kumar & Florencio 2016 Lu et al., (013)) or other audio representations like Mel Frequency Cepstral Coefficients (MFCC) (Pirhosseinloo & Brumberg 2018). Our work uses STFT spectrograms as one possible compact feature representation.
Input Representation:
LIMITATIONS, GENERALIZATION, AND CONCLUSION
Mutual information between multiple-modes: The IMU signal L is a projection of the audio signal H onto lower dimensional space. This paper shows a very specific instance in which the projected low-dimensional) signal can teach/optimize the higher dimensional signal without any supervision. The generalization of this question is of interest, i.c., how low does the lower dimensional projection need to be before (self-supervised) convergence fails. Perhaps mutual information [[. H) could shed light on this line of questioning, leading to possible notion around the "capacity of self-supervision" We leave this to future work.
Alternating epochs between Translator and Denoiser: We have empirically chosen the number of epochs for the Trans lator and Denoiser as they alternate in a cycle. Although rigorously designing hyperparameters is difficult, we believe there is room to make this decision tighter, or adaptive. One idca is to model them in the saplore-exploi framework, where running (the Translator Denoiser) for longer epochs is the exploration phase, and switching is exploitation. We leave this to future work as well.
We envision generalization of IMUV in audio-visual speech enhancement as well. Consider a laptop camera visually recording the user's 's lip movements (e.g.. in a video call), and the microphone recording her voice signal, polluted by ambient interference. Observe that the lip movement is akin to IMU in IMUV, where it is uninterfered and a low-dimensional projection of actual speech. We believe our self supervisdd architecture is relevant here, offering the possibility to relax the need for clean audio data.
Beyond Audio and IMU:
Multi-modal signal synthesis: This paper has aimed to enhance the target speech signals where the signal exists (only interference has polluted them). In other applications, the target signal may be absent (e.g., clouds completely occluding satellite image, or a night vision camera unable to see some parts of the environment). The guidance from the second, low-dimensional, modality (e.g., RF radar that penetrates through the cloud) would help the denoiser learn the latent distribution of the target object. This learned distribution could be harnessed to synthesize/reconstrued the occluded parts, perhaps by training a GAN on top of the Denoiser.
Conclusion:
This paper presents IMUV, self-supervised speech denoising network that leverages low resolution (IMU) modality. With an iterative network design, we show that the low-modality signal effectively guides the multi-m modal denoiser. Results are promising, and could aid wearable- audio applications in the near term, and other modal mixture-man mixture-models in the future.

---

Under review as a conference paper at ICLR 2023
REFERENCES
2022
URL
Lemtoo..o/Petttceeccoo Xr-.::/.000...1.0000000000-----0
googl-teeearcc//wssstttemmmiii at master, 2022.
URL
IMV demo, 2022a. wRL Utt.atee.//aa1--.....tt......./0
IMUV dataset, 2022b. UtLLptttss///ithbb.mmm///2222223
Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. The conversation: Deep audio-visual speech enhancement. arXiv preprint arXiv: 1804.04/2,, 2018.
Misty Blue, Maranda McBride, Rachel Weatherless, and Tomasz Letowski. Impact of a bone conduction communication channel on multichannel communication system effectiveness. Hum. Factors, 55(2):346-355, April 2013.
Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, al. Wavlm: Large-scale self-supee pre-training for full stack speech processing. arXiv preprint 1XXv:2110.13900 2021.
Arthur P Dempster, Nan M Laird, and Donald Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1): 1-22, 1977.
Ariel Ephrat, Inbar Mosseri, Oran Lang. Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein. Looking to listen at the cocktail party: A indeeendndd audio-visual model for speech separation. arXiv preprint arXiv: /804 03619, 2018.
Aviv Gabbay, Ariel Ephrat, Tavi Halperin, and Shmuel Peleg. Seeing through noise: Visually driven speaker separation and enhancement. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 3051-3055. IEEE, 2018.
Francois Germain, Qifeng Chen, and Vladlen Koltun. Speech denoising with deep feature losses. arXiv preprini arXiv: 1806. /0522, 2018
Jen-Cheng Hou, Syu-Siang Wang, Ying-Hui Lai, Yu Tsao, Hsiu- Wen Chang, and Hsin-Min Wang. Audio-visual speech enhancement using multimodal deep convolutional neural networks. IEEE Transactions on Emerging Topics in Computational Intelligence, 17-1:17 2018
Jabra. Commercial carphnees equipped with imu, 2022. URL https: www. jabra. com/ busineas/office-heee htw.:eiiiiieeeattteeeeeeeaaeeeaa
mot mon 6600 900-105
Madhav Mahesh Kashyap, Anuj Tambwekar, Krishnamoorthy Manohara, and S. Natarajan. Speech Denoising Without Clean Training Data: A Noise2Noise Approach. In Proc. Interspeech 2021, pp. 2716-2720, 2021.
doi: 1021445//ieespecchh220111111 20211/21
Morten Kolbæk, Zheng-Hus Tan, Søren Holdt Jensen, and Jesper Jensen. On loss functions for supervised monaural time-domain speech enhancement. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:825 838, 2020.
Anurag Kumar and Dinei Florencio. Speech enhancement in multiple-noise conditions using deep neural networks. arXiv preprint
2016.
Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron C Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. In H. Wallach, Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox. and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL htt.o//oonuuuttppo////ooocosnnn. wtw.::eeeeeddin....eeeeeeee cc/ f
10

---

Under review as a conference paper at ICLR 2023
Wei-Sheng Lai, Jia- Bin Huang, Narendra Ahuja, and Ming Hsuan Yang. Deep laplacian pyramid networks for fast and accurate super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 624-632, 2017.
Ming-Hsuan
Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R Hershey. Sdr-half-baked or well done? ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 626-630. IEEE, 2019.
Juho Lee, Younsun Kim, Yongjun Kwak, Jianzhong Zhang. Aris Papasakellariou, Thomas Novlan, Chengjun Sun, and Yingyang Li. Lte-advanced 3gpp -13/14 an evolution toward 5g. IEEE Communications Magazine, 54(3):36 42, 2016. doi: 10.1
Rui Lu, Zhiyao Duan, and Changshui Zhang. Audio-visual deep clustering for speech separation. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 27(11):1997-171, 2019.
Xugang Lu, Yu Tsao, Shigeki Matsuda, and Chiori Hori. Speech enhancement based on deep denoising autoencoder. In Interspeech, volume 2013, pp. 436-436440 2013.
Soha A Nossier, Julie Wall, Mansour Moniri, Cornelius Glackin, and Nigel Cannings. A comparative study of time and frequency domain approaches to deep learning based speech enhancement. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2020.
Ashutosh Pandey and DeLiang Wang. Tenn: Temporal convolutional neural network for real-time speech enhancement in the time domain. In ICASSP 2019-2019 JEEE International Conference on Acousties, Speech and Signal Processing (ICASSP). pp. 6875-6879. IEEE, 2019.
Se Rim Park and Jinwon Lee. A fully convolutional neural network for speech enhancement. arXiv preprint
2016.
Shadi Pirhosseinloo and Jonathan Brumberg. new feature set for masking- monaural speech separation. In 2018 52nd Asilomar Conference on Signals, Systems, and Computers, pp. 828-832. IEEE, 2018.
Oleg Rybakov, Natasha Kononenko, Niranjan Subrahmanya, Mirko Visontai, and Stella Laurenzo. Streaming keyword spotting mobile devices. arXiv preprint 2rXiv:2005.067200 2020.
Eric Schwartz. Alexa will automatically adjust volume to be heard when it's loud, serexaawwwlll miill-automatienn
2022. URL https::///// http::://voocobboo http://.ceoo...a//////////////// adjust- volumm to be heard when its loud/]
Marco Tagliasacchi, Yunpeng Li, Karolis Misiunas, and Dominik Roblck. SEANet: A Multi- Modal Speech Enhancement Network. In Proc. Interspeech 2020, pp. 1126-1130, 2020. doi: 10.2 10021447//nterspeech.222 200.cchhh.2221556.
Tong Tong, Gen Li, Xiejie Liu, and Qinquan Gao. Image super-resolution using dense skip connec- tions. In Proceedings of the IEEE international conference computer vision, pp. 1799-48077 2017.
Hitesh Tulsiani, Ashtosh Sapru, Harish Arsikere, Surabhi Punjabi, and Sri Garimella. Improved training strategies for end-to-end speech recognition in digital voice assistants. In Interspeech 2020, 2020. URL https:: http://wwwwwaaa..ooo//amm. amazon.
Lmprovee training sorr -ndooo------- seeedd-ppecc--he www.onition-ononooe te cigital--oolo-
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Del.ing Wang and Jitong Chen. Supervised speech separation based deep learning: An overview. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26010)::702-1726, 2018.
Tianshi Wang, Shuochao Yao, Shengzhong Liu, Jinyang Li, Dongxin Liu, Huajie Shao, Ruijie Wang, and Tarek Abdelzaher. Audio keyword reconstruction from on-device motion sensor signals via neural frequency unfolding. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 5(3):1-29, 2021.
11

---

Under review as a conference paper at ICLR 2023
Yu-Che Wang, Shrikant Venkataramani, and Paris Smaragdis. Self-supervised learning for speech enhancement. arXiv preprint arXiv:2006. 10388, 2020.
Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint 18X:::1804,0329,, 2018.
Dong Yu and Li Deng. Automatic speech recognition, volume 1. Springer, 2016.
12


Reviews:
[{"review_id": "wrX4fDL9am", "reviewer": "Reviewer_euGt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims at doing denoising (speech) without supervision. It pursues self-supervised learning while leveraging paired data from another modality (vibration IMU i.e. inertial measurement device). Speech is sampled at high rate while IMU data is sampled at only 400Hz. The proposed approach consists of two models (translator and denoiser) which are trained alternatively for 3 cycles. This optimization is seen akin to EM algorithm. The work is different from prior works in many respects. Previous works either don't explore self-supervised learning or use a more idealistic experimental setup. This work compares results with a regular Supervised Denoiser and find that when clean data is also available, their technique is close to Supervised Denoiser. When clean data is not available, the proposed approach helps a lot in SDR and KWS metrics.", "summary_of_the_review": "The choice of problem is very apt given the current state of earable technology. It is interesting to see self-supervised paradigm being explored in an interesting setup. Denoising without clean data is a recently popularized problem which can have great implications in future. Authors develop connection to EM for their two-network alternating learning model. Results are encouraging but not very strong. Hence, this paper is of theoretical importance and is a good step in their choice of problem.", "strength_and_weaknesses": "Strengths:\n1. The experimental setup is vastly improved over the previous works.\n2. In the world of earables, the choice of problem is very apt.\n\nLimitations:\n1. Perhaps a basic issue, but the intuition for why this should work is missing. It deserves atleast one paragraph.\n2. Novelty should discussed further in Section 4 (Related work) under \"Self-supervision\".\n3. Number of users to collect data is low. Age/gender details are missing.\n4. Correlation loss under equation 2 has missing equation number. Also, why is weight not different for the two (postive and negative) terms?\n5. Results could be more encouraging -- especially, given the fact supervised denoiser is working so well already.\n6. Semi-supervised setup/fine-tuning may be explored in future.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kkw8heWX5R9", "reviewer": "Reviewer_uCfP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a self-supervised speech enhancement method for a scenario when both a microphone recording and a signal from an accelerometer sensor is available. Activity of the latter signal is well correlated with the speech signal activity, while it's also not corrupted by the acoustic noise from the environment, e.g., by interfering speech and noise. This fact is often exploited by in speech enhancement for improving the speech signal estimate.", "summary_of_the_review": "The paper is interesting and self-supervised training is of interest for many applications.\nHowever, the paper is not covering the existing work well enough, it's not clear how it performs to state-of-the-art speech enhancement systems, and some aspects of the design are not clear.\nIf the above would be addressed, I believe the paper could be considered for publication. However, in its current form, it's just not good enough from the perspective of this reviewer.", "strength_and_weaknesses": "The main strength of the paper is the self-supervised training formulation, since it does not rely on clean speech signal data.\nOne of the main weaknesses of this paper is that it does not mention a lot of relevant prior work.\nMore specifically, the authors cite the following relevant work:\n- SEANet: A Multi-modal Speech Enhancement Network, 2020\nHowever, this topic has been covered extensively in more classical speech enhancement work, e.g.,\n- Speech enhancement for an in‐ear communication system using bone‐conducted and acoustic signals, JASA 2001\n- Speech enhancement for an in-ear communication system using optimal-filtered accelerometer signals, Acoustics 2002\n- Survey of speech enhancement supported by a bone conduction microphone, ITG Speech communication 2012\n- Time-Domain Multi-Modal Bone/Air Conducted Speech Enhancement, 2020\nany many references in the above-mentioned.\n\nFurthermore, even the single mentioned multi-modal speech enhancement method (SEANet) is not described correctly. More specifically, the paper claims that \"uses an IMU that samples at 4 kHz. In contrast, IMUV is self-supervised [...] and operates entirely on 400 Hz, the viable sampling rate for real-world earphones.\" This is not exactly true. Figure 4 in the SEANet paper depicts the performance in speech and noise for accelerometer sampling rates as low as 160Hz. Moreover, those results indicate that sample rates as low as 400Hz are not severely impacting the performance of the system.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cNQzWettoO", "reviewer": "Reviewer_NmkT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A self-supervised speech enhancement method that uses both audio and IMU data is presented. The method is self-supervised in the sense that the audio does not need to be a clean audio signal. The method is targeted towards earphone recorded audio data.\n\nThe self-supervision is handled with an alternating minimization like training algorithm where a \"translator\" model that maps IMU recordings to spectrogram masks that multiply the input audio spectrogram and another \"denoiser\" model that takes both IMU and input spectrograms and tries to predict a clean spectrogram without having access to the clean spectrogram. The denoiser model has an internal representation of the input audio and IMU and it tries to pick the part of the representation that correlates with the IMU data to reconstruct the spectrogram. By doing this, it is assumed that the part of the audio that correlates with the IMU is estimated only which refines the estimated clean audio output. This new estimate is used in the \"translator\" loss function once again. Thus, each module provides the target for the other module in an alternating fashion. Additional \"correlation\" losses are used in the denoiser to force the model to output signals that correlate with IMU data.", "summary_of_the_review": "The paper introduces a method for self-supervised training of speech enhancement models.\n\nI think the method described in the paper has a risk of divergence during training due to complexity of the training method. I think there are more straightforward ways to perform self-supervised training for speech enhancement. One of them is to use MixIT method for speech enhancement that is described in the MixIT paper. In that approach, we add noises to existing data (whether the data is clean or not) and then separate it into three outputs which should satisfy the MixIT loss. A simplification of this method is something called pseudo-speech-enhancement in a paper (Siwaraman, Kim and Kim) which can also be tried. In that one, we just add noise to data and assume the data is clean whether it is clean or not.\n\nIn the dataset, an interfering speech is added to the clean speech utterances but it is not always realistic (as also mentioned in the paper). The interference could be environmental noise, or silence or another speaker. It would be more interesting to study the case where each alternative is added with some probability. Also, the interference would typically have reverberation which is not used in the dataset.\n\nIn the models for comparison, it is not clear how \"Supervised IMUV\" is trained. Is it trained using alternating minimization, or is it trained in a straightforward way with noisy input and clean output (so no translator is used?) ? This should be clarified. Also, for \"supervised denoiser\" model, why not use the same architecture without the IMU input?\n\nIn Table 1, when using a general model, the Si-SNR gets worse for the \"supervised denoiser\" which probably means the data is not enough to train a reliable denoiser.\n\nIn the dataset, the interference is at a fixed SIR. In Figure 8, the SIR value is changed, but it is the same for train and test data. I think to be more realistic, the SIR values should be in a realistic range for both training and test data.\n\n\n\n", "strength_and_weaknesses": "Strengths:\n1. New speech + IMU dataset.\n2. New self-supervised enhancement model based on alternating minimization training.\n\nWeaknesses:\n1. Comparison with more direct self-supervised techniques like pseudo-SE or Mixit like training when adding IMU data to the input.\n2. Dataset assumes there is always interfering speech which is actually quite rate in real use.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ou_Jzh4Mx0", "reviewer": "Reviewer_L1Xs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to utilize IMU signals for providing self supervision to audio signals for the problem of speech enhancement. This in itself is a good idea, and some that reasonable from the perspective that bone conduction is measurable for low level frequencies. The authors propose a simple algorithm the evaluate this. The evaluations are minimal and the presentation is good.  ", "summary_of_the_review": "As mentioned in the weaknesses, the main aspect is generalizability to reasonable sized datasets. The idea in itself is good/interesting. Additionally many details about the data, the model and its robustness are missing, and will improve the contribution. ", "strength_and_weaknesses": "A meta comment is that, although the idea and proposal makes sense, since the evaluations set consisted of only 4 people, its uncertain how the proposal generalizes to large datasets, with realistic and noisy scenarios (e.g., noise in the collected samples). \n\n1) Given that IMU signals are only relevant in low-frequency regimes, its important to understand the data more carefully here. Firstly how much variance there is in the IMU data that was collected from the 4 people; and if that is replicated to more people how would that change be?\n2) We need to understand the trends as iterations progress i.e., how much quantifiably are the attention maps evolving as iterations increase and how much effect does that have on the overall performance? \n3) Can you comment on the non-monotonic trends of self-supervised and supervised IMU models in Figure 8? Is this an aberration from data? \n4) A natural baseline it to take the average of all the IMU inputs/spectra from the datasets and use that single average spectrum to regularize the enhancer? This is mainly because the IMU signals have smaller range compared to natural speech and so the networks may end up using an average signal from the dataset unless we have large variance in the dataset (goes back to comment 1) and meta comment above). \n5) Speech enhancement community has been quite active in the past decade and there have been more audio-only enhancement algorithms since Park,Lee 2016. Whta was the rationale for not choosing more recent baselines? \n6) Word recognition accuracy may not represent the subtle changes in signals resulting from different enhancement algorithms? What was the rationale for using this instead of say traditional perceptual/quality metrics like pesq, mos etc. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "G_gn8qkMKNh", "reviewer": "Reviewer_dV4m", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a self-supervised approach for training a speech enhancement network, given data from an inertial measurement unit (IMU) and audio recorded from an earphone microphone. The IMU captures a nonlinear and lower-frequency version of the user's speech through the bone condution channel. The proposed approach iterates between applying a \"translator\" model that predicts a mask M upsampled from the lower-frequency IMU signal L (uses 3 stages of upsampling, 400 Hz -> 800 Hz -> 3200 Hz -> 16 kHz). This mask M can be applied to the output of a \"denoiser\" stage, which takes audio mixture as input and conditions on the IMU signal L, and predicts denoised audio \\hat{H}_T. Then the denoised audio is fed into another stage of the translator model, and the process is iterated. Experiments use 3 iterations of this process. The translator has a loss between the translator's output (the mask M applied to mixture audio H) and the denoiser output. The denoiser has multiple losses: reconstruction loss between output and translator's output, a loss that promotes part of its intermediate embedding at the middle of the U-Net matches L, and a correlation loss that minimizes the correlation between L and embedding for higher-level features (modeling the gap between audio and IMU), and maximizes correlation with the other component of Z and L. \n\nThe paper presents a new dataset of 4 participants with earphone and IMU, saying multiple keyword multiple times, and interfering speech audio is added. Performance is measured with SI-SNR (not clear if it's spectrogram or waveform domain) and keyword accuracy error. A supervised baseline with different architecture is trained on the same data. The proposed method is trained with noisy H (self-supervised) and clean speech for H (supervised). Method are also compared on personalized vs general setup. The proposed method improves in terms of SI-SNR, but is a bit worse in terms of keyword spotting error. Some ablation is done with respect to the skip connection in the U-Net.\n\nPaper has two main contributions: \n1) Self-supervised setup to learn from noisy audio with corresponding IMU\n2) Dataset of audio recordings with corresponding IMU", "summary_of_the_review": "Overall an interesting approach to learn to enhancement speech given an IMU signal without access to clean speech audio. The paper has some flaws that could be corrected (such as a stronger supervised baseline, ablations of the correlation losses, more clarity in presentation). If these weaknesses were addressed, I would be inclined to rate the paper higher. The paper releases data and code, which seems useful to the community.", "strength_and_weaknesses": "Strengths\n\nS1) The proposed self-supervised method seems effective at leveraging the additional conditional information provided by the IMU.\n\nS2) The dataset (already released on anonymous github) seems to be the first publicly available dataset with audio and corresponding IMU.\n\nWeaknesses\n\nW1) The paper would benefit from some additional discussion of how the Translator can bootstrap its training by treating the (downsampled) noisy signal H as a target in the first stage. Does this work because the IMU is only correlated with the user's speech? Perhaps the correlation losses are quite important for this. An ablation with respect to the correlation losses would be interesting to see.\n\nW2) It's not clear if (magnitude) spectrograms are used throughout the model. Is SI-SNR computed on waveforms, or spectrograms? If waveforms, how are the predicted spectrograms inverted back to waveforms? If spectrograms, it would be great to invert these back to waveforms and provide audio demos for listening.\n\nW3) I think the supervised baseline could be stronger. It should be possible to directly train a single stage of the Denoiser part of the model pipeline, then the architecture would exactly match with the proposed method. For the existing baseline, please report comparison in terms of number of parameters and computation (e.g. FLOPs)\n\nW4) I object to the use of the term \"attention map\" for the Translator's predicted M. This is just a mask, as it just elementwise multiplies a spectrogram. An attention map would be attending across values (i.e. multiplying and summing) given keys and queries. Please just call this a \"mask\".\n\nW5) I have a number of minor comments on clarity, formatting, etc (see below)\n\nMinor comments\n\nm1) Spell out \"SINR' acronym when first used, I think it's signal to interference and noise ratio.\n\nm2) Maybe use italics instead of underlining in \"Summary of results\". Also, later on the format switched to using \\texttt. Please pick a consistent format. i think italics throughout is fine.\n\nm3) Some inconsistent capitalzation, e.g. \"the Supervised audio Denoiser\" -> \"the supervised audio Ddenoiser\"\n\nm4) I find it a little odd to report percentage numbers\n\nm5) \"from the input data (H + L)\": this should be \"from the input data (H, L)\", the + makes it seem like addition.\n\nm6) \"is the Correlation loss,\" -> \"is the correlation loss,\"\n\nm7) In losses for Denoiser (by the way, would be good to add an equation number for these), the T(x) and D(x) notation strikes me as a little funny, but I guess it works.\n\nm8) \"Expectation-Maximization framework (Moon, 1996).\": I think the most classic reference is Dempster, A.P.; Laird, N.M.; Rubin, D.B. (1977). \"Maximum Likelihood from Incomplete Data via the EM Algorithm\". Journal of the Royal Statistical Society, Series B. 39 (1): 1–38. JSTOR 2984875. MR 0501537.\n\nm9) For the dataset description, please describe how much data in terms of duration was recorded. Also please report duration of the Google speech command dataset data that is used to construct the 990 hours of training data. It seems like the recorded data is not very big, did you observe issues with overfitting of the model? I generally find that having enough source data to create synthetic mixtures is important.\n\nm10) This sentence seems unnecessary: \"We call the target user Alice for ease of explanation.\"\n\nm11) \"distils\" -> \"distills\"\n\nm12) \"we can either choose to obtain 1.5 to 4 dB SI-SNR gain\": not clear which rows and columns this is comparing\n\nm13) Do not italicize dB\n\nm14) I think best practice is to use a single decimal place to report units in terms of decibels, since humans can barely even hear 0.1 dB of difference.\n\nm15) Maybe swap order of Self-supervised IMUV and Supervised IMUV rows in Table 1?\n\nm16) Spell out SIR when first used, I think it's signal to interference ratio\n\nm17) \"the SI-SNR drops if we remove the skip connection (0%).\": what does 0% mean here?\n\nm18) \"and achieves promising Si-SDRi.\" -> \"and achieves promising SI-SNRi\"\n\nm19) \"operates entirely on 400 Hz, the viable sampling rate for real-world earphones. The energy budget for higher sampling rates substantively degrades the device’s battery-life.\": Just because this reduces FLOPs? If the streaming network is small enough and/or the earphone accelerator is fast enough, it's also fine to process audio. But the paper does mention power consumption in the next sentence. Is that the main point of 400 Hz?\n\nm20) \"The IMU signal L is a projection of the audio signal H onto a lower dimensional space.\": but isn't it also nonlinearly transformed?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject