Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: A self-supervised speech enhancement method that uses both audio and IMU data is presented. The method is self-supervised in the sense that the audio does not need to be a clean audio signal. The method is targeted towards earphone recorded audio data.

The self-supervision is handled with an alternating minimization like training algorithm where a "translator" model that maps IMU recordings to spectrogram masks that multiply the input audio spectrogram and another "denoiser" model that takes both IMU and input spectrograms and tries to predict a clean spectrogram without having access to the clean spectrogram. The denoiser model has an internal representation of the input audio and IMU and it tries to pick the part of the representation that correlates with the IMU data to reconstruct the spectrogram. By doing this, it is assumed that the part of the audio that correlates with the IMU is estimated only which refines the estimated clean audio output. This new estimate is used in the "translator" loss function once again. Thus, each module provides the target for the other module in an alternating fashion. Additional "correlation" losses are used in the denoiser to force the model to output signals that correlate with IMU data.
strength_and_weaknesses: Strengths:
1. New speech + IMU dataset.
2. New self-supervised enhancement model based on alternating minimization training.

Weaknesses:
1. Comparison with more direct self-supervised techniques like pseudo-SE or Mixit like training when adding IMU data to the input.
2. Dataset assumes there is always interfering speech which is actually quite rate in real use.
clarity,_quality,_novelty_and_reproducibility: The paper is a bit hard to follow due to changing math notation. More consistent notation would help. For example, the variable x in equations after Equation (2) is not consistent with earlier notation. T and D networks have different inputs, so using x for both is confusing. The training algorithm is confusing to describe, so making the code available would help for reproducibility.
summary_of_the_review: The paper introduces a method for self-supervised training of speech enhancement models.

I think the method described in the paper has a risk of divergence during training due to complexity of the training method. I think there are more straightforward ways to perform self-supervised training for speech enhancement. One of them is to use MixIT method for speech enhancement that is described in the MixIT paper. In that approach, we add noises to existing data (whether the data is clean or not) and then separate it into three outputs which should satisfy the MixIT loss. A simplification of this method is something called pseudo-speech-enhancement in a paper (Siwaraman, Kim and Kim) which can also be tried. In that one, we just add noise to data and assume the data is clean whether it is clean or not.

In the dataset, an interfering speech is added to the clean speech utterances but it is not always realistic (as also mentioned in the paper). The interference could be environmental noise, or silence or another speaker. It would be more interesting to study the case where each alternative is added with some probability. Also, the interference would typically have reverberation which is not used in the dataset.

In the models for comparison, it is not clear how "Supervised IMUV" is trained. Is it trained using alternating minimization, or is it trained in a straightforward way with noisy input and clean output (so no translator is used?) ? This should be clarified. Also, for "supervised denoiser" model, why not use the same architecture without the IMU input?

In Table 1, when using a general model, the Si-SNR gets worse for the "supervised denoiser" which probably means the data is not enough to train a reliable denoiser.

In the dataset, the interference is at a fixed SIR. In Figure 8, the SIR value is changed, but it is the same for train and test data. I think to be more realistic, the SIR values should be in a realistic range for both training and test data.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose to utilize IMU signals for providing self supervision to audio signals for the problem of speech enhancement. This in itself is a good idea, and some that reasonable from the perspective that bone conduction is measurable for low level frequencies. The authors propose a simple algorithm the evaluate this. The evaluations are minimal and the presentation is good.  
strength_and_weaknesses: A meta comment is that, although the idea and proposal makes sense, since the evaluations set consisted of only 4 people, its uncertain how the proposal generalizes to large datasets, with realistic and noisy scenarios (e.g., noise in the collected samples). 

1) Given that IMU signals are only relevant in low-frequency regimes, its important to understand the data more carefully here. Firstly how much variance there is in the IMU data that was collected from the 4 people; and if that is replicated to more people how would that change be?
2) We need to understand the trends as iterations progress i.e., how much quantifiably are the attention maps evolving as iterations increase and how much effect does that have on the overall performance? 
3) Can you comment on the non-monotonic trends of self-supervised and supervised IMU models in Figure 8? Is this an aberration from data? 
4) A natural baseline it to take the average of all the IMU inputs/spectra from the datasets and use that single average spectrum to regularize the enhancer? This is mainly because the IMU signals have smaller range compared to natural speech and so the networks may end up using an average signal from the dataset unless we have large variance in the dataset (goes back to comment 1) and meta comment above). 
5) Speech enhancement community has been quite active in the past decade and there have been more audio-only enhancement algorithms since Park,Lee 2016. Whta was the rationale for not choosing more recent baselines? 
6) Word recognition accuracy may not represent the subtle changes in signals resulting from different enhancement algorithms? What was the rationale for using this instead of say traditional perceptual/quality metrics like pesq, mos etc. 
clarity,_quality,_novelty_and_reproducibility: The technical depth and quality is quite minimal since the idea is mainly empirical. 
The clarity of the paper can be improved. Reproducibility rests on whether these results can be replicated to larger datasets and with more complicated input features from the IMUs. 
summary_of_the_review: As mentioned in the weaknesses, the main aspect is generalizability to reasonable sized datasets. The idea in itself is good/interesting. Additionally many details about the data, the model and its robustness are missing, and will improve the contribution.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims at doing denoising (speech) without supervision. It pursues self-supervised learning while leveraging paired data from another modality (vibration IMU i.e. inertial measurement device). Speech is sampled at high rate while IMU data is sampled at only 400Hz. The proposed approach consists of two models (translator and denoiser) which are trained alternatively for 3 cycles. This optimization is seen akin to EM algorithm. The work is different from prior works in many respects. Previous works either don't explore self-supervised learning or use a more idealistic experimental setup. This work compares results with a regular Supervised Denoiser and find that when clean data is also available, their technique is close to Supervised Denoiser. When clean data is not available, the proposed approach helps a lot in SDR and KWS metrics.
strength_and_weaknesses: Strengths:
1. The experimental setup is vastly improved over the previous works.
2. In the world of earables, the choice of problem is very apt.

Limitations:
1. Perhaps a basic issue, but the intuition for why this should work is missing. It deserves atleast one paragraph.
2. Novelty should discussed further in Section 4 (Related work) under "Self-supervision".
3. Number of users to collect data is low. Age/gender details are missing.
4. Correlation loss under equation 2 has missing equation number. Also, why is weight not different for the two (postive and negative) terms?
5. Results could be more encouraging -- especially, given the fact supervised denoiser is working so well already.
6. Semi-supervised setup/fine-tuning may be explored in future.
clarity,_quality,_novelty_and_reproducibility: Clarity: very clear, except the concerns I listed under "Limitations". Figures are clear.
Quality and originality: The work seems original from many angles. Again consult "Limitations" for concerns.
Reproducibility: Authors have created publishable dataset. If code can be provided, it would be great. This is important since I think hyper-parameter choice is important here.
summary_of_the_review: The choice of problem is very apt given the current state of earable technology. It is interesting to see self-supervised paradigm being explored in an interesting setup. Denoising without clean data is a recently popularized problem which can have great implications in future. Authors develop connection to EM for their two-network alternating learning model. Results are encouraging but not very strong. Hence, this paper is of theoretical importance and is a good step in their choice of problem.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a self-supervised approach for training a speech enhancement network, given data from an inertial measurement unit (IMU) and audio recorded from an earphone microphone. The IMU captures a nonlinear and lower-frequency version of the user's speech through the bone condution channel. The proposed approach iterates between applying a "translator" model that predicts a mask M upsampled from the lower-frequency IMU signal L (uses 3 stages of upsampling, 400 Hz -> 800 Hz -> 3200 Hz -> 16 kHz). This mask M can be applied to the output of a "denoiser" stage, which takes audio mixture as input and conditions on the IMU signal L, and predicts denoised audio \hat{H}_T. Then the denoised audio is fed into another stage of the translator model, and the process is iterated. Experiments use 3 iterations of this process. The translator has a loss between the translator's output (the mask M applied to mixture audio H) and the denoiser output. The denoiser has multiple losses: reconstruction loss between output and translator's output, a loss that promotes part of its intermediate embedding at the middle of the U-Net matches L, and a correlation loss that minimizes the correlation between L and embedding for higher-level features (modeling the gap between audio and IMU), and maximizes correlation with the other component of Z and L. 

The paper presents a new dataset of 4 participants with earphone and IMU, saying multiple keyword multiple times, and interfering speech audio is added. Performance is measured with SI-SNR (not clear if it's spectrogram or waveform domain) and keyword accuracy error. A supervised baseline with different architecture is trained on the same data. The proposed method is trained with noisy H (self-supervised) and clean speech for H (supervised). Method are also compared on personalized vs general setup. The proposed method improves in terms of SI-SNR, but is a bit worse in terms of keyword spotting error. Some ablation is done with respect to the skip connection in the U-Net.

Paper has two main contributions: 
1) Self-supervised setup to learn from noisy audio with corresponding IMU
2) Dataset of audio recordings with corresponding IMU
strength_and_weaknesses: Strengths

S1) The proposed self-supervised method seems effective at leveraging the additional conditional information provided by the IMU.

S2) The dataset (already released on anonymous github) seems to be the first publicly available dataset with audio and corresponding IMU.

Weaknesses

W1) The paper would benefit from some additional discussion of how the Translator can bootstrap its training by treating the (downsampled) noisy signal H as a target in the first stage. Does this work because the IMU is only correlated with the user's speech? Perhaps the correlation losses are quite important for this. An ablation with respect to the correlation losses would be interesting to see.

W2) It's not clear if (magnitude) spectrograms are used throughout the model. Is SI-SNR computed on waveforms, or spectrograms? If waveforms, how are the predicted spectrograms inverted back to waveforms? If spectrograms, it would be great to invert these back to waveforms and provide audio demos for listening.

W3) I think the supervised baseline could be stronger. It should be possible to directly train a single stage of the Denoiser part of the model pipeline, then the architecture would exactly match with the proposed method. For the existing baseline, please report comparison in terms of number of parameters and computation (e.g. FLOPs)

W4) I object to the use of the term "attention map" for the Translator's predicted M. This is just a mask, as it just elementwise multiplies a spectrogram. An attention map would be attending across values (i.e. multiplying and summing) given keys and queries. Please just call this a "mask".

W5) I have a number of minor comments on clarity, formatting, etc (see below)

Minor comments

m1) Spell out "SINR' acronym when first used, I think it's signal to interference and noise ratio.

m2) Maybe use italics instead of underlining in "Summary of results". Also, later on the format switched to using \texttt. Please pick a consistent format. i think italics throughout is fine.

m3) Some inconsistent capitalzation, e.g. "the Supervised audio Denoiser" -> "the supervised audio Ddenoiser"

m4) I find it a little odd to report percentage numbers

m5) "from the input data (H + L)": this should be "from the input data (H, L)", the + makes it seem like addition.

m6) "is the Correlation loss," -> "is the correlation loss,"

m7) In losses for Denoiser (by the way, would be good to add an equation number for these), the T(x) and D(x) notation strikes me as a little funny, but I guess it works.

m8) "Expectation-Maximization framework (Moon, 1996).": I think the most classic reference is Dempster, A.P.; Laird, N.M.; Rubin, D.B. (1977). "Maximum Likelihood from Incomplete Data via the EM Algorithm". Journal of the Royal Statistical Society, Series B. 39 (1): 1–38. JSTOR 2984875. MR 0501537.

m9) For the dataset description, please describe how much data in terms of duration was recorded. Also please report duration of the Google speech command dataset data that is used to construct the 990 hours of training data. It seems like the recorded data is not very big, did you observe issues with overfitting of the model? I generally find that having enough source data to create synthetic mixtures is important.

m10) This sentence seems unnecessary: "We call the target user Alice for ease of explanation."

m11) "distils" -> "distills"

m12) "we can either choose to obtain 1.5 to 4 dB SI-SNR gain": not clear which rows and columns this is comparing

m13) Do not italicize dB

m14) I think best practice is to use a single decimal place to report units in terms of decibels, since humans can barely even hear 0.1 dB of difference.

m15) Maybe swap order of Self-supervised IMUV and Supervised IMUV rows in Table 1?

m16) Spell out SIR when first used, I think it's signal to interference ratio

m17) "the SI-SNR drops if we remove the skip connection (0%).": what does 0% mean here?

m18) "and achieves promising Si-SDRi." -> "and achieves promising SI-SNRi"

m19) "operates entirely on 400 Hz, the viable sampling rate for real-world earphones. The energy budget for higher sampling rates substantively degrades the device’s battery-life.": Just because this reduces FLOPs? If the streaming network is small enough and/or the earphone accelerator is fast enough, it's also fine to process audio. But the paper does mention power consumption in the next sentence. Is that the main point of 400 Hz?

m20) "The IMU signal L is a projection of the audio signal H onto a lower dimensional space.": but isn't it also nonlinearly transformed?
clarity,_quality,_novelty_and_reproducibility: Clarity

The paper is generally clear, but there are some missing details and clarifications that could be made (see weaknesses and minor comments above).

Quality

I think this is good quality paper. The quality could be improved by comparing to a stronger supervised baseline and addressing my comments in weaknesses.

Novelty

Conditioning separation on IMU has certainly been done before, but I think the proposed self-supervised approach is novel. It would be nice to have more discussion about why and how the proposed approach works. An ablation across correlation losses, and other relevant mechanisms, would be welcome.

Reproducibility

The paper releases the recorded data, which is great. And the github seems to have code to replicate training and evaluation. So reproducibility should be straight-forward.
summary_of_the_review: Overall an interesting approach to learn to enhancement speech given an IMU signal without access to clean speech audio. The paper has some flaws that could be corrected (such as a stronger supervised baseline, ablations of the correlation losses, more clarity in presentation). If these weaknesses were addressed, I would be inclined to rate the paper higher. The paper releases data and code, which seems useful to the community.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents a self-supervised speech enhancement method for a scenario when both a microphone recording and a signal from an accelerometer sensor is available. Activity of the latter signal is well correlated with the speech signal activity, while it's also not corrupted by the acoustic noise from the environment, e.g., by interfering speech and noise. This fact is often exploited by in speech enhancement for improving the speech signal estimate.
strength_and_weaknesses: The main strength of the paper is the self-supervised training formulation, since it does not rely on clean speech signal data.
One of the main weaknesses of this paper is that it does not mention a lot of relevant prior work.
More specifically, the authors cite the following relevant work:
- SEANet: A Multi-modal Speech Enhancement Network, 2020
However, this topic has been covered extensively in more classical speech enhancement work, e.g.,
- Speech enhancement for an in‐ear communication system using bone‐conducted and acoustic signals, JASA 2001
- Speech enhancement for an in-ear communication system using optimal-filtered accelerometer signals, Acoustics 2002
- Survey of speech enhancement supported by a bone conduction microphone, ITG Speech communication 2012
- Time-Domain Multi-Modal Bone/Air Conducted Speech Enhancement, 2020
any many references in the above-mentioned.

Furthermore, even the single mentioned multi-modal speech enhancement method (SEANet) is not described correctly. More specifically, the paper claims that "uses an IMU that samples at 4 kHz. In contrast, IMUV is self-supervised [...] and operates entirely on 400 Hz, the viable sampling rate for real-world earphones." This is not exactly true. Figure 4 in the SEANet paper depicts the performance in speech and noise for accelerometer sampling rates as low as 160Hz. Moreover, those results indicate that sample rates as low as 400Hz are not severely impacting the performance of the system.
clarity,_quality,_novelty_and_reproducibility: The paper is relatively easy to read, and the proposed self-supervised training for mic+accel speech enhancement seems to be novel enough. However, there is room for improvements to improve quality of the material and clarity of presentation.

Some remarks:
- Baseline "Supervised Denoiser" seems to be a relatively simple system. It's not clear how it compares to the proposed system in terms of parameters, nor how does it perform to more recent end-to-end speech enhancement work.
- Choice of sampling frequency (4kHz) is pretty odd. This sampling frequency is too low for modern communication devices. Could this design choice be clarified?
- Terminology is sometimes odd. For example, "unclean" dataset should likely be replaced with "noisy" or "perturbed". Also, "translated to an audio signal" is an unusual phrase. The whole "Translator" block could be better described as a converter, or even more generally, a mask estimator.
- "Attention map" is actually just a real-valued gain and it's basically the way speech enhancement has been done for decades
- Input and output is not well defined. It's mentioned that it's an STFT, but it's not clear if it's power, magnitude, what's the STFT setup, and if any other transformation has been applied to it
- Section 3.2: “the testing data also follows the same training data distribution”. This makes Figure 7 relatively useless, since the average SI-SNR of the test data is expected to increase, i.e., denoising is becoming easier, so increasing SI-SNR is expected.
- Reproducibility is definitely helped by the published GitHub repo. However, the paper is not clear enough on the parameter of the network and the training procedure.
- Section 5 is pitching the presented system as a solution to many problems, e.g., audio-visual enhancement and multi-modal synthesis. This seems a bit too general, especially the latter.
summary_of_the_review: The paper is interesting and self-supervised training is of interest for many applications.
However, the paper is not covering the existing work well enough, it's not clear how it performs to state-of-the-art speech enhancement systems, and some aspects of the design are not clear.
If the above would be addressed, I believe the paper could be considered for publication. However, in its current form, it's just not good enough from the perspective of this reviewer.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors present a technique for self-supervised learning using audio and IMU data. Targeting earphones / earables, wherein such information is available, the presented technique has the advantage that one does not need access to ground truth clean data for training. Broadly, the approach works by finding correlations between noisy audio and IMU data, which is not affected by background noise.

The reviewers pointed out that there is novelty in how the authors use the 2 modalities for optimization in an alternating fashion, without the need for clean data. The authors also open source a new dataset for supporting more work in this space, which is great.

Multiple reviewers pointed out shortcomings with evaluations. For example, comparisons with other more direct self-supervised techniques using IMU data as an additional modality would have been useful, along with other more recent approaches. Importantly, the training and test settings are also not diverse enough, making it challenging to assess the overall quality. The reviewers also brought up clarity of presentation as a weak point of the submission. 
justification_for_why_not_higher_score: Although the technique is interesting, evaluations and presentation are both lacking. The novelty is not enough to overcome these limitations.
Decision: Reject