SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 4 displays our experiments with different self-supervised training objectives. We report performance on the LibriSpeech evaluation setting. The first three rows correspond to experiments in which we do not utilize the shortcut training strategy (referenced in Sec. 3 — "Training" of main paper). Using SRMR alone (row 1) produces the largest (worst) RTE. Training with the acoustic residue metric instead (row 2) leads to a large improvement in RTE, providing empirical support for our metric as an effective training objective. Using our combined metric and the shortcut training strategy (both described in Sec. 3 — "Training" of our main paper) further improves the performance by a small margin.
7.4 Evaluation of de-biaser performance
Table 5 evaluates the quality of de-biased audio compared to both reverberant and dereverberated audio. De-biased audio is less reverberant than dereverberated audio as measured by both absolute RT60 metric and RTE (computed against the ground truth anechoic waveform for SoundSpaces-Speech). De-biased audio also achieves a higher SRMR score than dereverberated audio, indicating superior quality and intelligibility.
7.5 Model/training details
RT60 estimator We adopt the RT60 estimator from [3]. The estimator takes a spectrogram as input, encodes it with a ResNet18 [18], and outputs a scalar RT60 estimate. The model is trained on 2.56s clips of reverberant speech simulated on the SoundSpaces platform [4] paired with the ground truth RT60 computed from the RIR used to generate the reverberant speech. The model trains using MSE loss between predicted and ground truth RT60 values. Ground truth RT60 is computed using the Schroeder method [39].
De-biaser architecture The de-biaser  $G$  takes a magnitude spectrogram as input. This is passed to a bi-directional LSTM with input size 257 and two hidden layers each of size 200, which produces an output with the same temporal length as the input spectrogram. This is passed through a linear layer of size 300 and a leakyReLU activation, followed by another linear layer of size 257 and a Sigmoid activation. The final mask is multiplied with the input magnitude spectrogram to create the generated
Table 5: Evaluation of de-biased audio vs dereverberated audio on various metrics.
Audio	SoundSpaces-Speech	AVSpeech-Rooms
RT60	RTE	SRMR (↑)	RT60	SRMR (↑)
Reverberant	0.40	0.36	5.97	0.42	6.76
Dereverberated	0.06	0.02	8.35	0.06	8.99
De-biased	0.04	0.01	9.50	0.03	13.14
magnitude spectrogram. A resynthesis module computes phase information from the input audio waveform, combines this with the generated magnitude spectrogram, and performs an inverse STFT to produce the generated waveform. The discriminator  $D$  consists of 4 2D Convolutional layers with kernel size (5,5) and 15 output channels, followed by a channel averaging operation and two linear layers of sizes 50 and 10. A LeakyReLU activation with negative slope  $= 0.3$  is used after each intermediate layer. The final layer outputs a scalar-valued metric score estimate.
De-biaser training In stage (1) (see Sec. 3 — "Training" of our main paper), we train with batch size 32. During stage (3) fine-tuning, we use a batch size of 2.  $G$  and  $D$  are trained with learning rates of 2e-6 and 5e-4 respectively in both stages. In each epoch, We train on 10k samples randomly selected from the train set without replacement. The reverberator models  $R_v$  and  $R_b$  are updated with the target networks at a frequency of  $E = 8$  epochs. For all models, we clip each audio sample to 2.56s during training and evaluation.
Reverberator training We train the reverberators with batch size 4 and a learning rate of 1e-2 in stage (2). During stage (3) fine=tuning, we use batch size 2 and a learning rate of 1e-6. Both reverberator models and the ViGAS baseline are trained with MSE loss between the log magnitude spectrogram of predicted and ground truth audio.
Baseline training details We use a learning rate of 1e-2 and a batch size of 4 to train ViGAS. We train AViTAR with batch size 4 — all other hyperparameters are set as described in [3].
Compute All models are trained on 8 NVIDIA Quadro RTX 6000 GPUs.
7.6 Augmentation strategy
We follow a data augmentation strategy similar to that proposed in [3] for training the baseline models, which was shown to produce better generalization performance on the LibriSpeech setting than when trained without this augmentation strategy. In particular, to each batch of dereverberated audio we add colored noise, perform a polarity inversion on the waveform with  $p = 0.5$ , and convolve the waveform with a randomly selected Room Impulse Response (RIR) from a different acoustic environment with  $p = 0.9$ . At test time, we evaluate without these audio augmentations. This strategy is designed to mask over residual acoustic information in dereverberated audio during training. We do not use this augmentation strategy in our approach as our model directly learns to remove residual acoustic information, obviating the need for a heuristic strategy to mask it out.
7.7 Limitations
Our approach focuses on visual acoustic matching on mono-channel audio exclusively. However, binaural cues in audio play a fundamental role in our perception of reverberation and room acoustics [8]. We leave it to future work to extend our approach to binaural audio.
7.8 Broader impact
While training on in-the-wild web videos allows wider access to a diverse variety of speakers and environments, it also introduces uncontrolled biases, speaker privacy concerns, and potentially harmful content into the model.
7.9 Data examples
Refer to video to view samples from both SoundSpaces-Speech and AVSpeech-Rooms.
7.10 LeMARA training procedure
Dataset
Train Step #	SoundSpaces-Speech	AVSpeech-Rooms
1	Pre-train G with SRMR	-
2	Pre-train Rv,Rb	Pre-train Rv,Rb
3	Train G,Rv,Rb	Train G,Rv,Rb
Figure 8: LeMARA three-step training. Refer to Section 3.4 for a detailed description of each step.
7.11 Discriminator epoch - Reverberator update procedure
Algorithm 1 Discriminator training epoch on generated data.
$R_{v}^{t}\gets R_{v}$  Initialize target networks with current reverberator network weights.  $R_{b}^{t}\leftarrow R_{b}$    
n  $\leftarrow$  current epoch #   
for  $\{(G(A_i),V_i)\}$  in generated data batches do  $\{s_i\} \leftarrow \mathcal{M}(\{(G(A_i),V_i))\}$  Compute metric scores (using  $R_{v}$  and  $R_{b}$  -  $D\gets \nabla \mathcal{L}_D\big(D(\{G(A_i)\}),\{s_i\})$  Update discriminator.  $R_{v}^{t}\gets \nabla \mathcal{L}_{visual}\big(R_{v}^{t}(\{(G(A_{i}),V_{i})\}),A_{i}\big)$  Update target networks during disc. training.  $R_{b}^{t}\gets \nabla \mathcal{L}_{\text{blind}}\big(R_{b}^{t}(\{(G(A_{i}),V_{i})\}),A_{i}\big)$    
end for   
if  $n\% E$  is 0 then every E epochs, copy target network weights into current reverberators.  $R_{v}\gets R_{v}^{t}$ $R_{b}\gets R_{b}^{t}$    
end if
Direct-to-Reverberant Energy Ratio (DRR) vs. Speaker Distance
Figure 9: DRR vs. Speaker distance. We plot the Direct-to-Reverberant Energy Ratio (DRR) as a function of the distance between the source and receiver for SoundSpaces-Speech audio samples. As the distance between speaker and receiver increases, the listener receives more reverberant sound than direct sound from the speaker, leading to the observed decrease in DRR.
STFT Error vs. Speaker Distance (SoundSpaces-Speech)
Figure 10: STFT Error vs. Speaker distance. We plot STFT Error as a function of the source-receiver distance for SoundSpaces-Speech samples. LeMARA produces the lowest STFT Error at all source-receiver distances.
Figure 11: De-biaser performance at various reverberation levels. We stratify samples from both datasets by their RT60 and compute the mean SRMR scores of dereverberated and de-biased data within each bin. De-biased audio consistently produces higher SRMR scores than dereverberated audio across all reverberation levels and datasets.