SlowGuess/NeurIPS_2023_OCR huggingface dataset:
B.5 MVTec-AD Experiment
We use MPDR-R with a single manifold (i.e., without the manifold ensemble). Both the manifold and the energy are a fully connected autoencoder of 256D spherical latent space. For input-space Langevin Monte Carlo, the number of MCMC steps, the step size, and the noise standard deviation are 10, 0.1, and 0.1, respectively. No latent chain is used. The manifold is trained for 200 epochs with Adam of learning rate 1e-3, and the energy is trained for 20 epochs with Adam of learning rate 1e-4.
B.6 Acoustic Anomaly Detection
B.6.1 Dataset
The dataset consists of audio recordings with a duration of approximately 10 seconds, obtained through a single channel and downsampled to  $16\mathrm{kHz}$ . Each recording includes both the operational sounds of the target machine and background noise from the surrounding environment. The addition of noise was intended to replicate the conditions of real-world inspections, which often occur in noisy factory environments. The dataset covers six types of machinery, including four sampled from the MIMII Dataset (i.e., valve, pump, fan, and slide rail) and two from the ToyADMOS dataset (i.e., toy-car and toy-conveyor).
B.6.2 Preprocessing
We follow the standard preprocessing scheme used in the challenge baseline and many of challenge participants. The approach involves the use of Short Time Fourier Transform (STFT) to transform each audio clip into a spectrogram, which is then converted to Mel-scale. We set the number of Mel bands as 128, the STFT window length as 1024, and the hop length (i.e., the number of audio samples between adjacent STFT columns) as 512. This configuration results in a spectrogram with 128 columns that represented the number of Mel bands. To construct the final spectrogram, the mel spectra of five consecutive frames are collected and combined to form a single row, resulting in a spectrogram with 640 columns. Each row of the spectrogram is sampled and used as input to the models under investigation, with a batch size of 512, meaning that 512 rows were randomly selected from the spectrogram at each iteration. We standardize all data along the feature dimension to zero mean and unit variance.
B.6.3 Performance Measure
We refer to the scoring method introduced in [41] and use the area under the receiver operating characteristic curve (AUROC) and partial-AUROC ( $p$  AUROC) as a quantitative measure of performance. pAUROC measures the AUC over the area corresponding to the false positive rate from 0 to a reasonably small value  $p$ , which we set in all our experiments as 0.1. Each measure is defined as follows:
$$
\mathrm {A U R O C} = \frac {1}{N _ {-} N _ {+}} \sum_ {i = 1} ^ {N _ {-}} \sum_ {i = j} ^ {N _ {+}} \mathcal {H} \left(\mathcal {A} _ {\theta} \left(x _ {j} ^ {+}\right) - \mathcal {A} _ {\theta} \left(x _ {i} ^ {-}\right)\right) \tag {11}
$$
$$
p \mathrm {A U R O C} = \frac {1}{\lfloor p N _ {-} \rfloor N _ {+}} \sum_ {i = 1} ^ {\lfloor p N _ {-} \rfloor} \sum_ {i = j} ^ {N _ {+}} \mathcal {H} \left(\mathcal {A} _ {\theta} \left(x _ {j} ^ {+}\right) - \mathcal {A} _ {\theta} \left(x _ {i} ^ {-}\right)\right) \tag {12}
$$
where  $\{x_{i}^{-}\}_{i = 1}^{N_{-}}$  and  $\{x_j^+\}_{j = 1}^{N_+}$  are normal and anomalous samples, respectively,  $\lfloor \cdot \rfloor$  indicates the flooring function, and  $\mathcal{H}(\cdot)$  represents a hard-threshold function whose output is 1 for a positive argument and 0 otherwise.
B.6.4 Implementation of Models
We utilize a standard autoencoder model provided by DCASE 2020 Challenge organizers and compare it to our approach. The model comprises a symmetrical arrangement of fully-connected layers in the encoder and decoder; 5 fully connected hidden layers in the input and 5 in the output, with 128-dimensional hidden layers and 32-dimensional latent space. In addition, we incorporate the IDNN model, which predicts the excluded frame using all frames except the central one instead of reconstructing the entire sequence. The IDNN model outperforms the autoencoder on non-stationary sound signals, i.e., sounds with short durations. The model consists of a encoder and decoder, which is similar to the components of an autoencoder but has an asymmetric layout; 3 fully-connected hidden layers that contract in dimension (64, 32, 16) are used in the encoder and 3 layers that expand in dimension (16, 32, 64) compose the decoder. The architectural design of each model follows the specifications outlined in their respective papers.
The MPDR models used for the experiment, MPDR-R and MPDR-IDNN, consists of an ensemble of autoencoders and an energy function, where the terms "R" and "IDNN" specify whether an autoencoder of IDNN is used to compute the energy, respectively. Each autoencoder consists of an encoder, spherical embedding layer, decoder; the encoder and decoder consist of 3 fully-connected layers each, with 1024-dimensional hidden space and a latent space with a dimension chosen among 32, 64, or 128. The autoencoder energy function used in the experiment is built using layers 5 fully-connected encoder layers and 5 decoder layers, whose hidden layers have 128 nodes and latent layer is composed of 32 nodes. The layers of IDNN are identical to that of the autoencoder version, except for the input and output layers whose dimensions differ and add up to the total number of sampled frames. LMC hyperparameters used are listed in Table 7.
C Empirical Guidelines for Implementing MPDR
Here, we present empirical tips and observations we found useful in achieving competitive OOD detection performance with MPDR. The list also includes heuristics that did not work when we tried.
- The training progress can be monitored by measuring an OOD detection metric (i.e., AUROC) computed between test (or validation) inliers and synthetic samples uniformly sampled over the autoencoder manifold  $\mathcal{M}$ . During a successful and stable run, this score tends to increase smoothly.
- Metropolis-style adjustment for LMC [64] did not improve OOD performance.
- The choice of activation function in the energy function affects the OOD detection performance significantly. We found that ReLU and LeakyReLU provide good results in general.
- In image datasets, stopping the training of the autoencoder manifold before convergence improves OOD detection performance.
- A longer Markov chains, both visible and latent, do not always lead to better OOD detection performance.
- A larger autoencoder does not always lead to better OOD detection performance.
- A larger energy function does not always lead to better OOD detection performance.