SlowGuess/NeurIPS_2023_OCR huggingface dataset:
MCC is a standard metric for evaluating the recovery of latent factors in ICA literature. MCC first calculates the absolute values of the correlation coefficient between every ground-truth factor against every estimated latent variable. Pearson correlation coefficients or Spearman's rank correlation coefficients can be used depending on whether componentwise invertible nonlinearities exist in the recovered factors. The possible permutation is adjusted by solving a linear sum assignment problem in polynomial time on the computed correlation matrix.
B.8 Network Architecture
We summarize our network architecture below and describe it in detail in Table 2 and Table 3.
Table 2: Architecture details. BS: batch size, T: length of time series, i_dim: input dimension, z_dim: latent dimension, LeakyReLU: Leaky Rectified Linear Unit.
Configuration	Description	Output
ARHMM	Autoregressive HMM for Synthetic Data
Input: x1:T	Observed time series	BS × T × i_dim
Emission Module	Compute μzt+1, σzt+1	BS × T × 2 × z_dim
MLP-Encoder	Encoder for Synthetic Data
Input: x1:T	Observed time series	BS × T × i_dim
Dense	128 neurons, LeakyReLU	BS × T × 128
Dense	128 neurons, LeakyReLU	BS × T × 128
Dense	128 neurons, LeakyReLU	BS × T × 128
Dense	Temporal embeddings	BS × T × z_dim
MLP-Decoder	Decoder for Synthetic Data
Input:ẑ1:T	Sampled latent variables	BS × T × z_dim
Dense	128 neurons, LeakyReLU	BS × T × 128
Dense	128 neurons, LeakyReLU	BS × T × 128
Dense	i_dim neurons, reconstructed x1:T	BS × T × i_dim
Factorized Inference Network	Bidirectional Inference Network
Input	Sequential embeddings	BS × T × z_dim
Bottleneck	Compute mean and variance of posterior	μ1:T, σ1:T
Reparameterization	Sequential sampling	ẑ1:T
Prior Network	Nonlinear Transition Prior Network
Input	Sampled latent variable sequenceẑ1:T	BS × T × z_dim
InverseTransition	Compute estimated residuals εit	BS × T × z_dim
JacobianCompute	Compute log (|det(J)|)	BS
Table 3: Architecture details on CNN encoder and decoder. BS: batch size, T: length of time series, h_dim: hidden dimension, z_dim: latent dimension, F: number of filters, (Leaky)ReLU: (Leaky) Rectified Linear Unit.
Configuration	Description	Output
CNN-Encoder	Feature Extractor
Input: x1:T	RGB video frames	BS × T × 3 × 64 × 64
Conv2D	F: 32, BatchNorm2D, LeakyReLU	BS × T × 32 × 64 × 64
Conv2D	F: 32, BatchNorm2D, LeakyReLU	BS × T × 32 × 32 × 32
Conv2D	F: 32, BatchNorm2D, LeakyReLU	BS × T × 32 × 16 × 16
Conv2D	F: 64, BatchNorm2D, LeakyReLU	BS × T × 64 × 8 × 8
Conv2D	F: 64, BatchNorm2D, LeakyReLU	BS × T × 64 × 4 × 4
Conv2D	F: 128, BatchNorm2D, LeakyReLU	BS × T × 128 × 1 × 1
Dense	F: 2 * z_dim = dimension of hidden embedding	BS × T × 2 * z_dim
CNN-Decoder	Video Reconstruction
Input: z1:T	Sampled latent variable sequence	BS × T × z_dim
Dense	F: 128, LeakyReLU	BS × T × 128 × 1 × 1
ConvTranspose2D	F: 64, BatchNorm2D, LeakyReLU	BS × T × 64 × 4 × 4
ConvTranspose2D	F: 64, BatchNorm2D, LeakyReLU	BS × T × 64 × 8 × 8
ConvTranspose2D	F: 32, BatchNorm2D, LeakyReLU	BS × T × 32 × 16 × 16
ConvTranspose2D	F: 32, BatchNorm2D, LeakyReLU	BS × T × 32 × 32 × 32
ConvTranspose2D	F: 32, BatchNorm2D, LeakyReLU	BS × T × 32 × 64 × 64
ConvTranspose2D	F: 3, estimated scene x1:T	BS × T × 3 × 64 × 64