SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Data generation To generate the latent DAG and sample the latent variables we use the sampler package [21]. We always sample  $\mathrm{ER}(d,k)$  graphs and the non-zero weights are sampled from the distribution  $w_{ij}\sim \mathcal{U}(\pm [0.25,1.0])$ . To generate linear synthetic data, we sample all entries of the mixing matrix i.i.d. from a standard Gaussian distribution. For non-linear synthetic data  $f$  is given by the architecture in Table 9 with weights sampled from  $\mathcal{U}([-\frac{1}{\sqrt{in_{feat}}},\frac{1}{\sqrt{in_{feat}}}])$ . For image data, similar to [2], the latents  $Z$  are the coordinates of balls in an image (but we sample them differently as per our setting) and a  $64\times 64\times 3$  RGB image is rendered using Pygame [77] which encompasses the nonlinearity. Other parameter choices such as dimensions, DAGs, variance parameters, and shifts are already outlined in Section 6 but for clarity we also outline them in Tables 6, 7, and 8.
Models For synthetic data, the model architecture for  $h(x,\theta)$  is a one hidden-layer MLP with 512 hidden units and LeakyReLU activation functions. For the parametric final layer, we decompose the matrix  $W$  as  $W = D(\mathrm{Id} - A)$  where  $D$  is a diagonal matrix and  $A$  is a matrix with a masked diagonal and they are both learned. This factorization shall improve stability. We initialize  $A = 0$  and  $D = \mathrm{Id}$ . Also, the sparsity penalty and the DAGness penalty are only applied to  $A$ . For the baseline VAE we use the same architecture for both encoder and decoder. The code for the linear baseline is from [84].
Table 6: Parameters used for linear synthetic data.
d	5
d'	10
k	3/2
n	{2500,5000,10000,25000,50000}
σ2obs	U([2,4])
σ2int	U([6,8])
η(i)	0
runs	5
mixing	linear
Table 7: Parameters used for nonlinear synthetic data.
d	{5,10}
d'	{20,100}
k	{1,2}
n	10000
σ2obs	U([1,2])
σ2int	U([1,2])
η(i)	U(±[1,2])
runs	5
mixing	3 layer mlp
Table 8: Parameters used for image data.
d	{4,6}
d'	3·642
k	{1,2}
n	25000
σ2obs	U([0.01,0.01])
σ2int	U([0.01,0.02])
η(i)	U(±[0.1,0.2])
runs	5
mixing	image rendering
Table 9: Synthetic data generation
Architecture	Layer Sequence
MLP Embedding	Input: z ∈ Rd
FC 512, LeakyReLU(0.2)
FC 512, LeakyReLU(0.2)
FC 512, LeakyReLU(0.2)
FC d'← Output x
For image data, the model architectures for  $h(x,\theta)$  are small convolutional networks. For the contrastive approach the architecture can be found in Table 10. For the VAE model, we use the encoder architecture as in Table 11 and the decoder architecture as in Table 12 respectively.
Table 10: Contrastive model architecture for image data.
Architecture	Layer Sequence
Conv Embedding	Input: x ∈ R64×64×3
Conv (3, 10, kernel size = 5, stride = 3), ReLU()
MaxPool (kernel size = 2, stride = 2)
FC 64, LeakyReLU()
FC d ← Output z
Training For training we use the hyperparameters outlined in Table 13. We use a 80-20 split for train and validation/test set respectively. After subsampling each dataset to the same size, we copy the observation samples for each interventional dataset in order to have an equal number of observational and interventional samples so they can be naturally paired during the contrastive learning. We select the model with the smallest validation loss on the validation set, where we only use the cross entropy loss for validation. For the VAE baseline we use the standard VAE validation loss for model selection.
Table 11: VAE Encoder architecture for image data
Architecture	Layer Sequence
Conv Encoder	Input: x ∈ R64×64×3
Conv(3, 16, kernel size = 4, stride = 2, padding = 1), LeakyReLU()
Conv(16, 32, kernel size = 4, stride = 2, padding = 1), LeakyReLU()
Conv(32, 32, kernel size = 4, stride = 2, padding = 1), LeakyReLU()
Conv(32, 64, kernel size = 4, stride = 2, padding = 1), LeakyReLU()
FC 2d← Output (mean, logvar)
Table 12: VAE Decoder architecture for image data
Architecture	Layer Sequence
Deconv Decoder	Input:hat∈Rd
FC (3×3)×d
DeConv (d, 32, kernel size = 4, stride = 2, padding = 0), LeakyReLU()
DeConv (32, 16, kernel size = 4, stride = 2, padding = 1), LeakyReLU()
DeConv (16, 8, kernel size = 4, stride = 2, padding = 1), LeakyReLU()
DeConv (8, 3, kernel size = 4, stride = 2, padding = 1), LeakyReLU()
Sigmoid()←Output hat
Table 13: Hyperparameters used for training.
τ1(see Eq. (9))	10-5
τ2(see Eq. (9))	10-4
learning rate	5·10-4
batch size	512
epochs	200 (synthetic data), 100 (image data)
optimizer	Adam [38]
learning rate scheduler	CosineAnnealing [51]
Compute and runtime The experiments using synthetic data were run on 2 CPUs with 16Gb RAM on a compute cluster. For image data we added a GPU to increase the speed. The runtime per epoch for the synthetic data and 10000 samples was roughly 10s. The total run-time of the reported experiments was about 100h (i.e. 200 CPU hours) and 20h on a GPU, but hyperparameter selection and preliminary experiments required about 10 times more CPU compute.
Mean Correlation Coefficient (MCC) Mean Correlation Coefficient (MCC) is a metric that has been utilized in prior works [37, 41] to quantify identifiability. MCC measures linear correlations up to permutation of the components. To compute the best permutation, a linear sum assignment problem is solved and finally, the correlation coefficients are computed and averaged over. A high MCC indicates that the true latents have been recovered. In this work, we compute the transformation using half of the samples and then report the MCC using the other half, i.e. the out-of-distribution samples.
Further experimental attempts and dead Ends We will briefly summarize a few additional settings we tried in our experiments.
- We fixed  $D = \operatorname{Id}$  in the parametrization of  $W$ . This fixes the scaling of the latent variables in some non-trivial way. This resulted in very similar results.
- We tried to combine the contrastive algorithm with a VAE approach. Since the ground truth latent variables  $Z$  are highly correlated, they are not suitable for an autoencoder that typically assumes a factorizing prior. Therefore, we map the estimated latent variables for the observational distribution to the noise distribution  $\epsilon$  which factorizes. Note that for the ground truth latent variables the relation  $\epsilon = B^{(0)}Z^{(0)}$  holds so we consider  $\hat{\epsilon} = W\hat{Z}$ . We
use  $\hat{\epsilon}$  as the latent space of an autoencoder on which we then stack a decoder. We train using the observational samples and the ELBO for the VAE part and with the contrastive loss (and interventional and observational samples) for the  $\hat{Z}$  embedding. This resulted in a slightly worse recovery of the latent variables (as measured by the MCC score) but much worse recovery of the graph. We suspect that this originates from the fact that the matrix  $W$  is used both in the parametrization of the log-odds and to map  $\hat{Z}$  to  $\epsilon$  which might make the learning more error-prone. Note that this is different from the suggestion in Section F.4.
- Choosing a larger learning rate for the parametric part than for the non-parametric part seemed to help initially, but using the same sufficiently small learning rate for the entire model was more robust.
- We tried larger architectures for both synthetic and image data. They turned out to be more difficult to train without offering any benefit.
- For image data we often observed overfitting when using smaller sample sizes. The overfitting persisted even when we used a pre-trained ResNet [24] with frozen layers. We suspect that for image data the performance (in particular the sample complexity) can be further improved by carefully tuning regularization and model size.