SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathcal {L} = \sum_ {n} \left(E ^ {\text {p h i s}} \left(\boldsymbol {R} ^ {n}, \boldsymbol {Z} ^ {n}\right) - E ^ {\text {r e f}, n}\right) ^ {2} + (18) \\ + \sum_ {n I \zeta} \left(\frac {\partial}{\partial R _ {I \zeta} ^ {n}} E ^ {\text {p h i s}} \left(\boldsymbol {R} ^ {n}, \boldsymbol {Z} ^ {n}\right) - G _ {I \zeta} ^ {\text {r e f}, n}\right) ^ {2} + (19) \\ + \sum_ {n I J \mu \nu} \left(F _ {I J \mu \nu} ^ {\mathrm {p h i s}} \left(\boldsymbol {R} ^ {n}, \boldsymbol {Z} ^ {n}\right) - F _ {I J \mu \nu} ^ {\mathrm {r e f}, n}\right) ^ {2} + (20) \\ + \sum_ {n I J \mu \nu} \left(S _ {I J \mu \nu} ^ {\mathrm {p h i s}} \left(\boldsymbol {R} ^ {n}, \boldsymbol {Z} ^ {n}\right) - S _ {I J \mu \nu} ^ {\mathrm {r e f}, n}\right) ^ {2}. (21) \\ \end{array}
$$
Here  $E$  denotes energies,  $G$  denotes gradients of energies,  $F$  Fock matrices, and  $S$  overlap matrices. The indices  $I,J$  run over nuclei, the indices  $\mu ,\nu$  over basis functions, and the index  $n$  over samples in a batch.
D Molecule datasets
Bicyclobutane For the Bicyclobutane to 1,3-butadiene transition we use the geometries from Kinal et al. [31] and compare against the reference energies stated in Spencer et al. [5].
$\mathbf{N}_2$  For the  $\mathrm{N}_2$  potential energy surface with various bond-lengths we used the geometries including reference calculations from [8].
Propadiene The global rotation of  $360^{\circ}$  degrees for propadiene is performed on the geometry which is part of the test set for 3 heavy atoms. For the torsion experiment we used the equilibrium geometry and rotated the torsion angle by  $90^{\circ}$  degrees in steps of  $10^{\circ}$  degrees.
Zero-shot and fine-tuning dataset The results on zero-shot and few-shot predictions for increasing number of heavy atoms are performed on random subsets of molecules. For 5-7 heavy atoms we sample 4 unique and distorted molecules from QM7-X [26]. For 4 heavy atoms we use all geometries from the Bicyclobutane dataset. For 3 heavy atoms we use the ablation dataset.
Ablation dataset For the ablation study, we use one geometry per molecule from the out-of-distribution test set from Scherbela et al. [25], leading to a set of four distinct molecules. We ensure that these molecules are not part of the training set.
Large scale experiment For the large scale experiment we used a stratified random sample of 250 molecules from QM7 [32]. It contains all molecules with up to 4 heavy atoms, and additionally 65 randomly chosen molecules for 5, 6 and 7 heavy atoms each.
Pre-training dataset for transferable neural wavefunctions To train our pre-trained wavefunctions we use two datasets, consisting of 18 and 98 disparate molecules. For part of the ablation we use the dataset proposed in [25] and an extended version with 80 additional molecules. The additional compounds are a combination of all valid SMILES generated with RDKit [38] with 3 heavy atoms, allowing only Nitrogen, Oxygen and Carbon with single-, double- or triple-bonds, and all molecules up to four heavy atoms from QM7-X [26] (excluding molecule containing Fluorine). To prevent a train-test leakage, we remove Bicyclobutane (including all conformations) and the four molecules from the ablations dataset. Since the normal-mode-distortions by design do not generate strongly distorted geometries, we augment the 98-molecule-dataset with rotated dihedral angles. To generate a subset of all possible dihedral angles for a heavy-atom bond we first generate samples with equidistant angles for all possible dihedral angles and compute Hartree-Fock energies with a minimal basis-set. We include the equilibrium geometry and all extrema of the potential energy surface with respect to the rotation of a single dihedral angle if the energy of the extrema is significantly different to already included geometries of the same molecule. Additionally, we include the transition geometry towards the respective extrema and again only include energetic diverse states. Finally, to make sure that certain molecules are not underrepresented in the dataset we make sure that all molecules have at least 5 geometries that get distorted during pre-training by adding copies of the equilibrium geometry. Overall this yields 699 initial geometries  $R^0$  for pre-training.
E Energy variance
Fig. 7 depicts the energy variance obtained by various models. Analogously to the energies presented in Fig. 2 of the main text, we also find that our pre-trained model achieves substantially lower energy variance compared to previous generalizing wavefunctions. Notably on the test set with 3 heavy atoms, the variance of our zero-shot model is on par with the variance of a PsiFormer model trained for 16k steps.
F Relative energies
To better depict the accuracy of the models' relative energies for the test systems in Fig. 3 of the main text, we depict the difference of relative energies between our model and a references method in Fig. 8. Furthermore we list numerical values vor all energies of Fig. 3a in Tab. 1. We find that on these
Figure 7: Variance of energies: Variance when re-using the pre-trained model on molecules of varying size without optimization (a) and after fine-tuning (b). For the test set containing 3 heavy atoms as a function of optimization steps, comparing against current state-of-the-art methods. Solid lines correspond to pre-trained models, dashed lines to models without pre-training (c).
challenging systems our model yields relative energy errors of up to  $15\mathrm{mHa}$  when only fine-tuned for 400 steps per geometry. When fine-tuning our pre-trained model for 3200 steps per geometry, we obtain relative energies that are accurate within  $2 - 3\mathrm{mHa}$ .
Figure 8: Challenging relative energies: The difference between relative energies of a reference method vs our work for two challenging systems. a) Nitrogen dimer: Difference of relative energies to the equilibrium geometry for our approach to reference calculation from Gerard et al. for  $\mathrm{N}_2$ , a) Propadiene: Difference of relative energies of the transition barrier of twisted vs. untwisted propadiene. As reference method we choose PsiFormer with 64k optimization steps. This figure complements Fig. 3b,d of the main text with results for additional fine-tuning steps.
G Reference energies
CCSD(T) All CCSD(T) energies – except explicitly stated otherwise – were obtained using ORCA [39] starting from a restricted Hartree-Fock calculation. We use correlation consistent basis sets of the cc-pCVXZ family, with X in  $\{2, 3, 4\}$ . To extrapolate to the complete basis set limit (CBS), we use
structure	CCSD(T) [31]	DMC [31]	FermiNet 200k [5]	FermiNet 10k [5]	Our work zero-shot	Our work 700 per geom
con_TS	64.4	64.4	64.1	63.9	94.0	66.6
dis_TS	34.7	93.4	92.0	87.1	183.8	94.5
g-but	-40.0	-40.2	-40.3	-44.9	-48.5	-40.4
gt-TS	-35.5	-35.4	-35.9	-42.9	-46.9	-36.7
t-but	-44.6	-44.5	-45.3	-47.5	-51.8	-43.2
the approach outlined in [3] and fit the following functions with free parameters  $E_{\mathrm{CBS}}^{\mathrm{HF}}, E_{\mathrm{CBS}}^{\mathrm{corr}}, a, b, c$ :
$$
E _ {X} ^ {\mathrm {H F}} = E _ {\mathrm {C B S}} ^ {\mathrm {H F}} + a e ^ {- b X}
$$
$$
E _ {X} ^ {\text {c o r r}} := E _ {X} ^ {\mathrm {H F}} - E _ {X} ^ {\text {C C S D (T)}} = E _ {\text {C B S}} ^ {\text {c o r r}} + c X ^ {- 3}
$$
$$
E _ {\mathrm {C B S}} ^ {\mathrm {C C S D (T)}} = E _ {\mathrm {C B S}} ^ {\mathrm {H F}} + E _ {\mathrm {C B S}} ^ {\mathrm {c o r r}}
$$
We stress that although CCSD(T)-energies are often considered as "gold-standard", they do not necessarily represent the actual ground-state energy. There are many cases, where CCSD(T) either overestimates the true ground-state energy, or even underestimates it, because CCSD(T) does not yield upper bounds to the true ground-state energy.
PsiFormer For Fig. 2 we used the open-source FermiNet codebase [40]. The codebase didn't allow for inference calculation, therefore a slight fix was applied. All calculations were performed with the small settings as proposed in von Glehn et al. [6].
H Hyperparameters
A detailed description of the hyperparameter used in this work can be found below (cf. Tab. 2). For the mapping of the orbital descriptors to the electron embeddings to build the orbitals we rely on the hyperparameter from [25]. For optimization we rely on the second-order method KFAC [30] and use their Python implementation [41]. During the continuous sampling of the geometries we allow each geometry to perform a maximum of 20 steps of normal-mode distortion from the initial geometry and reset to the original one once the threshold is reached.
I Runtime and computational resources
Tab. 3 lists the run-times for our method and a comparable conventional reference method (CCSD(T)-4Z) for a typical molecule with 5 heavy atoms. We find that our fine-tuning our model for  $4\mathrm{k}$  steps yields lower absolute energies than CCSD(T)-4Z (cf. Fig. 2) at similar computational cost. This is in contrast to many earlier works that achieve highly accurate energies, but at computational cost that far surpasses the cost of conventional methods for small molecules.
Overall we used  $\approx 5\mathrm{k}$  GPUs (A100) for development and training of our base models, and another  $5\mathrm{k}$  GPUs (A40) on evaluations and fine-tuning. Additionally we required  $\approx 20\mathrm{k}$  CPUs for CCSD(T) reference calculations.
Table 1: Energies relative to the energy of bicyclobutane in mHa, including the zero-point vibrational energy correction from Kinal et al. [31]. This data complements Fig. 3a in the main text.
Method	Runtime / h
Our work, zero-shot	1.0
Our work, 4k fine-tuning	9.5
PsiFormer, 16k steps	53.0
CCSD(T), 4Z basis	10.5
Table 3: Runtime of various quantum chemistry methods for  $\mathrm{C_4OH_6}$ . All timings are in node hours, being 2 A100-hours for our work and PsiFormer, and 256 CPU-hours for CCSD(T).
J Code and data availability
All code, configuration files, geometries, datasets and obtained energies are available on GitHub under https://github.com/mdsunivie/deeperwin. Model weights are available on figshare under https://doi.org/10.6084/m9.figshare.23585358.v1.
Electron Embedding	Hidden dimension Nemb	256
N0iterations	4
Nuclear Embedding	Hidden dimension xnuc	64
N0layer MLP	1
Message passing	Activation function	SiLU
N0layer edge embedding	3
Dimension edge embedding	64
Dimension linear layer	32
Markov Chain Monte Carlo	N0walkers	2048
N0 decorrelation steps	50
Target acceptance prob.	50%
PhisNet [20]	Pre-trained against basis set	STO-6G
N0iterations	2
Harmonic degree L	2
N0radial basis functions	128
Hidden dimension of xnuc	128
Distance cutoff (bohr)	30
Transferable atomic orbitals [25]	N0determinants Ndet	8
N0hidden layers forb	2
Hidden dimension of forb	256
N0hidden layers gorb	2
Hidden dimension gorb	128
N0iterations MPNN	2
N0radial basis functions	16
Hidden edge embedding dimension	32
Hidden node embedding dimension	16
Activation function	SiLU
Variational pre-training	Optimizer	KFAC
Batch size	2048
Norm constraint	3 × 10-3
Initial damping d0	1
Minimal damping dmin	0.001
Damping rate decay	d(t) = d0exp(-t/20000)
Initial learning rate lro	0.1
Learning rate decay	lro(t) = lro(1 + t/6000)-1
Optimization steps	128,000 - 256,000
Changes for fine-tuning	Learning rate decay	lro(t) = lro(7 + t/6000)-1
Optimization steps	0 - 32,000
Sampling geometries	Distortion energy β	0.005 Ha
Max age	20
Bias towards original geometry α	0.2
Table 2: Hyperparameter settings used in this work