SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
L _ {\mathcal {F}} (\boldsymbol {x}) _ {v} := \sum_ {v, u \triangleleft e} \mathcal {F} _ {v \triangleleft e} ^ {\top} \left(\mathcal {F} _ {v \triangleleft e} \boldsymbol {x} _ {v} - \mathcal {F} _ {u \triangleleft e} \boldsymbol {x} _ {u}\right).
$$
In our case, the sheaf Laplacian can be expressed as:
$$
L _ {\mathcal {F}} (\boldsymbol {x}) _ {v} = \sum_ {v, u \leq e} \boldsymbol {x} _ {v} - F _ {v} ^ {T} F _ {u} \boldsymbol {x} _ {u}, \tag {24}
$$
using the properties of orthogonal transformations. Thus, this operator quantifies the collective "disagreement of local geometries" at each node. Neural sheaf diffusion propagates information through nodes using the following partial differential equation (PDE):
$$
\boldsymbol {X} (0) = \boldsymbol {X}, \quad \dot {\boldsymbol {X}} (t) = - L _ {\mathcal {F}} \boldsymbol {X} (t). \tag {25}
$$
Here,  $X$  represents the node feature matrix. In accordance with the nonlinear parametrization proposed in [62], we have the following model:
$$
\boldsymbol {X} (t + 1) = \sigma \left(\left(\boldsymbol {I} _ {n d} - L _ {\mathcal {F}}\right) \left(\boldsymbol {I} _ {n} \otimes \boldsymbol {W} _ {1}\right) \boldsymbol {X} (t) \boldsymbol {W} _ {2}\right). \tag {26}
$$
where any applicable nonlinear activation function  $\sigma$  can be utilized since our cellular sheaf and its Laplacian are invariant. It is important to note that the key component of  $L_{\mathcal{F}}$  (denoted by  $F_v^T F_u$  in Wq. 24) corresponds to the frame transition (FTE) module between node frames. Hence, Equation (26) provides an invariant realization of the FTE module based on equivariant node frames. For equivariant outputs, tensorization can be applied after the final layer of Equation (26). To incorporate the LSE module, we can include the LSE information as a part of invariant node features to  $\mathcal{F}_v$  prior to the message aggregation step in Equation (26).
I Additional experiments
Experiment Detail for Table 1 and 2. For QM9, baseline results are taken from Liao and Smidt [36]. For MD17, baseline results are taken from the original papers (with unit conversions if needed). All models are trained on energies and forces, and WoFE is the weight of force over energy in loss functions. All experiments are conducted on a single NVIDIA GeForce RTX 2080 Ti 11GB GPU. Our implementation is based on the libraries including PyTorch [63], PyG [64], and DIG [65].
Ablation Study. As discussed in Section 5, there are two main modules in LEFTNet, namely LSE and FTE. We conduct experiments on QM9 and MD17 to show the importance of each component. Experimental results are summarized in Table 5 and Table 6. The results show that using LSE can outperform the model without both LSE and FTE on all tasks. Adding FTE can further improve the performance. The results demonstrate the importance of LSE and FTE modules.
Table 5: Ablation study on QM9 dataset. The evaluation metric is MAE for each property. The best performances are bolded and the second best are underlined. Detailed LEFTNet (LSE only) is shown in Algorithm 2. LEFTNet (LSE + vector FTE) is our LEFTNet introduced in the main paper, and the detailed algorithm is in Algorithm 1. LEFTNet (LSE + tensor FTE) means the message passing and updating contains higher order tensors, built by Eq. 22.
Task Units	α bohr-3	Δε meV	εHOMO meV	εLUMO meV	μ D	Cν cal/mol K	G meV	H meV	R2 bohr-3	U meV	U0 meV	ZPVE meV
LEFTNet (w/o LSE and FTE)	.053	49	33	25	.038	.026	9	8	.425	8	8	1.59
LEFTNet (LSE only)	.043	49	31	23	.031	.025	8	7	.156	8	7	1.34
LEFTNet (LSE + vector FTE)	.039	39	23	18	.011	.022	6	5	.094	5	5	1.19
LEFTNet (LSE + tensor FTE)	.038	38	22	17	.011	.022	7	6	.096	5	6	1.20
Results on rMD17. Following Batatia et al. [24], we conduct experiments on rMD17 to compare with recent studies. Results show that our LEFTNet can achieve comparable performance to state-of-the-art methods such as MACE and NequIP, while outperforming other baseline methods like GemNet and PaiNN.
Model and training hyperparameters. Model and training hyperparameters for our method on different datasets are listed in Table 8.
Table 6: Abalation Study on MD17 dataset. The evaluation metric is MAE for per-atom forces prediction (kcal/mol Å). The best performances are bolded and the second best are underlined. Detailed LEFTNet (LSE only) is shown in Algorithm 2. LEFTNet (LSE + vector FTE) is our LEFTNet introduced in the main paper, and detailed algorithm is in Algorithm 1. LEFTNet (LSE + tensor FTE) means the message passing and updating contains higher order tensors, built by Eq. 22.
Molecule	LEFTNet (w/o LSE and FTE)	LEFTNet (LSE only)	LEFTNet (LSE + vector FTE)	LEFTNet (LSE + tensor FTE)
Aspirin	1.083	0.451	0.300	0.210
Benzene	0.425	0.185	0.145	0.176
Ethanol	0.341	0.149	0.138	0.118
Malonaldehyde	0.594	0.276	0.209	0.159
Naphthalene	0.658	0.175	0.073	0.063
Salicylic acid	0.828	0.313	0.167	0.141
Toluene	0.625	0.166	0.084	0.070
Uracil	0.581	0.206	0.116	0.117
Table 7: Mean Absolute Error for energy(meV) per-atom forces prediction (meV Å) on rMD17 dataset. Baseline results are taken from Batatia et al. [24]. The best results are bolded.
LEFTNet	MACE	Allegro	BOTNet	NequIP	GemNet (T/Q)	ACE	FCHL	GAP	ANI	PaiNN
Aspirin	E	2.1	2.2	2.3	2.3	2.3	-	6.1	6.2	17.7	16.6	6.9
F	6.4	6.6	7.3	8.5	8.2	9.5	17.9	20.9	44.9	40.6	16.1
Azobenzene	E	0.7	1.2	1.2	0.7	0.7	-	3.6	2.8	8.5	15.9	-
F	3.3	3.0	2.6	3.3	2.9	-	10.9	10.8	24.5	35.4	-
Benzene	E	0.05	0.4	0.3	0.03	0.04	-	0.04	0.35	0.75	3.3	-
F	0.3	0.3	0.2	0.3	0.3	0.5	0.5	2.6	6	10	-
Ethanol	E	0.4	0.4	0.4	0.4	0.4	-	1.2	0.9	3.5	2.5	2.7
F	3.6	2.1	2.1	3.2	2.8	3.6	7.3	6.2	18.1	13.4	10
Malonaldehyde	E	0.8	0.8	0.6	0.8	0.8	-	1.7	1.5	4.8	4.6	3.9
F	5.4	4.1	3.6	5.8	5.1	6.6	11.1	10.3	26.4	24.5	13.8
Naphthalene	E	0.8	0.5	0.2	0.2	0.9	-	0.9	1.2	3.8	11.3	5.1
F	1.9	1.6	0.9	1.8	1.3	1.9	5.1	6.5	16.5	29.2	3.6
Paracetamol	E	1.3	1.3	1.5	1.3	1.4	-	4	2.9	8.5	11.5	-
F	4.7	4.8	4.9	5.8	5.9	-	12.7	12.3	28.9	30.4	-
Salicylic acid	E	0.9	0.9	0.9	0.8	0.7	-	1.8	1.8	5.6	9.2	4.9
F	4.1	3.1	2.9	4.3	4	5.3	9.3	9.5	24.7	29.7	9.1
Toluene	E	0.3	0.5	0.4	0.3	0.3	-	1.1	1.7	4	7.7	4.2
F	2.2	1.5	1.8	1.9	1.6	2.2	6.5	8.8	17.8	24.3	4.4
Uracil	E	0.4	0.5	0.6	0.4	0.4	-	1.1	0.6	3	5.1	4.5
F	2.8	2.1	1.8	3.2	3.1	3.8	6.6	4.2	17.6	21.4	6.1
Table 8: Model and training hyperparameters for our method on different tasks.
Hyperparameter	Values/Search Space
QM9	MD17	rMD17
Number of layers	4, 5, 6	4, 6	4, 6
Hidden channels	128, 192, 256	256	256
Number of radial basis	24, 32, 96	16, 32, 64	16, 32, 64
Cutoff	5, 6, 6.5, 8	6, 8, 10	6, 8, 10
Epochs	800	1000	1000
Batch size	32	1, 4	1, 4
Learning rate	1e-4, 5e-4	5e-4	5e-4
Learning rate scheduler	steplr	steplr	steplr
Learning rate decay factor	0.5	0.5	0.5
Learning rate decay epochs	100	200	200