SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\boldsymbol {x} ^ {\top} (\boldsymbol {I} - \tilde {\boldsymbol {A}}) \boldsymbol {x} = \sum_ {(i, j) \in \mathbb {E}} \left(\frac {x _ {i}}{\sqrt {d _ {i}}} - \frac {x _ {j}}{\sqrt {d _ {j}}}\right) ^ {2} \geq 0.
$$
$$
\boldsymbol {x} ^ {\top} \hat {\boldsymbol {A}} \boldsymbol {x} = \boldsymbol {x} ^ {\top} (- \boldsymbol {I} + \tilde {\boldsymbol {A}}) \boldsymbol {x} + 2 \boldsymbol {x} ^ {\top} \boldsymbol {x} \leq 2 \boldsymbol {x} ^ {\top} \boldsymbol {x}.
$$
This shows that the Rayleigh quotient is bounded by  $\frac{\boldsymbol{x}^\top\hat{A}\boldsymbol{x}}{\boldsymbol{x}^\top\boldsymbol{x}}\leq 2$ , therefore  $\lambda_{i}\leq 2$
Then we give the proof of Theorem 15.
Proof. Let  $0 \leq \lambda_1 \leq \dots \leq \lambda_n \leq 2$  be the eigenvalues of  $\hat{A}$  and  $u_1, \ldots, u_n$  be the corresponding eigenvectors. Then
$$
\hat {\boldsymbol {A}} = \lambda_ {1} \boldsymbol {u} _ {1} \boldsymbol {u} _ {1} ^ {\top} + \dots + \lambda_ {n} \boldsymbol {u} _ {n} \boldsymbol {u} _ {n} ^ {\top}.
$$
$$
\hat {\boldsymbol {A}} ^ {\prime} := \lambda_ {n - k + 1} \boldsymbol {u} _ {n - k + 1} \boldsymbol {u} _ {n - k + 1} ^ {\top} + \dots + \lambda_ {n} \boldsymbol {u} _ {n} \boldsymbol {u} _ {n} ^ {\top}.
$$
By Theorem 1 and the assumptions in our theorem, we know that there exists a permutation-invariant network on sets such that
$$
\left| F \left(\left[ \boldsymbol {X}, \left(\boldsymbol {U} \boldsymbol {\Lambda} ^ {\frac {1}{2}}\right) _ {: -, k:}, \mathbf {0} \right]\right) - \mathrm {N N} \left(\left[ \boldsymbol {X}, \left(\boldsymbol {U} \boldsymbol {\Lambda} ^ {\frac {1}{2}}\right) _ {: -, k:}, \mathbf {0} \right]\right) \right| <   \frac {\varepsilon}{2}.
$$
Since  $f$  can be approximated by an  $L_{p}$ -Lipschitz continuous function, we have
$$
\begin{array}{l} \left| f \left(\left[ \boldsymbol {X}, \hat {\boldsymbol {A}} \right]\right) - F \left(\left[ \boldsymbol {X}, \left(\boldsymbol {U} \boldsymbol {\Lambda} ^ {\frac {1}{2}}\right) _ {: -, k:,:}, \mathbf {0} \right]\right) \right| = \left| f \left(\left[ \boldsymbol {X}, \hat {\boldsymbol {A}} \right]\right) - f \left(\left[ \boldsymbol {X}, \hat {\boldsymbol {A}} ^ {\prime} \right]\right) \right| \\ \leq L _ {p} \big \| [ \boldsymbol {X}, \hat {\boldsymbol {A}} ] - [ \boldsymbol {X}, \hat {\boldsymbol {A}} ^ {\prime} ] \big \| _ {\mathrm {F}} + \frac {\varepsilon}{2} \\ = L _ {p} \big \| [ \mathbf {0}, \lambda_ {1} \pmb {u} _ {1} \pmb {u} _ {1} ^ {\top} + \dots + \lambda_ {n - k} \pmb {u} _ {n - k} \pmb {u} _ {n - k} ^ {\top} ] \big \| _ {\mathrm {F}} + \frac {\varepsilon}{2} \\ = L _ {p} \sqrt {\lambda_ {1} ^ {2} + \cdots + \lambda_ {n - k} ^ {2}} + \frac {\varepsilon}{2} \\ \leq \sqrt {n - k} L _ {p} \lambda_ {n - k} + \frac {\varepsilon}{2}. \\ \end{array}
$$
Combining the two inequalities above gives us
$$
\left| f ([ \boldsymbol {X}, \hat {\boldsymbol {A}} ]) - \mathrm {N N} ([ \boldsymbol {X}, (\boldsymbol {U} \boldsymbol {\Lambda} ^ {\frac {1}{2}}) _ {: -, k:}, \mathbf {0} ]) \right| <   \sqrt {n - k} L _ {p} \lambda_ {n - k} + \varepsilon .
$$
L Dataset details
ZINC (MIT License) consists of 12K molecular graphs from the ZINC database of commercially available chemical compounds. These molecular graphs are between 9 and 37 nodes large. Each node represents a heavy atom (28 possible atom types) and each edge represents a bond (3 possible types). The task is to regress constrained solubility  $(\log P)$  of the molecule. The dataset comes with a predefined 10K/1K/1K train/validation/test split.
OGBG-MOLTOX21 and OGBG-MOLPCBA (MIT License) are molecular property prediction datasets adopted by OGB from MoleculeNet. These datasets use a common node (atom) and edge (bond) featurization that represent chemophysical properties. OGBG-MOLTOX21 is a multi-mask binary graph classification dataset where a qualitative (active/inactive) binary label is predicted against 12 different toxicity measurements for each molecular graph. OGBG-MOLPCBA is also a multi-task binary graph classification dataset from OGB where an active/inactive binary label is predicted for 128 bioassays.
Details of the three datasets are summarized in Table 16.
M Hyperparameter settings
M.1 Real-world tasks
We evaluate the proposed MAP on three real-world datasets: ZINC, OGBG-MOLTOX21 and OGBGMOLPCBA, on a server with 6 NVIDIA 3080 Ti GPUs and 2 NVIDIA 1080 Ti GPUs. We consider
Table 16: Details of the datasets.
Dataset	ZINC	ogbg-moltox21	ogbg-molpcba
#Graphs	12000	7831	437929
Avg #Nodes	23.2	18.6	26.0
Avg #Edges	24.9	19.3	28.1
Task Type	Regression	Binary Classification	Binary Classification
Metric	MAE	ROC-AUC	AP
4 GNN architectures: GatedGCN, PNA, SAN and GraphiT, with 4 different positional encodings: no PE, LapPE with random Sign, SignNet and MAP. We follow the same settings as Dwivedi et al. [18] for models with no PE or LapPE, and same settings as Lim et al. [29] for models with SignNet or MAP. All baseline scores reported in Table 3, 4 & 5 are taken from the original papers. As shown in Figure 7, for models with no PE or LapPE, the input features are directly fed into the base model; for models with SignNet, the eigenvectors are first processed by SignNet and then concatenated with the original node features as input to the base model; for models with MAP, the PEs are first processed by a normal GNN and then concatenated with the original node features as input to the base model. These settings align with the original papers.
Figure 7: Our experiment settings with different PEs.
The main hyperparameters in our experiments are listed as follows.
-  $k$  : the number of eigenvectors used in the PE.
-  $L_{1}$ : the number of layers of the base model.
-  $h_1$ : the hidden dimension of the base model.
-  $h_2$ : the output dimension of the base model.
-  $\lambda$ : the initial learning rate.
-  $t$  : the patience of the learning rate scheduler.
-  $r$ : the factor of the learning rate scheduler.
-  $\lambda_{\mathrm{min}}$ : the minimum learning rate of the learning rate scheduler.
-  $L_{2}$ : the number of layers of SignNet or the normal GNN (when using MAP as PE).
-  $h_3$ : the hidden dimension of SignNet or the normal GNN (when using MAP as PE).
The values of these hyperparameters in our experiments are listed in Table 17.
Table 17: Hyperparameter details for experiments on real-world datasets.
Model	PE	k	L1	h1	h2	λ	t	r	λmin	L2	h3
ZINC	GatedGCN	None	0	16	78	78	0.001	25	0.5	1e-6	-	-
GatedGCN	LapPE + RS	8	16	78	78	0.001	25	0.5	1e-6	-	-
GatedGCN	SignNet	8	16	67	67	0.001	25	0.5	1e-6	8	67
GatedGCN	MAP	8	16	69	67	0.001	25	0.5	1e-5	6	69
PNA	None	0	16	70	70	0.001	25	0.5	1e-6	-	-
PNA	LapPE + RS	8	16	80	80	0.001	25	0.5	1e-6	-	-
PNA	SignNet	8	16	70	70	0.001	25	0.5	1e-6	8	70
PNA	MAP	8	16	70	70	0.001	25	0.5	1e-6	6	70
SAN	None	0	10	64	64	0.0003	25	0.5	1e-6	-	-
SAN	MAP	16	10	40	40	0.0007	25	0.5	1e-5	6	40
GraphiT	None	0	10	64	64	0.0003	25	0.5	1e-6	-	-
GraphiT	MAP	16	10	48	48	0.0007	25	0.5	1e-6	6	48
MOLTOX21	GatedGCN	None	0	8	154	154	0.001	25	0.5	1e-5	-	-
GatedGCN	LapPE + RS	3	8	154	154	0.001	25	0.5	1e-5	-	-
GatedGCN	MAP	3	8	150	150	0.001	22	0.14	5e-6	8	150
PNA	None	0	8	206	206	0.0005	10	0.8	2e-5	-	-
PNA	MAP	16	8	115	113	0.0005	10	0.8	8e-5	7	115
SAN	None	0	10	88	88	0.0007	25	0.5	1e-6	-	-
SAN	MAP	12	10	88	88	0.0007	25	0.5	1e-5	8	88
GraphiT	None	0	10	88	88	0.0007	25	0.5	1e-6	-	-
GraphiT	MAP	16	10	64	64	0.0007	25	0.5	1e-6	6	64
MOLPCBA	GatedGCN	None	0	8	154	154	0.001	25	0.5	1e-4	-	-
GatedGCN	LapPE + RS	3	8	154	154	0.001	25	0.5	1e-4	-	-
GatedGCN	MAP	3	8	200	200	0.001	25	0.5	1e-5	8	200
PNA	None	0	4	510	510	0.0005	4	0.8	2e-5	-	-
PNA	MAP	16	4	304	304	0.0005	10	0.8	2e-5	8	304
M.2 Synthetic tasks
To verify the expressive power of RSE, we conduct experiments on the synthetic EXP dataset. The dataset consists of a set of 1-WL indistinguishable non-isomorphic graph pairs. If a network reaches above  $50\%$  accuracy on this dataset, it must have expressive power beyond the 1-WL test. DeepSets-RSE is a two-layer DeepSets model with RSE as PE, whereas Linear-RSE is a one-layer linear model with RSE as PE. We use Optuna [3] to optimize the hyperparameters of our models. The values of hyperparameters of our models are as follows:
- **DeepSets-RSE:** the learning rate  $\lambda = 0.002385602941230316$ , the hidden dimension of the first linear layer  $w_{1} = 60$ , the hidden dimension of the second linear layer  $w_{2} = 76$ , the dropout rate [46]  $p = 0.13592575703525184$ , the weight decay of Adam optimizer  $wd = 0.0005$ .
- Linear-RSE: the learning rate  $\lambda = 0.0006867736568978745$ , the hidden dimension  $w = 109$ , the weight decay of Adam optimizer  $wd = 0.0001$ .