SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\frac {\partial E [ \tilde {d} _ {s} ]}{c _ {s}} = \sum_ {t \rightarrow s} \mathbb {1} [ c _ {s} \pi_ {t s} <   1 ] \pi_ {t s}
$$
Then to compute the derivative of the variance with respect to the expected number of sampled vertices for a given seed vertex  $s$ , which is  $\frac{\partial v_s}{\partial E[\tilde{d}_s]}$ , we can use the chain rule and get:
$$
\frac {\partial v _ {s}}{\partial E \left[ \tilde {d} _ {s} \right]} = \frac {\partial v _ {s}}{\partial c _ {s}} \frac {\partial c _ {s}}{\partial E \left[ \tilde {d} _ {s} \right]} = \frac {\frac {- 1}{\left(A _ {* s}\right) ^ {2}} \sum_ {t \rightarrow s} \frac {\mathbb {1} \left[ c _ {s} \pi_ {t s} <   1 \right] A _ {t s} ^ {2}}{c _ {s} ^ {2} \pi_ {t s}}}{\sum_ {t \rightarrow s} \mathbb {1} \left[ c _ {s} \pi_ {t s} <   1 \right] \pi_ {t s}} \tag {26}
$$
Then, what one would do is to choose a constant  $C(k)$  as a function of the fanout parameter  $k$  and set  $\frac{\partial v_{s}}{\partial E[d_{s}]} = C(k)$  and solve for  $c_{s}$ .  $C(k)$  would be a negative quantity whose absolute value decreases as  $k$  increases. It would probably look like  $C(k) = \frac{-C'}{k^2}$  for some constant  $C' > 0$ . We leave this as future work.
A.6 Convergence speed with respect to wall time
In this section, we perform hyperparameter optimization on Neighbor Sampler and Labor Sampler so that the training converges to a target validation accuracy as fast as possible. We leave LADIES out of this experiment because it is too slow as can be seen in the last column of Table 2. We ran this experiment on an A100 GPU and stored the input features on the main memory, which were accessed over the PCI-e directly during training by pinning their memory. This kind of training scenario is commonly used when training on large datasets whose input features don't fit in the GPU memory. We use larger of the two datasets we have for this experiment, products and yelp. For products, the validation accuracy target we set is  $91.5\%$ , and for yelp it is  $60\%$ . We tune the learning rate between  $[10^{-4}, 10^{-1}]$ , the batch size between  $[2^{10}, 2^{15}]$  and the fanout for each layer between [5, 25]. We use the same model used in Section 4. For LABOR, we additionally tune over the number of importance sampling iterations i between [0, 3] so that it can switch between LABOR-  $i$  and also a layer dependency boolean parameter that makes LABOR use the same random variates  $r_t$  for different layers when enabled, which has the effect of increasing the overlap of sampled vertices across layers. We use the state-of-the-art hyperparameter tuned HEBO [Cowen-Rivers et al., 2020], the winning submission to the NeurIPS 2020 Black-Box Optimization Challenge, to tune the parameters of the sampling algorithm with respect to runtime required to reach the target validation accuracy with a timeout of 300 seconds, terminating the run if the configuration doesn't reach the target accuracy.
We let HEBO run overnight and collect the minimum runtimes required to achieve the target accuracies. For products, the fastest hyperparameter corresponding to the 38.2s runtime had fanouts (18,5,25), batch size 10500, learning rate 0.0145, used LABOR-1, layer dependency False.
For Neighbor Sampler, the fastest hyperparameter corresponding to 43.82s runtime had fanouts (15, 5, 21), batch size 12000, learning rate 0.0144.
For the Yelp dataset, the fastest hyperparameter corresponding to the 41.60s runtime had fanouts (6,5,7), batch size 5400, learning rate 0.000748, used LABOR-1 and layer dependency True.
For Neighbor Sampler, the fastest hyperparameter corresponding to the 47.40s runtime had fanouts  $(5,6,6)$ , batch size 4600, learning rate 0.000931.
Figure 4: The runtimes to reach a validation F1-score of  $91.5\%$  on products and  $60\%$  on yelp, belonging to runs tried by the HEBO hyperparameter tuner, sorted with respect to their runtimes. Having a lower curve means that a method is faster overall compared to the others. HEBO could not find any hyperparameter configuration to reach the set target on products, hence its curve is left out.
These results clearly indicate that training with LABOR is faster compared to Neighbor Sampling when it comes to time to convergence.
We run the same experiment with GraphSAINT [Zeng et al., 2020] using the DGL example code, both on ogbn-products and yelp using the same model architecture. We used their edge sampler, the edge sampling budget between  $[2^{10}, 2^{15}]$  and learning rate between  $[10^{-4}, 10^{-1}]$ . We disabled batch normalization to make it a fair comparison with NS and LABOR since the models they are using does not have batch normalization. We use HEBO to tune these hyperparameters to reach the set validation accuracy, and let it run overnight. The results show that HEBO was not able to find any configuration of the hyperparameters reaching  $91.5\%$  accuracy on products faster than 1500 seconds.
For the Yelp dataset, the fastest runtime to reach  $60\%$  accuracy was 92.2s, with an edge sampling budget of 12500 and learning rate 0.0214.
A.7 Reducing debiased LADIES complexity from quadratic to linear time
Our proposed PLADIES algorithm in Section 3.1 is linear time and unbiased by construction via the use of Poisson Sampling, which has the same comparative advantages as sampling without replacement over sampling with replacement. However, LADIES authors Zou et al. [2019] formulate their sampling framework via sampling with replacement while their implementation uses sampling without replacement, without addressing the bias created thereby. Chen et al. [2023] concurrently noticed the same bias issue as us, and proposed a way to "de-bias" sampling without replacement by computing actual sampling probabilities in quadratic time (Chen et al. [2023], Section 5.5).
When one uses weighted sampling without replacement as in Chen et al. [2023], the sampling probability is actually not the weights provided initially and Chen et al. [2023] proposes a way to compute the actual probabilities and a way to de-bias them in quadratic time. There is no convergence difference regarding PLADIES and the de-bias procedure proposed in Chen et al. [2023], only the computational complexity of sampling is different. They also propose a way to flatten the probabilities and these flattened probabilities can be fed to PLADIES as well. So the flattening approach Chen et al. [2023] proposes is orthogonal to our work.
To give an example of why weighted sampling without replacement is biased, consider the probabilities [0.7, 0.2, 0.1]. Now, if we want to sample 2 items without replacement, then notice that the probability of the first item, 0.7, is more than 2 times the rest of the items, 0.3. Thus, the first item needs to be sampled always. But even if we always sample the first item, the second item is going to be either 0.2 or 0.1, the probability of which one it is going to be is [ \frac{2}{3}, \frac{1}{3} ]. Thus, in the end, the probability of sampling individual items is [ \frac{1}{2}, \frac{2}{3}, \frac{1}{3} ]. However, these resulting probabilities are not linearly proportional to the probabilities we started with. Chen et al. [2023] fixes this in quadratic time. We propose to use Poisson Sampling instead. With Poisson sampling, we can have a constant  $c$  to adjust how many items we want to sample. We want 2 items in expectation, so  $0.7c + 0.2c + 0.1c = 2$ . But the solution is  $c = 2$ , and the probability of sampling the first item becomes 1.4, which is not possible. So, we need to solve for  $c$  in the following equation:  $\min(0.7c, 1) + \min(0.2c, 1) + \min(0.1c, 1) = 2$ . The solution is  $c = 10/3$ , and this can be found in linear time. The resulting probabilities become:  $\min(0.7c, 1) = 1$ ,  $\min(0.2c, 1) = 2/3$ ,  $\min(0.1c, 1) = 1/3$ . Because we use these adjusted probabilities during GNN aggregation, our method is unbiased, and faster than Chen et al. [2023] due to the drastic computational complexity difference (quadratic vs linear).
A.8 Links to the experimental code
Experiments in Section 4 can be reproduced using https://github.com/dmlc/dgl/tree/ f971e25a4cff33d5d219dde523e32c62360ffd2/examples/pytorch/labor. Our initial contribution to DGL can be found in https://github.com/dmlc/dgl/pull/4668.