SlowGuess/ACL_2020_OCR huggingface dataset:
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1532-1543.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2227-2237.
Nils Reimers and Iryna Gurevych. 2017. Reporting score distributions makes a difference: Performance study of LSTM-networks for sequence tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 338-348.
Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. 2019. Green AI. arXiv:1907.10597.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642.
Model	Mode	Batch Size	Learning Rate	Seed	Dropout	# Layers	Hidden Dim.	WDrop	EDrop	βEMA
MLP	-	(16,32,64)	0.001	[0,107]D	[0.05,0.7]	1	[256,768]D	-	-	-
LSTM	(nonstatic[0.5], static[0.4], rand[0.1])	(16,32,64)	TExp[0.001,0.099]	[0,107]D	[0.05,0.7]	(1[0.75],2[0.25])	[384,768]D	[0,0.3]	[0,0.3]	[0.985,0.995]
A Cautionary Notes
We caution that the estimator described in the text of Dodge et al. is  $\hat{V}_n^n$ . This is clear from their equation (7) where the empirical distribution is defined over the first  $n$  samples, instead of the  $B$  samples that we use here. In other words, they claim, at least in the text, to use  $\hat{F}_n$  instead of  $\hat{F}_B$  for their estimator  $\hat{V}_n^n$ . Clearly, the estimator  $\hat{V}_n^n$  is (much) worse than  $\hat{V}_n^B$  since the latter exploits all  $B$  samples while the former only looks at the first  $n$  samples. However, close examination of their codebase<sup>1</sup> reveals that they use  $\hat{V}_n^B$ , so the paper discrepancy is a simple notation error.
Lastly, we mention that our notation for  $\hat{U}_n^B$  and  $\hat{V}_n^B$  is motivated by the fact that the former is a  $U$ -statistic while the latter is a  $V$ -statistic. The relation between the two has been heavily studied in statistics since Hoeffding's seminar work. For us, it suffices to point out that  $\hat{V}_n^B \leq \hat{U}_n^B$ , with the latter being unbiased while the former is only asymptotically unbiased. The difference between the two is more pronounced when  $n$  is close to  $B$ . We note that  $\hat{U}_n^B$  can be computed by a reasonable approximation of the binomial coefficients, using say Stirling's formula.
Theorem 3. If the sample does not contain the population maximum,  $\mathrm{KS}(\hat{F}_B^n,F^n)\to 1$  exponentially quickly as  $n$  and  $B$  increase.
Proof. Suppose  $v^{*}$  is not in the sample  $v_{1},\ldots ,v_{B}$ , where  $v_{1}\leq \dots \leq v_{B} < v^{*}$ . Then
$$
\sup _ {x \in \mathbb {R}} | \hat {F} _ {B} ^ {n} (x) - F ^ {n} (x) | \geq | \hat {F} _ {B} ^ {n} (v _ {B}) - F ^ {n} (v _ {B}) |.
$$
From Equation 2.1,  $\hat{F}_B^n (v_B) = (\hat{F}_B(v_B))^n = 1>$ $(F(v_{B}))^{n} = F^{n}(v_{B})$  , hence
$$
| \hat {F} _ {B} ^ {n} (v _ {B}) - F ^ {n} (v _ {B}) | = 1 - (F (v _ {B})) ^ {n}.
$$
Thus concluding the proof.
Table 1: Hyperparameter random search bounds.  $[\cdot, \cdot]_D$  indicates a discrete uniform range, while  $[\cdot, \cdot]$  continuous uniform.  $\mathrm{TEXP}[\cdot, \cdot]$  denotes the truncated exponential distribution. Tuples represent categorical distributions, uniform by default. WDrop and EDrop denote weight and embed dropout. For the GloVe- and ELMo-based search bounds, see https://github.com/allenai/show-your-work.
Model	#Runs	Bandwidth	Support	Bins
MLP	145	0.0049	[0.72, 0.82]	511
LSTM	152	0.059	[-0.18, 1.08]	511
GloVe	114	0.018	[0.46, 0.97]	511
ELMo	84	0.041	[0.39, 0.99]	511
Table 2: Model kernel parameters. Bandwidth chosen using Scott's normal reference rule. Bins denote the number of discretized slots.
Figure 5: Gaussian kernel density estimators fitted to each model's results, along with the histograms of the original runs.
We present hyperparameters in Tables 1 and 2 and Figure 5. We conduct all GloVe and ELMo experiments using PyTorch 1.3.0 with CUDA 10.0 and cuDNN 7.6.3, running on NVIDIA Titan RTX, Titan V, and RTX 2080 Ti graphics accelerators. Our MLP and LSTM experiments use PyTorch 0.4.1 with CUDA 9.2 and cuDNN 7.1.4, running on RTX 2080 Ti's. We use  $\mathrm{Hedwig}^2$  for the document classification experiments and the Show Your Work codebase (see link in Table 1) for the sentiment classification ones.
<https://github.com/allenai/allentune>
$^{2}$ https://github.com/castorini/hedwig