SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {E} \| \lambda_ {T} - \lambda^ {*} \| _ {2} ^ {2} \leq \frac {1 6 T _ {\kappa} ^ {2} \| \lambda_ {0} - \lambda^ {*} \| _ {2} ^ {2}}{\mathrm {e} ^ {2} T ^ {2}} + \frac {8 \sigma^ {2}}{\mu^ {2} T} \leq \epsilon
$$
By multiplying both sides with  $T^2$  as
$$
T ^ {2} \varepsilon - \frac {8 \sigma^ {2}}{\mu^ {2}} T - \frac {1 6 T _ {\kappa} ^ {2} \left\| \lambda_ {0} - \lambda^ {*} \right\| _ {2} ^ {2}}{\mathrm {e} ^ {2}} \geq 0, \tag {11}
$$
we can see that we are looking for the smallest positive integer that is larger than the solution of a quadratic equation with respect to  $T$ . This is given as
$$
T \geq \frac {\frac {8 \sigma^ {2}}{\mu^ {2}} + \sqrt {\left(\frac {8 \sigma^ {2}}{\mu^ {2}}\right) ^ {2} + 6 4 \epsilon \frac {T _ {\kappa} ^ {2} \| \lambda_ {0} - \lambda^ {*} \| _ {2} ^ {2}}{\mathrm {e} ^ {2}}}}{2 \epsilon}.
$$
Applying the inequality  $\sqrt{a + b} \leq \sqrt{a} + \sqrt{b}$ ,
$$
\begin{array}{l} \frac {\frac {8 \sigma^ {2}}{\mu^ {2}} + \sqrt {\left(\frac {8 \sigma^ {2}}{\mu^ {2}}\right) ^ {2} + 6 4 \epsilon \frac {T _ {\kappa} ^ {2} \| \lambda_ {0} - \lambda^ {*} \| _ {2} ^ {2}}{\mathrm {e} ^ {2}}}}{2 \epsilon} \leq \frac {\frac {8 \sigma^ {2}}{\mu^ {2}} + \left(\frac {8 \sigma^ {2}}{\mu^ {2}}\right) + \sqrt {6 4 \epsilon \frac {T _ {\kappa} ^ {2} \| \lambda_ {0} - \lambda^ {*} \| _ {2} ^ {2}}{\mathrm {e} ^ {2}}}}{2 \epsilon} \\ = \frac {\frac {1 6 \sigma^ {2}}{\mu^ {2}} + \sqrt {\epsilon} \frac {8 T _ {\kappa} \| \lambda_ {0} - \lambda^ {*} \| _ {2}}{\mathrm {e}}}{2 \epsilon} \\ = \frac {8 \sigma^ {2}}{\mu^ {2} \epsilon} + \frac {4 T _ {\kappa} \| \lambda_ {0} - \lambda^ {*} \| _ {2}}{\mathrm {e} \sqrt {\epsilon}}. \\ \end{array}
$$
Thus,  $\mathbb{E}\| \lambda_T - \lambda^*\| _2^2\leq \epsilon$  can be satisfied with a number of iterations at least
$$
T \geq \max \left(\frac {8 \sigma^ {2}}{\mu^ {2} \epsilon} + \frac {4 T _ {\kappa} \| \lambda_ {0} - \lambda^ {*} \| _ {2}}{\mathrm {e} \sqrt {\epsilon}}, 4 T _ {\kappa}\right).
$$
G Details of Experimental Setup
Table 2: Summary of Datasets and Problems
Abbrev.	Model	Dataset	d	N
LME-election	Linear Mixed Effects	1988 U.S. presidential election (Gelman & Hill, 2007)	90	11,566
LME-radon	U.S. household radon levels (Gelman & Hill, 2007)	391	12,573
BT-tennis	Bradley-Terry	ATP World Tour tennis	6030	172,199
LR-keggu	KEGG-undirected (Shannon et al., 2003)	31	63,608
LR-song	Linear Regression	million songs (Bertin-Mahieux et al., 2011)	94	515,345
LR-buzz	buzz in social media (Kawala et al., 2013)	81	583,250
LR-electric	household electric	15	2,049,280
AR-ecg	Sparse Autoregression	Long-term ST ECG (Jager et al., 2003)	63	20,642,000
Linear Regression (LR-\*) We consider a basic Bayesian hierarchical linear regression model
$$
\sigma_ {\alpha} \sim \mathcal {N} _ {+} (0, 1 0 ^ {2}), \quad \sigma_ {\beta} \sim \mathcal {N} _ {+} (0, 1 0 ^ {2}), \quad \sigma \sim \mathcal {N} _ {+} (0, 0. 3 ^ {2})
$$
$$
\boldsymbol {\beta} \sim \mathcal {N} \left(\mathbf {0}, \sigma_ {\beta} ^ {2} \boldsymbol {I}\right), \quad \alpha \sim \mathcal {N} \left(0, \sigma_ {\alpha} ^ {2}\right),
$$
$$
y _ {i} \sim \mathcal {N} \left(\boldsymbol {\beta} ^ {\top} \boldsymbol {x} _ {i} + \alpha , \sigma^ {2}\right),
$$
where a weakly informative half-normal hyperprior  $\mathcal{N}_{+}$ , a normal distribution with the support restricted to  $\mathbb{R}_+$ , is assigned on the hyperparameters. For the datasets, we consider large-scale regression problems obtained from the UCI repository (Dua & Graff, 2017), shown in Table 2. For all datasets, we standardize the regressors  $\pmb{x}_i$  and the outcomes  $y_{i}$ .
Radon Levels (MLE-radon) MLE-radon is a radon level regression problem by Gelman & Hill (2007). It fits a hierarchical mixed-effects model for estimating household radon levels across different counties while considering the floor elevation of each site. The model is described as
$$
\sigma \sim \mathcal {N} _ {+} (0, 1 ^ {2}), \quad \sigma_ {\alpha} \sim \mathcal {N} _ {+} (0, 1 ^ {2}), \quad \mu_ {\alpha} \sim \mathcal {N} (0, 1 0 ^ {2}), \quad \epsilon \sim \mathcal {N} (\mathbf {0}, 1 0 ^ {2} \mathbf {I})
$$
$$
\beta_ {1} \sim \mathcal {N} (0, 1 0 ^ {2}), \quad \beta_ {2} \sim \mathcal {N} (0, 1 0 ^ {2})
$$
$$
\alpha = \mu_ {\alpha} + \sigma_ {\alpha} \epsilon
$$
$$
\mu_ {i} = \alpha \left(\text {c o u n t y} _ {i}\right) + \beta_ {1} \log \left(\operatorname {u p p m} _ {i}\right) + \text {f l o o r} _ {i} \beta_ {2}
$$
$$
\log \operatorname {r a d o n} _ {i} \sim \mathcal {N} (\mu_ {i}, \sigma^ {2}),
$$
which uses variable slopes and intercepts with non-centered parameterization. The dataset was obtained from PosteriorDB (Magnusson et al., 2022). Also, for the radon regression problem, the Minnesota subset is often used due to computational reasons. Here, we use the full national dataset.
Presidential Election (MLE-election) MLE-election is a model for studying the effects of sociological factors on the 1988 United States presidential election (Gelman & Hill, 2007). The model is described as
$$
\sigma_ {\text {a g e}} \sim \mathcal {N} (0, 1 0 0 ^ {2}), \quad \sigma_ {\text {e d u}} \sim \mathcal {N} (0, 1 0 0 ^ {2}), \quad \sigma_ {\text {a g e} \times \text {e d u}} \sim \mathcal {N} (0, 1 0 0 ^ {2})
$$
$$
\sigma_ {\text {s t a t e}} \sim \mathcal {N} \left(0, 1 0 0 ^ {2}\right), \quad \sigma_ {\text {r e g i o n}} \sim \mathcal {N} \left(0, 1 0 0 ^ {2}\right),
$$
$$
\boldsymbol {b} _ {\mathrm {a g e}} \sim \mathcal {N} \left(\boldsymbol {0}, \sigma_ {\mathrm {a g e}} ^ {2} \mathbf {I}\right), \quad \boldsymbol {b} _ {\mathrm {e d u}} \sim \mathcal {N} \left(\boldsymbol {0}, \sigma_ {\mathrm {e d u}} ^ {2} \mathbf {I}\right), \quad \boldsymbol {b} _ {\mathrm {a g e} \times \mathrm {e d u}} \sim \mathcal {N} \left(\boldsymbol {0}, \sigma_ {\mathrm {a g e} \times \mathrm {e d u}} ^ {2} \mathbf {I}\right),
$$
$$
\boldsymbol {b} _ {\text {s t a t e}} \sim \mathcal {N} \left(\boldsymbol {0}, \sigma_ {\text {s t a t e}} ^ {2} \mathbf {I}\right), \quad \boldsymbol {b} _ {\text {r e g i o n}} \sim \mathcal {N} \left(\boldsymbol {0}, \sigma_ {\text {r e g i o n}} ^ {2} \mathbf {I}\right)
$$
$$
\begin{array}{l} \boldsymbol {\beta} \sim \mathcal {N} (\mathbf {0}, 1 0 0 ^ {2} \mathbf {I}) \\ p _ {i} = \beta_ {1} + \beta_ {2} \text {b l a c k} _ {i} + \beta_ {3} \text {f e m a l e} _ {i} + \beta_ {4} v _ {\text {p r e v}, i} + \beta_ {5} \text {f e m a l e} _ {i} \text {b l a c k} _ {i} \\ + b _ {\text {a g e}} \left[ \text {a g e} _ {i} \right] + b _ {\text {e d u}} \left[ \text {e d u} _ {i} \right] + b _ {\text {a g e} \times \text {e d u}} \left[ \text {a g e} _ {i} \text {e d u} _ {i} \right] + b _ {\text {s t a t e}} \left[ \text {s t a t e} _ {i} \right] + b _ {\text {r e g i o n}} \left[ \text {r e g i o n} _ {i} \right] \\ \end{array}
$$
$$
y _ {i} \sim \operatorname {b e r n o u l l i} \left(p _ {i}\right).
$$
The dataset was obtained from PosteriorDB (Magnusson et al., 2022).
Bradley-Terry (BT-Tennis) BT-Tennis is a Bradley-Terry model for estimating the skill of professional tennis players used by Giordano et al. (2023). The model is described as
$$
\sigma \sim \mathcal {N} _ {+} (0, 1)
$$
$$
\boldsymbol {\theta} \sim \mathcal {N} (\mathbf {0}, \sigma^ {2} \mathbf {I})
$$
$$
p _ {i} \sim \theta [ \operatorname {w i n} _ {i} ] - \theta [ \log_ {i} ]
$$
$$
y _ {i} \sim \operatorname {b e r n o u l l i} (p),
$$
where  $\mathrm{win}_i$ ,  $\mathrm{los}_i$  are the indices of the winning and losing players for the  $i$ th game, respectively. While we subsample over the games  $i = 1, \dots, N$ , each player's involvement is sparse in that each player plays only a handful of games. Consequently, the subsampling noise is substantial. Therefore, we use a larger batch size of 500. Similarly to Giordano et al. (2023), we use the ATP World Tour data publicly available online<sup>1</sup>.
Autoregression (AR-ecg) AR-ecg is a linear autoregressive model. Here, we use a Student-t likelihood as originally proposed by Christmas & Everson (2011). While they originally imposed an automatic relevance detection prior on the autoregressive coefficients, we instead set a horseshoe shrinkage prior (Carvalho et al., 2009, 2010). Since the horseshoe is known to result in complex posterior geometry, this should make the problem more challenging. The model is described as
$$
\alpha_ {d} = 1 0 ^ {- 2}, \quad \beta_ {d} = 1 0 ^ {- 2}, \quad \alpha_ {d} = 1 0 ^ {- 2}, \quad \beta_ {d} = 1 0 ^ {- 2},
$$
$$
d \sim \operatorname {g a m m a} \left(\alpha_ {d}, \beta_ {d}\right),
$$
$$
\sigma^ {- 1} \sim \text {i n v e r s e - g a m m a} \left(\alpha_ {\sigma}, \beta_ {\sigma}\right),
$$
$$
\tau \sim \text {c a u c h y} _ {+} (0, 1),
$$
$$
\lambda \sim \operatorname {c a u c h y} _ {+} (\mathbf {0}, \mathbf {1}),
$$
$$
\theta \sim \mathcal {N} (0, \tau \operatorname {d i a g} (\lambda))
$$
$$
y [ n ] \sim \text {s t d u e n t - t} (d, \theta_ {1} y [ n - 1 ] + \theta_ {2} y [ n - 2 ] + \dots + \theta_ {P} y [ n - P ], \sigma),
$$
where  $d$  is the degrees-of-freedom for the Student-t likelihood,  $\mathrm{cauchy}_{+}$  is a half-Cauchy prior.
For the dataset, we use the long-term electrocardiogram measurements of Jager et al. (2003) obtained from Physionet (Goldberger et al., 2000). The data instance we used has a duration of 23 hours sampled at  $250\mathrm{Hz}$  with 12-bit resolution over a range of  $\pm 10$  millivolts. During the experiments, we observed that the hyperparameters suggested by Christmas & Everson are sensitive to the signal amplitude. Therefore, we scaled the signal amplitude to be  $\pm 10$ .
<https://databhub.io/sports-data/atp-world-tour-tennis-data>
(c) BT-tennis
Figure 5: BBVI convergence speed (ELBO v.s. Iteration) and robustness against stepsize (ELBO at  $T = 50,000$  v.s. Base stepsize). The error bands are the  $80\%$  quantiles estimated from 20 (10 for AR-eeg) independent replications. The initial point was  $\mathbf{m}_0 = \mathbf{0}, C_0 = \mathbf{I}$ .
(d) LR-keggu
(a) LR-song
(b) LR-buzz
Figure 6: BBVI convergence speed (ELBO v.s. Iteration) and robustness against stepsize (ELBO at  $T = 50,000$  v.s. Base stepsize). The error bands are the  $80\%$  quantiles estimated from 20 (10 for AR-eeg) independent replications. The initial point was  $m_0 = 0$ ,  $C_0 = 1$ .
(c) LR-electric
(d) AR-ecg