SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In a word, our analysis combines the idea of decomposing the error into bias and variance terms as in nonparametric regression methods and the spirit of the covering number arguments in the analysis of parametric models. Such a combination is novel and non-trivial, which is critical for the relaxation of the assumptions.
E Numerical Experiment Details
In this part, we provide detailed experiment settings for Section 4.
E.1 Evaluation metrics
- Mean Absolute Calibration Error (MACE): for (expected) quantile level  $\tau$  and a quantile prediction model  $Q(X, \tau)$ , define "observed quantile level" by
$$
\tau^ {o b s} := \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {1} \left\{Y _ {i} \leq Q \left(X _ {i}, \tau\right) \right\}.
$$
For pre-specified quantile levels  $0 \leq \tau_{1} < \dots < \tau_{M} \leq 1$ , MACE is calculated by:
$$
\operatorname {M A C E} \left(\left\{\left(X _ {i}, Y _ {i}\right) \right\} _ {i = 1} ^ {n}, Q\right) := \frac {1}{M} \sum_ {j = 1} ^ {M} \left| \tau_ {j} - \tau_ {j} ^ {\text {o b s}} \right|.
$$
In practice, we set  $M = 100$  and pick  $\tau$ 's from 0.01 to 0.99 with equal step size.
- Adversarial Group Calibration Error (AGCE): introduced in Zhao et al. [2020], AGCE is calculated in practice by first sub-sampling (possibly with replacement)  $B_{1}, \dots, B_{r} \subseteq \{(X_{i}, Y_{i})\}_{i=1}^{n}$  and then retrieve the maximum MACE from the pool:
$$
\operatorname {A G C E} (\{B _ {k} \} _ {k = 1} ^ {r}, Q) := \max  _ {k} \{\operatorname {M A C E} (B _ {k}, Q) \}.
$$
- Check Score: check score is an empirical version of the expected pinball loss. For prespecified quantile levels  $0 \leq \tau_{1} < \dots < \tau_{m} \leq 1$  (in our experiments  $\tau$ 's are set in the same way as in MACE), the check score is defined as:
$$
\operatorname {C h e c k S c o r e} \left(\left\{\left(X _ {i}, Y _ {i}\right) \right\} _ {i = 1} ^ {n}, Q\right) := \frac {1}{m} \sum_ {j = 1} ^ {m} \rho_ {\tau_ {j}} \left(Q (X, \tau_ {j}), Y\right),
$$
with  $\rho_{\tau}(Q(X,\tau),Y)\coloneqq \frac{1}{n}\sum_{i = 1}^{n}l_{\tau}(Q(X_i,\tau),Y_i)$
- Length: our length metric gives the average interval length of a predicted  $90\%$  confidence interval. The interval is given as the area between the lower quantile (at level  $\tau_{\mathrm{low}} = 0.05$ ) and the upper quantile (at level  $\tau_{\mathrm{up}} = 0.95$ ):
$$
\operatorname {L e n g t h} \left(\left\{\left(X _ {i}, Y _ {i}\right) \right\} _ {i = 1} ^ {n}, Q\right) := \frac {1}{n} \sum_ {i = 1} ^ {n} \left(Q \left(X, \tau_ {\mathrm {u p}}\right) - Q \left(X, \tau_ {\mathrm {l o w}}\right)\right).
$$
- Coverage: to encourage a fairer comparison, we always attach the empirical coverage rate to the length metric. Coverage rate evaluates the percentage of real data that is covered by the given  $90\%$  confidence intervals:
$$
\operatorname {C o v e r a g e} \left(\left\{\left(X _ {i}, Y _ {i}\right) \right\} _ {i = 1} ^ {n}, Q\right) := \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {1} \left\{Q \left(X _ {i}, \tau_ {\text {l o w}}\right) \leq Y _ {i} \leq Q \left(X _ {i}, \tau_ {\text {u p}}\right) \right\}.
$$
We make a brief explanation for the selection of the performance metrics as follows. The former three metrics are typical measurements in the literature of quantile regression and uncertainty calibration that measure the quality of the calibrations, while the latter two are classic metrics in the field of conformal prediction. Although the quantile calibration task is very closely related to the conformal prediction task, the goals are still different. Conformal prediction aims to provide as sharp as possible prediction sets that cover the true outcomes with desired rates. While a precise quantile prediction guarantees the coverage rate, such a covering band may not be the sharpest. As is discussed in Appendix B, the additional sharpness regularization term could harm the goal of precise quantile prediction. For example, people may select 0.05 and 0.95 quantiles to construct a 0.9 coverage band, but the sharpest covering band probably would deviate from such quantiles. However, a high-quality quantile prediction alone is of independent interest for many downstream tasks, such as the newsvendor problem. This points to a difference between the objectives of calibration and conformal prediction. As a result, the popular performance metrics applied in conformal prediction such as the average interval length and the coverage rate are not direct measurements for the quantile calibration problem. Nevertheless, we still present the Length and the Coverage metrics for completeness. Some other measurements, for example, the independence between the coverage indicator and the band length in Feldman et al. [2021], are only a necessary condition of the quantile calibration and are therefore not considered here.
E.2 Synthetic Dataset
For an example of heteroscedastic Gaussian noises, consider an underlying generation scheme of  $(X,Y)\in \mathbb{R}^2$  from  $X\sim \mathrm{Unif}[0,15]$  and  $Y|X\sim \mathcal{N}(\mu_0(X),\sigma_0^2 (X))$ , with  $\mu_0(X) = 4\cdot \sin (\frac{2\pi}{15} X)$  and  $\sigma_0(X) = \max \{0.2X\cdot |\sin (X)|,0.1\}$ . We generate  $n_{\mathrm{train}} = 40000$  samples for training and
$n_{\mathrm{test}} = 1000$  for testing. For our NRC method, we randomly select  $n_1 = 36000$  for training a regression model and  $n_{\mathrm{train}} - n_1 = 4000$  samples for the quantile calibration step. All neural networks used in this example have hidden layers of size [100, 50]. As shown in Figure 1, our NRC method successfully captures the actual fluctuation of conditional variance, while the traditional benchmark model Deep Ensemble which relies on 5 heteroscedastic neural networks cannot capture the individual dependence on the feature of the noise variance.
E.3 Dimension reduction implementation
As given in Theorem 1, for high dimensional data (i.e., large  $d$ ) the upper bound guarantee may get too loose as a theoretical guarantee. Motivated by this dilemma, we propose applying dimension-reduction techniques when doing nonparametric approximation. We summarize them as the NRC-DR algorithms. There are loads of choices for dimension reduction, in this work we only implement three of them: random projection, covariate selection, and second-to-last layer feature extraction.
- Random Projection Dasgupta [2013]: we use the Gaussian random projection method, in which we project the  $d$ -dimensional inputs on the column space of a randomly generated  $d_0 \times d$  matrix (assuming that  $d_0 < d$ , otherwise original input will be retained), whose elements are independently generated from  $\mathcal{N}(0, \frac{1}{d})$ .
- Covariate Selection: we select the most relevant features from the input vector. In this experiment, we select  $d_0$  out of  $d$  features that have the largest absolute value of Pearson correlation coefficient with the target variable.
- Second-to-Last Layer Embedding: The second-to-last layer of a trained neural network has been widely used for feature extraction when the input dimension is high. We add an additional layer of size  $d_0$  to the regression network  $\hat{f}(\cdot)$  in Algorithm 2 and directly use this layer as the feature embedding.
NRC algorithms implemented with each of these dimension-reduction techniques are called NRC-RP, NRC-Cov, and NRC-Embed respectively. In the UCI-8 experiments we implement NRC-RP and NRC-Cov, with the full result presented in Table 4, while for high-dimensional datasets we choose the medical expenditure panel survey (MEPS) datasets [panel 19, 2017, panel 20, 2017, panel 21, 2017] as suggested in [Romano et al., 2019] to test out our NRC-RP and NRC-Embed variants. The sizes and feature dimensions of these datasets are listed in Table 3. Due to the relative abundance of training data, we set the hidden layers of neural networks to [64, 64], the rest hyperparameters are pre-determined by grid search. For the results presented in Table 2, we reduce the dimension to  $d_0 = 30$ .
E.4 Benchmark methods
We explain the benchmark models and uncertainty quantification methods we are competing against in Section 4. The first type of method is based on the Gaussian assumption. They assume that the conditional distribution of  $Y$  given  $X$  can be fully determined by the conditional mean and variance. These models are of the same output structure of two neurons: one for prediction of  $\mu(X)$ , another for  $\sigma(X)$ . At the implementation of the experiments, we assume that  $\mu(X)$  and  $\sigma(X)$  share the same overall network structure, with two hidden layers of size [20, 20] for small datasets and [64, 64] for large or high dimensional datasets. Amongst them, MC-Dropout (MCDrop) Gal and Ghahramani [2016], Deep Ensembles (DeepEnsemble) Lakshminarayanan et al. [2017] and Heteroscedastic Neural Network (HNN) Kendall and Gal [2017] are trained by minimizing the negative log-likelihood (NLL) error. The Maximum Mean Discrepancy method (MMD) Cui et al. [2020] trains by first minimizing the NLL loss of an HNN model, and then re-train the model to minimize the MMD between the model prediction and the same training set at the second phase. The method introduced in section 3.2 of Chung et al. [2021] introduces a combined calibration loss (CCL) with an average calibration loss and a sharpness regularization term, and the model is trained by minimizing it. Apart from these feed-forward structures, there is also a Deep Gaussian Process model (DGP) Damianou and Lawrence [2013] which assumes that the data are generated from a Gaussian process, and is trained by the doubly stochastic variational inference algorithm Salimbeni and Deisenroth [2017]. The DGP model is of high training cost, we thus only test it on small datasets. We also run experiments on post-hoc calibration methods, such as isotonic regression (ISR) suggested by Kuleshov et al. [2018]. In our experiments, we evaluate the post-hoc ISR method by attaching it
to a pre-trained HNN model. We also implement another post-hoc method named Model Agnostic Quantile Regression (MAQR) method given in section 3.1 of Chung et al. [2021]. Their algorithm is similar to ours in terms of training a regression model first, but different in two aspects: first, MAQR uses the same dataset to do both regression training and quantile calibration; second, MAQR obtains a quantile result at the second phase via training a new quantile regression model, which in our implementation is a neural network. For the algorithms coming from the conformal prediction society, we test the Conformalized Quantile Regression (CQR) method introduced by Romano et al. [2019] and the Orthogonal Quantile Regression (OQR) approach from Feldman et al. [2021]. The CQR algorithm also takes a training step followed by a calibration step, with the training step learning the target quantiles by minimizing the pinball loss and the calibration step further calibrating the quantile predictions by conformal prediction. Based on the CQR paradigm, the OQR algorithm further improves conditional coverage by introducing an additional regularization term to the vanilla pinball loss used for quantile regression. The regularization term encourages the independence between interval length and the violation (falling out of the predicted interval) event of the response variable.
E.5 Experiment details on the 8 UCI datasets
We have run extensive experiments to compare our methods against multiple state-of-the-art benchmark models. The full results are listed in Table 4. A brief review of all the benchmarks has been provided in Section E.4, and realizations of algorithms NRC-RP and NRC-Cov are explained in Appendix E.3.
A short summary of the 8 UCI datasets is listed in 3. For all the network structures used in Section 4 we fix the size of the hidden layers to [20, 20] (and [10] for the DGP model). For our dimension reduction algorithms, the target dimension that we reduce to is set to 4. The rest hyperparameters (including the learning rate, minibatch size, kernel width, etc) are pre-determined by running a grid search. The learning rate is searched within  $[10^{-2}, 5 \times 10^{-3}, 10^{-3}]$ , the minibatch size is searched within [10, 64, 128], and the rest hyperparameters exclusive to each model are searched in the same fashion. For each possible combination of model and dataset, we save its optimal hyperparameters into separate configuration files.
During the main experiment, for each combination of model and dataset, we load its optimal hyperparameter configurations, on which we repeat the whole training process 5 times. Each time we randomly split out  $10\%$  of the whole sample as the testing set. On the remaining set, for our NRC algorithms we additionally separate out  $30\%$  for recalibration, and for the rest algorithms, no additional partitioning is required. The rest data is then fed into the whole train-validation loop, and we set up an early stopping scheme with a patience count of 20. That is, if for 20 epochs no decrease in loss is observed on the validation set, then the training is early stopped.
Table 3: Descriptions of Datasets. Each row gives the number of examples contained in the corresponding dataset and the number of features in the input variable.
(a)
Dataset	# Examples	# Features
Boston	506	13
Concrete	1030	8
Energy	768	8
Kin8nm	8192	8
Naval	11934	17
Power	9568	4
Wine	4898	11
Yacht	308	6
(b)
Dataset	# Examples	# Features
Meps_19	15785	139
Meps_20	17541	139
Meps_21	15656	139
Bike-Sharing	17379	11
Wine-Red	1599	11
Auto-MPG	392	6
E.6 Experiment details on the Bike-Sharing dataset
The Bike-Sharing dataset is a time series dataset that records the number of bike rentals as well as the value of 11 related features at each hour. As briefly described in Section 4, for this experiment we design the LSTM-HNN and LSTM-NRC models, each having 2 layers, 40 hidden neurons for each layer, and a sliding window of size 5. For the LSTM-NRC model, in the recalibration step we are required to calculate (as part of the whole kernel matrix calculation) the distance between to input variables, and we retrieve the input vector by concatenating all the features within the history window (into a 55-dimension vector). As the dimension is high, we adopt dimension-reduction techniques and use random projection to map the concatenated feature vector to a 4-dimensional space.
Most experiment details, in terms of hyperparameter selection, early stopping criteria, etc, as similar to the settings described in Appendix E.5. One difference lies in the splitting of the dataset. Due to the temporal property, we are only allowed to predict "the future" from "the past", thus we split the starting  $80\%$  of time stamps for training (and further split for recalibration, if required), and the ending  $20\%$  of time stamps for testing.
E.7 Robustness to Covariate Shift
Covariate shift [Quinonero-Candela et al., 2008] describes a frequently observed phenomenon that the distribution of the covariates (also called the independent variable, feature variable, etc) that differ between the training and testing datasets. In the presence of covariate shift, a calibration model attained from the training set that only achieves average calibration (i.e., naive predictions of quantiles) may no longer suit well for the testing dataset, while an individually calibrated model maintains its optimality in terms of all the metrics mentioned above.
Following the experiment design in Chen et al. [2016] and Wen et al. [2014], we experiment on both real datasets and half-synthetic ones (all of which are available on Dua and Graff [2017]). The experiment design is almost the same as in Section 4, except for a difference in the processing of testing sets. The experiment results on a subset of benchmarks are presented in Table 5, which further validates the robustness of our algorithm to covariate shift, especially from the dominating performance on the half-synthetic datasets. A few remarks on the datasets: the Wine dataset and Auto-MPG dataset both induce a natural interpretation of covariate shift, with the training set and the testing set having different colors in the experiment on the Wine dataset, and different origin cities on the Auto-MPG dataset; and on the rest two datasets we synthetically simulate a covariate shift for the testing set, as recommended in Chen et al. [2016]. To be specific, we introduce a covariate shift into the testing set by following steps:
1. Randomly split out  $10\%$  of the whole dataset as the "testing pool".
2. On the training set  $X_{T}$ , Calculate empirical mean and variance  $\hat{\mu} = \overline{X_T},\hat{\Sigma} = \mathrm{Cov}(X_T)$
3. Sample  $X_{\mathrm{seed}} \sim \mathcal{N}(\hat{\mu}, \hat{\Sigma})$ .
4. Resample (with replacement) 1000 examples from the testing pool, following density  $\mathcal{N}(X_{\mathrm{seed}},0.3\hat{\Sigma})$
Table 5: Experiments on covariate shift datasets. The first three rows are real datasets, while the last two are half-synthetic ones. For the Auto-MPG dataset, models are always trained on data samples from city 1, and tested on dataset of either city 2 or city 3, denoted by "MPG2" and "MPG3" respectively. Our algorithm has an outstanding performance, especially so on the two half-synthetic datasets.
Dataset	Metric	HNN	MCDrop	DeepEnsemble	ISR	DGP	NRC
Wine	MACE	0.32±0.1	0.22±0.07	0.32±0.06	0.31±0.09	0.26±0.06	0.2±0.03
AGCE	0.33±0.1	0.23±0.07	0.34±0.06	0.33±0.08	0.28±0.05	0.23±0.03
CheckScore	0.51±0.2	0.29±0.06	0.43±0.1	0.48±0.2	0.31±0.06	0.31±0.09
MPG12	MACE	0.065±0.03	0.12±0.01	0.07±0.03	0.059±0.02	0.17±0.01	0.069±0.008
AGCE	0.21±0.05	0.22±0.04	0.19±0.03	0.21±0.06	0.30±0.05	0.29±0.06
CheckScore	1.2±0.2	1.5±0.2	1.1±0.04	1.1±0.06	1.7±0.2	1.2±0.05
MPG13	MACE	0.1±0.02	0.16±0.02	0.11±0.02	0.1±0.03	0.23±0.03	0.08±0.03
AGCE	0.2±0.009	0.2±0.03	0.2±0.02	0.24±0.06	0.37±0.07	0.26±0.1
CheckScore	1.1±0.04	1.6±0.3	1.2±0.08	1±0.05	1.4±0.07	0.9±0.06
Concrete	MACE	0.1±0.04	0.17±0.09	0.12±0.03	0.059±0.01	0.19±0.02	0.05±0.008
AGCE	0.13±0.04	0.18±0.08	0.15±0.04	0.093±0.02	0.21±0.01	0.07±0.01
CheckScore	4.11±0.2	4.8±0.8	4.3±0.2	4±0.3	2.3±0.27	2.5±0.1
Boston	MACE	0.24±0.08	0.33±0.1	0.2±0.01	0.11±0.02	0.08±0.03	0.05±0.02
AGCE	0.26±0.07	0.34±0.1	0.21±0.01	0.14±0.02	0.1±0.03	0.075±0.02
CheckScore	2.1±0.7	3.2±1.2	1.5±0.05	1.1±0.1	0.71±0.3	0.61±0.05
F Some Basic Probability Results
In this section, we present several well-known probability theory results.
Lemma 1 (Bernstein's Inequality). Let  $\xi_1, \ldots, \xi_n$  be  $N$  independent random variables. Suppose  $|\xi_i| \leq M_0$  almost surely. Then  $\forall t > 0$ ,
$$
\mathbb {P} \left(\left| \frac {1}{N} \sum_ {i = 1} ^ {N} (\xi_ {i} - \mathbb {E} [ \xi_ {i} ]) \right| \geq t\right) \leq \exp \left(- \frac {\frac {1}{2} N t ^ {2}}{\sum_ {i = 1} ^ {N} \frac {1}{N} \operatorname {V a r} (\xi_ {i}) + \frac {1}{3} M _ {0} t}\right).
$$
Lemma 2 (Hoeffding's Inequality). Let  $\xi_1, \ldots, \xi_n$  be  $N$  independent random variables. Suppose  $a_i \leq \xi_i \leq b_i$  almost surely. Then  $\forall t > 0$ ,
$$
\mathbb {P} \left(\sum_ {i = 1} ^ {N} \left(\xi_ {i} - \mathbb {E} [ \xi_ {i} ]\right) \geq t\right) \leq \exp \left(- \frac {2 t ^ {2}}{\sum_ {i = 1} ^ {N} \left(b _ {i} - a _ {i}\right) ^ {2}}\right),
$$
$$
\mathbb {P} \left(\sum_ {i = 1} ^ {N} \left(\mathbb {E} \left[ \xi_ {i} \right] - \xi_ {i}\right) \geq t\right) \leq \exp \left(- \frac {2 t ^ {2}}{\sum_ {i = 1} ^ {N} \left(b _ {i} - a _ {i}\right) ^ {2}}\right).
$$
Lemma 3 (Poisson Approximation). Let  $S_{n} = \sum_{k=1}^{n} \xi_{n,k}$ , where for each  $n$  the random variables  $\xi_{n,k}, k \in [n]$  are mutually independent, each taking value in the set of non-negative integers. Suppose that  $p_{n,k} = \mathbb{P}(\xi_{n,k} = 1)$  and  $\epsilon_{n,k} = \mathbb{P}(\xi_{n,k} \geq 2)$  are such that as  $n \to \infty$ ,
(a)  $\sum_{k = 1}^{n}p_{n,k}\to \lambda <  \infty$
(b)  $\max_{k\in [n]}\{p_{n,k}\} \to 0$
(c)  $\sum_{k = 1}^{n}\epsilon_{n,k}\to 0$
Then,  $S_{n}\xrightarrow{\mathcal{D}}\mathrm{Poisson}(\lambda)$  of a Poisson distribution with parameter  $\lambda$  as  $n\to \infty$
G Proofs in Section 2
Proof of Proposition 1. From the fact that  $Y = U + \hat{f}(X)$ , one can replace the  $Y$  term in each probability with  $U + \hat{f}(X)$ , proving that the LHS and RHS probabilities are identical.
Proof of Proposition 2. By Lemma 6, we know that
$$
Q_{\tau}(x)\in \operatorname *{arg  min}_{u\in \mathbb{R}}\mathbb{E}\bigl[l_{\tau}(u,U)|X = x\bigr ].
$$
Since  $l_{\tau}(u_1, u_2) = l_{\tau}(u_1 + c, u_2 + c), \forall c \in \mathbb{R}$ , we have
$$
Q_{\tau}(x)\in \operatorname *{arg  min}_{u\in \mathbb{R}}\mathbb{E}\bigl[l_{\tau}(u + \hat{f} (X),U + \hat{f} (X))|X = x\bigr],
$$
H Proofs in Section 3.1
We start the analysis on a fixed point  $x \in \mathcal{X}$ . We first analyze the case where  $n(x) > 0$ , i.e.  $\exists i \in [n], \| X_i - x \| \leq h$ . We denote those  $\| X_i - x \| \leq h$  by  $\{X_{i_k}\}_{k=1}^{n(x)}$ . We denote  $F_{U|X_{i_k}}$  by  $F_k$ . We denote the average distribution of  $F_k$  by  $\bar{F} = \sum_{k=1}^{n(x)} \frac{1}{n(x)} F_k$ , and denote its  $\tau^{\text{-th}}$  quantile by  $Q_{\tau}(\bar{F})$ . We abbreviate the estimation  $\hat{Q}_{\tau}^{\text{SNQ}}$  obtained by Algorithm 1 to be  $\hat{Q}_{\tau}$  in this section. To bound the difference  $|\hat{Q}_{\tau}(x) - Q_{\tau}(U|X = x)|$ , we factorize it into two terms, as an analogy to the bias-variance factorization in nonparametric analysis:
$$
\left| \hat {Q} _ {\tau} (x) - Q _ {\tau} (U | X = x) \right| \leq \left| Q _ {\tau} (\bar {F}) - Q _ {\tau} (U | X = x) \right| + \left| \hat {Q} _ {\tau} (x) - Q _ {\tau} (\bar {F}) \right|,
$$
where the former term is called the bias term and the latter the variance term. We now deal with them one by one.
Lemma 4. Under Assumption 1 (a) with Lipschitz constant  $L$ , we have
$$
\left| Q _ {\tau} (\bar {F}) - Q _ {\tau} (U | X = x) \right| \leq L h.
$$
Proof of Lemma 4. By Assumption 1 (a),  $\forall k\in [n(x)],$
$$
\left| Q _ {\tau} \left(U | X = X _ {i _ {k}}\right) - Q _ {\tau} \left(U | X = x\right) \right| \leq L h,
$$
$$
\begin{array}{l} F _ {k} \left(\left(Q _ {\tau} (U | X = x) + L h\right) ^ {+}\right) = F _ {k} \left(Q _ {\tau} (U | X = x) + L h\right) \geq \tau , \\ F _ {k} \left(\left(Q _ {\tau} (U | X = x) + L h\right) ^ {-}\right) \leq \tau . \\ \end{array}
$$
$$
\begin{array}{l} \bar {F} \Big (\big (Q _ {\tau} (U | X = x) + L h \big) ^ {+} \Big) = \bar {F} \big (Q _ {\tau} (U | X = x) + L h \big) = \sum_ {k = 1} ^ {n (x)} \frac {1}{n (x)} F _ {k} \big (Q _ {\tau} (U | X = x) + L h \big) \geq \tau , \\ \bar {F} \left(\left(Q _ {\tau} (U | X = x) + L h\right) ^ {-}\right) = \sum_ {k = 1} ^ {n (x)} \frac {1}{n (x)} F _ {k} \left(\left(Q _ {\tau} (U | X = x) + L h\right) ^ {-}\right) \leq \tau , \\ \end{array}
$$
which justifies that
Lemma 4 controls the bias term. For the variance term, we need a more careful analysis. We start with inspecting the properties of the expected pinball loss function. We will show that under Assumption 1 (b) and (c), the empirical risk minimization solution obtained at the second step of Algorithm 1 is actually close to the true quantile of  $\overline{F}$  by two steps:
Step 1. The empirical risk minimizer  $\hat{Q}_{\tau}(x)$  concentrates around its corresponding population risk's unique minimizer for sufficiently large  $n(x) \geq C(\underline{p}, \underline{r}, M)$ , where the minimizer is denoted by  $u^{*}$ .
Step 2. The unique population risk minimizer  $u^{*}$  in Step 1 is identical to the population risk minimizer of  $\mathbb{E}_{U\sim \bar{F}}[l_{\tau}(u,U)]$ .
We prove the latter Step 2 first due to its simplicity. The following Lemma 5 proves Step 2.
Lemma 5. We define the population risk stated in Step 1 by  $\bar{l}_{\tau}(u) \coloneqq \mathbb{E}_{U_k \sim F_k} \left[ \sum_{k=1}^{n(x)} \frac{1}{n(x)} l_{\tau}(u, U_k) \right]$ . Suppose Assumption 1 hold for some  $L, \underline{p}, \underline{r}, M$ . Then