SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Maria-Florina Balcan, Alina Beygelzimer, and John Langford. Agnostic active learning. Journal of Computer and System Sciences, 75(1):78-89, 2009.
[2] Garrett Beatty, Ethan Kochis, and Michael Bloodgood. The use of unlabeled data versus labeled data for stopping active learning for text classification. In 2019 IEEE 13th International Conference on Semantic Computing (ICSC), pages 287-294. IEEE, 2019.
[3] Paul N Bennett and Vitor R Carvalho. Online stratified sampling: evaluating classifiers at web-scale. In Proceedings of the 19th ACM international conference on Information and knowledge management, pages 1581-1584, 2010.
[4] Andrew C Berry. The accuracy of the gaussian approximation to the sum of independent variates. Transactions of the american mathematical society, 49(1):122-136, 1941.
[5] Alina Beygelzimer, Sanjoy Dasgupta, and John Langford. Importance weighted active learning. In Proceedings of the 26th annual international conference on machine learning, pages 49-56, 2009.
[6] Michael Bloodgood and John Grothendieck. Analysis of stopping active learning based on stabilizing predictions. arXiv preprint arXiv:1504.06329, 2015.
[7] Michael Bloodgood and K Vijay-Shanker. A method for stopping active learning based on stabilizing predictions and the need for user-adjustable stopping. arXiv preprint arXiv:1409.5165, 2014.
[8] Corinna Cortes, Giulia DeSalvo, Claudio Gentile, Mehryar Mohri, and Ningshan Zhang. Adaptive region-based active learning. In International Conference on Machine Learning, pages 2144-2153. PMLR, 2020.
[9] Carl-Gustav Esseen. On the liapunov limit error in the theory of probability. Ark. Mat. Astr. Fys., 28:1-19, 1942.
[10] Sebastian Farquhar, Yarin Gal, and Tom Rainforth. On statistical bias in active learning: How and when to fix it. arXiv preprint arXiv:2101.11665, 2021.
[11] Claudio Gentile, Zhilei Wang, and Tong Zhang. Achieving minimax rates in pool-based batch active learning. In International Conference on Machine Learning, pages 7339-7367. PMLR, 2022.
[12] Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745, 2011.
[13] Namit Katariya, Arun Iyer, and Sunita Sarawagi. Active evaluation of classifiers on large datasets. In 2012 IEEE 12th International Conference on Data Mining, pages 329-338. IEEE, 2012.
[14] Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active testing: Sample-efficient model evaluation. In International Conference on Machine Learning, pages 5753-5763. PMLR, 2021.
[15] Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active surrogate estimators: An active learning approach to label-efficient model evaluation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[16] Anurag Kumar and Bhiksha Raj. Classifier risk estimation under limited labeling resources. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 3–15. Springer, 2018.
[17] Philip M. Long and Hanie Sedghi. Generalization bounds for deep convolutional neural networks. In International Conference on Learning Representations, 2020.
[18] Lutz Prechelt. Automatic early stopping using cross validation: quantifying the criteria. Neural networks, 11(4):761-767, 1998.
[19] Christoph Sawade. Active evaluation of predictive models. Universitätsverlag Potsdam, 2013.
[20] Christoph Sawade, Niels Landwehr, Steffen Bickel, and Tobias Scheffer. Active risk estimation. In ICML, 2010.
[21] Christoph Sawade, Niels Landwehr, and Tobias Scheffer. Active estimation of f-measures. Advances in Neural Information Processing Systems, 23, 2010.
[22] Christoph Sawade, Niels Landwehr, and Tobias Scheffer. Active comparison of prediction models. Advances in neural information processing systems, 25, 2012.
[23] Christoph Sawade, Niels Landwehr, and Tobias Scheffer. Active comparison of prediction models. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012.
[24] Burr Settles. Active learning literature survey. 2009.
[25] Surya T Tokdar and Robert E Kass. Importance sampling: a review. Wiley Interdisciplinary Reviews: Computational Statistics, 2(1):54-60, 2010.
[26] Andreas Vlachos. A stopping criterion for active learning. Computer Speech & Language, 22(3):295-312, 2008.
[27] Dan Wang and Yi Shang. A new active labeling method for deep learning. In 2014 International joint conference on neural networks (IJCNN), pages 112-119. IEEE, 2014.
[28] B. L. Welch and H. W. Peers. On formulae for confidence points based on integrals of weighted likelihoods. Journal of the Royal Statistical Society. Series B (Methodological), 25(2):318-329, 1963.
[29] Emine Yilmaz, Peter Hayes, Raza Habib, Jordan Burgess, and David Barber. Sample efficient model evaluation. arXiv preprint arXiv:2109.12043, 2021.
[30] Tiancheng Yu, Xiyu Zhai, and Suvrit Sra. Near optimal stratified sampling. arXiv preprint arXiv:1906.11289, 2019.
[31] Jingbo Zhu, Huizhen Wang, and Eduard Hovy. Multi-criteria-based strategy to stop active learning for data annotation. In Proceedings of the 22nd International Conference on Computational Linguistics-Volume 1, pages 1129-1136, 2008.
A Summary of Notations 14
B Proof and Additional Analysis of Main Theoretical Results 14
B.1 Proof of Theorem 1 14
Corollary 1. 15
B.2 Proposition 1 16
B.3 Proof of Theorem 2 and Active Feedback Analysis 17
C Additional Experiment Results 18
C.1 Synthetic Experiment 18
C.2 Real-world Experiments 19
C.2.1 Experimental Settings 19
C.2.2 Random Hold-out Test and Random Feedback 20
C.2.3 ATL with Various AL Strategies 20
C.2.4 Additional Active Feedback Comparisons 20
C.2.5 Single Feedback Round Comparison 22
C.2.6 Early Stopping in AL 22
D Details of Hardware for Experiments 23
E Limitation, Future work, and Social Impact 23
E.1 Limitation and Future Directions 23
Specifying AL Strategies 23
Feedback Size Control and Balancing the Learning-Testing Objective 23
Connecting Learning-Testing-Feedback Proposals 24
E.2 Social Impact 24
F Algorithm and Source Code 24
Appendix Overview
Organization. The Appendix is organized as follows. We first provide a summary of notations in Appendix A. Then, we present the details of the theoretical analysis in the main paper in Appendix B. We further provide additional experimental results in Appendix C, including a synthetic visualization for different feedback approaches in Appendix C.1 and more detailed real-world experiments in Appendix C.2. The hardware information is presented in Appendix D. We discuss the limitation, future direction, and social impact of the proposed work in Appendix E. Th algorithm block for the pseudo code and the link to the source code are presented in Appendix F.
A Summary of Notations
Table 6: Summary of key notations with definitions
Notation	Definition
(x,y)∈X×Y	Data points
D;p(x,y)	Data distribution
Lθ	Loss function of model fθ()
R	True risk
R(fθ|D)	True risk evaluated on model f whose parameter θ is learned from dataset D.
SL, SU	Labeled set and unlabeled pool
Qt={xt}nt	The t-th quiz set
q(x), q*(x)	Test sample selection proposal and the optimal proposal
Rq,Rt	Risk estimator indexed by the test proposal q or time step t
R	Integrated risk estimator
Ct, vt	Model confidence of ft and the weight coefficient for time step t in final R
SFB	Active feedback set
NL, NT, NFB	Number of samples in learning, testing and feedback sets
d(·, ·), AL, ε	Diversity metric, diversity norm matrix, small positive value ε to avoid singular issues
qFB(x),η	Feedback proposal, balancing parameter between the proposal-loss term and the diversity term in the feedback proposal
λ	Balancing parameter for the risk estimation in unlabeled-information-combined early stopping criterion
B Proof and Additional Analysis of Main Theoretical Results
B.1 Proof of Theorem 1
Proof. We start by presenting the asymptotic convergence of the active risk estimator and the solution for the optimal testing selection proposal  $q^{*}(\mathbf{x})$ . From [20], we know that using the risk estimator  $\hat{R}_{n,q}$  we would get an unbiased estimate of the true risk  $R$  because it is essentially an importance sampling based estimator. Then from the central limit theorem,  $\hat{R}_{n,q}^{0} = \sum_{i=1}^{n} w^{(i)} l^{(i)}$  and  $W_{n} = \sum_{i=1}^{n} w^{(i)}$  are asymptotically normally distributed with
$$
\sqrt {n} \left(\frac {1}{n} \hat {R} _ {n, q} ^ {0} - R\right) \xrightarrow {n \rightarrow \infty} \mathcal {N} (0, \operatorname {v a r} [ w ^ {(i)} l ^ {(i)} ]) \tag {11}
$$
$$
\sqrt {n} \left(\frac {1}{n} W _ {n} - 1\right) \xrightarrow {n \rightarrow \infty} \mathcal {N} (0, \operatorname {v a r} [ w ^ {(i)} ]) \tag {12}
$$
Then, with the multivariate delta method, we know that if  $Y_{n} = (Y_{n1},\dots,Y_{nk})$  is a sequence and  $\sqrt{n}(Y_n - \mu)\xrightarrow{n\to\infty}\mathcal{N}(0,\Sigma)$ , then
$$
\sqrt {n} \left(g \left(Y _ {n}\right) - g (\mu)\right) \xrightarrow {n \rightarrow \infty} \mathcal {N} \left(0, \bigtriangledown g (y) ^ {\top} \Sigma \bigtriangledown g (y)\right) \tag {13}
$$
Here the function is  $g(x,y) = \frac{x}{y}$  with  $x = \frac{1}{n}\hat{R}_{n,q}^{0}$  and  $y = \frac{1}{n} W_{n}$ . The result is
$$
\sqrt {n} \left(\frac {\frac {1}{n} \hat {R} _ {n , q} ^ {0}}{\frac {1}{n} W _ {n}} - R\right) \xrightarrow {n \rightarrow \infty} \mathcal {N} \left(0, \sigma_ {q} ^ {2}\right) \tag {14}
$$
where  $\sigma_q^2 = \int \frac{p(\mathbf{x})}{q(\mathbf{x})}\left(\int [\mathcal{L}(f(\mathbf{x}),y) - R(f)]^2 p(y|\mathbf{x})dy\right)p(\mathbf{x})\mathrm{d}\mathbf{x}.$
Then, the optimal test proposal is obtained by minimizing  $\sigma_q^2$ . By introducing a Lagrange multiplier  $\beta$  for the constraint  $\int q(\mathbf{x})d\mathbf{x} = 1$ , we have
$$
L (q, \beta) = \sigma_ {q} ^ {2} + \beta \left(\int q (\mathbf {x}) d \mathbf {x} - 1\right) \tag {15}
$$
$$
\frac {\partial L}{\partial q} = - \frac {p (\mathbf {x}) ^ {2} \int [ \mathcal {L} (f (\mathbf {x}) , y) - R (f) ] ^ {2} p (y | \mathbf {x}) d y}{q (\mathbf {x}) ^ {2}} + \beta = 0 \tag {16}
$$
Thus, we have  $q^{*}(\mathbf{x})\propto p(\mathbf{x})\sqrt{\int[\mathcal{L}(f(\mathbf{x}),y) - R(f)]^{2}p(y|\mathbf{x})\mathrm{d}y}$ .
Now, we provide the detailed proof for Theorem 1. As shown in Section 3.4,  $\widehat{\mathbf{R}}$  satisfies
$$
\sqrt {n _ {t}} \left(\widehat {\mathbf {R}} - R \mathbb {1}\right) \sim \mathcal {N} \left( \right.\mathbf {0}, \operatorname {d i a g} \left[ \right. \sigma_ {1} ^ {2},.. \sigma_ {T} ^ {2}\left. \right) ^ {\top} \left. \right]\left. \right) \tag {17}
$$
Next, we apply the multi-variant delta method. Define  $g: \mathbb{R}^T \to \mathbb{R}$ ,  $g(\widehat{\mathbf{R}}) = \sum_{t=1}^{T} v_t \widehat{R}_{\mathcal{Q}_t}$ . Then, we have  $\nabla g = (v_1, \dots, v_t)^\top$ . Given the diagonal covariance matrix, the final variance is:
$$
\begin{array}{l} \sigma_ {T} ^ {2} = (v _ {1},..., v _ {t}) \left( \begin{array}{c c c} \sigma_ {1} ^ {2} & & \\ & ... & \\ & & \sigma_ {T} ^ {2} \end{array} \right) (v _ {1},..., v _ {t}) ^ {\top} \\ = \sum_ {t = 1} ^ {T} \int \frac {p (\mathbf {x})}{q _ {t} (\mathbf {x})} v _ {t} ^ {2} \left(\int [ \mathcal {L} (f _ {T} (\mathbf {x}), y) - R (f _ {T}) ] ^ {2} p (y | \mathbf {x}) d y\right) p (\mathbf {x}) d \mathbf {x} \tag {18} \\ \end{array}
$$
When we perform the "final exam" estimation after gathering all quizzes  $\{\mathcal{Q}_1,\dots,\mathcal{Q}_T\}$ , the other factors including testing proposals are fixed. We analyze the optimal solution for  $v_{t}$  by constructing the Lagrangian objective  $\sigma_T^2 +\gamma (\sum_t v_t - 1)$  (where  $\gamma$  is a Lagrangian multiplier). By taking the derivative w.r.t each  $v_{t}$  along with the Lagrangian, we have
$$
\frac {\partial \left[ \sum_ {t = 1} ^ {T} v _ {t} ^ {2} \left(\sigma_ {t} ^ {2}\right) + \gamma \left(v _ {t} - 1 / T\right) \right]}{\partial v _ {t}} = 0 \tag {19}
$$
which leads to  $v_{t} = \frac{C_{t}}{\sum_{t=1}^{T} C_{t}}$ .
The Corollary below provides an alternative view of Theorem 1.
Corollary 1. If we do not change individual  $q_{t}$  but still combine all available test samples, then adjusting their importance weight by  $w_{t}^{\prime(i)} = v_{t} \times w_{t}^{(i)}$  gives the optimal estimator.
Proof. In the alternative view, we have:
$$
\widetilde {R} = \frac {\widetilde {R} ^ {0}}{W ^ {\prime}} = \frac {\sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {n _ {t}} v _ {t} w _ {t} ^ {(i)} l _ {t} ^ {(i)}}{\sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {n _ {t}} v _ {t} w _ {t} ^ {(i)}} \tag {20}
$$
where  $w_{i}^{(t)} = \frac{p(\mathbf{x}^{(i)})}{q_{t}(\mathbf{x}^{(i)}})$ . We can view the final estimate  $\widetilde{R}$  as a function of  $\widetilde{R}^0$  and  $W'$  that has the form  $f(X,Y) = \frac{X}{Y}$ . Then we directly analyze the expectation and variance of  $\widetilde{R}$  using the delta method:
First we have
$$
\begin{array}{l} \mathbb {E} (f (X, Y)) = \mathbb {E} [ f (\mu_ {X}, \mu_ {Y}) + f _ {Y} ^ {\prime} (\mu_ {X}, \mu_ {Y}) (X - \mu_ {X}) + f _ {Y} ^ {\prime} (\mu_ {X}, \mu_ {Y}) (Y - \mu_ {Y}) + R ] \\ \approx \mathbb {E} [ f (\mu_ {X}, \mu_ {Y}) ] + \mathbb {E} [ f _ {X} ^ {\prime} (\mu_ {X}, \mu_ {Y}) (X - \mu_ {X}) ] + \mathbb {E} [ f _ {Y} ^ {\prime} (\mu_ {X}, \mu_ {Y}) (Y - \mu_ {Y}) ] \\ = \mathbb {E} [ f (\mu_ {X}, \mu_ {Y}) ] + f _ {X} ^ {\prime} (\mu_ {X}, \mu_ {Y}) \mathbb {E} [ (X - \mu_ {X}) ] + f _ {Y} ^ {\prime} (\mu_ {X}, \mu_ {Y}) \mathbb {E} [ (Y - \mu_ {Y}) ] \\ = f \left(\mu_ {X}, \mu_ {Y}\right) \tag {21} \\ \end{array}
$$
where  $\mu_{X} = \mathbb{E}[X]$  and  $\mu_{Y} = \mathbb{E}[Y]$ . Applying (21) on our estimate, and we get:
$$
\mathbb {E} [ \tilde {R} (f _ {T}) ] = \mathbb {E} \left[ \frac {\sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {n} v _ {t} w _ {t} ^ {(i)} l _ {i t}}{\sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {n} v _ {t} w _ {t} ^ {(i)}} \right] = \frac {\sum_ {t = 1} ^ {T} v _ {t} \mathbb {E} [ \tilde {R} _ {t} ^ {0} ]}{\sum_ {t = 1} ^ {T} v _ {t} W _ {n , t}} = R
$$
where we utilize  $\sum_{t=1}^{T} v_t = 1$  and  $\mathbb{E}\left[\frac{\widetilde{R}_t^0}{W_t'}\right] = R$ . For the variance, we have:
$$
\begin{array}{l} \operatorname {V a r} [ f (X, Y) ] = \mathbb {E} [ (f (X, Y) - \mathbb {E} [ f (X, Y) ]) ^ {2} ] \\ \approx \mathbb {E} [ (f (X, Y) - f (\mu_ {X}, \mu_ {Y})) ^ {2} ] \\ \approx \mathbb {E} [ (f (\mu_ {X}, \mu_ {Y}) + f _ {X} ^ {\prime} (\mu_ {X}, \mu_ {Y}) (X - \mu_ {X}) + f _ {Y} ^ {\prime} (\mu_ {X}, \mu_ {Y}) (Y - \mu_ {Y}) - f (\mu_ {X}, \mu_ {Y})) ^ {2} ] \\ = \mathbb {E} \left[ f _ {X} ^ {\prime 2} \left(\mu_ {X}, \mu_ {Y}\right) \left(X - \mu_ {X}\right) ^ {2} + 2 f _ {X} ^ {\prime} \left(\mu_ {X}, \mu_ {Y}\right) \left(X - \mu_ {X}\right) f _ {Y} ^ {\prime} \left(\mu_ {X}, \mu_ {Y}\right) \left(Y - \mu_ {Y}\right) \right. \\ \left. + f _ {Y} ^ {\prime 2} \left(\mu_ {X}, \mu_ {Y}\right) \left(Y - \mu_ {Y}\right) ^ {2} \right] \\ = f _ {X} ^ {\prime 2} \left(\mu_ {X}, \mu_ {Y}\right) \operatorname {V a r} [ X ] + 2 f _ {X} ^ {\prime} \left(\mu_ {X}, \mu_ {Y}\right) f _ {Y} ^ {\prime} \left(\mu_ {X}, \mu_ {Y}\right) \operatorname {C o v} [ X, Y ] + f _ {Y} ^ {\prime 2} \left(\mu_ {X}, \mu_ {Y}\right) \operatorname {V a r} [ Y ] \tag {22} \\ \end{array}
$$
Applying to our estimate leads to
$$
\begin{array}{l} \operatorname {V a r} [ \widetilde {R} ] \approx R ^ {2} \operatorname {V a r} [ W ^ {\prime} ] + \operatorname {V a r} [ \widetilde {R} ^ {0} ] - 2 R \operatorname {C o v} [ W ^ {\prime}, \widetilde {R} ^ {0} ] \\ = R ^ {2} \big (\mathbb {E} [ W ^ {\prime 2} ] - \mathbb {E} ^ {2} [ W ^ {\prime} ] \big) + \big (\mathbb {E} [ (\widetilde {R} ^ {0}) ^ {2} ] - \mathbb {E} ^ {2} [ \widetilde {R} ^ {0} ] \big) - 2 R \big (\mathbb {E} [ W ^ {\prime} \widetilde {R} ^ {0} ] - \mathbb {E} [ \widetilde {R} ^ {0} ] \mathbb {E} [ W ^ {\prime} ] \big) \\ = R ^ {2} \mathbb {E} \left[ W ^ {\prime 2} \right] - 2 R \mathbb {E} \left[ W ^ {\prime} \widetilde {R} ^ {0} \right] + \mathbb {E} \left[ \left(\widetilde {R} ^ {0}\right) ^ {2} \right] \\ = \sum_ {t = 1} ^ {T} \int \frac {p (\mathbf {x})}{q _ {t} (\mathbf {x})} v _ {t} ^ {2} \left(\int [ \mathcal {L} (f _ {T} (\mathbf {x}), y) - R (f _ {T}) ] ^ {2} p (y | \mathbf {x}) d y\right) p (\mathbf {x}) d \mathbf {x} \tag {23} \\ \end{array}
$$
where we utilize  $f(X,Y) = \frac{X}{Y} \rightarrow f_X' = \frac{1}{Y}$ ,  $f_Y' = -\frac{X}{Y^2}$ ,  $\mu_X = R$ ,  $\mu_Y = 1$ . Note that since we assume  $q_t(\mathbf{x})$  are fixed, we have  $\mathbb{E}[W'\widetilde{R}^0] = \mathbb{E}_{p(y|\mathbf{x})}\mathbb{E}_{q_1}\dots \mathbb{E}_{q_T}[\sum_{t=1}^T v_t\sum_{i=1}^n(w_t^{(i)})^2l(\mathbf{x}_t^{(t)})] = \mathbb{E}_{p(y|\mathbf{x})}[\sum_{t=1}^T v_t\mathbb{E}_{q_t}\sum_{i=1}^n(w_t^{(i)})^2l(\mathbf{x}_t^{(t)})]$ .
B.2 Proposition 1
We show two concrete examples for Proposition 1. In each case, the estimated introspective loss is analogous to an uncertainty measure.
- The estimation of 0-1 loss is:
$$
R _ {\theta} = \frac {1}{\left| \mathcal {S} _ {U} \right|} \sum_ {\mathbf {x} \in \mathcal {S} _ {U}} \sum_ {y} \mathbb {1} \left(f _ {\theta} (\mathbf {x}) \neq y\right) p (y | \mathbf {x}; \theta) \tag {24}
$$
which is the sum of the predicted probability of all classes other than the most probable class.
- The estimation of cross-entropy loss is:
$$
R _ {\theta} = \frac {1}{\left| \mathcal {S} _ {U} \right|} \sum_ {\mathbf {x} \in \mathcal {S} _ {U}} \sum_ {y} p (y | \mathbf {x}; \theta) \log (p (y | \mathbf {x}; \theta)) \tag {25}
$$
which is the entropy of the predicted probability.
When we use deep learning models,  $R_{\theta}$  usually largely underestimates the risk over the entire pool. In other works such as [14, 15], the surrogate risk acts in a similar way. For the final risk estimator to be accurate, the introspective risk estimation or the surrogate risk first needs to be accurate, which somewhat beats the purpose of active risk estimation. However, we still try to improve this intermediate step without assuming that we have access to an unrealistically accurate estimation, leading to our proposed  $R_{\theta}$  in Section 3.3.
B.3 Proof of Theorem 2 and Active Feedback Analysis
In Theorem 2, we formalize the combined learning-testing objective as a joint optimization problem with the variable being a subset  $S_{\mathrm{FB}}$  that can be transferred from the testing set  $S_T$  to the learning set  $S_L$ . We define the process of selecting the subset as the "active feedback" process, which connects the learning and testing objectives through a balancing parameter  $C$  given in (8). Performing exact optimization of the subset along with a parameter  $C$  would require more detailed knowledge on the learning model and the AL strategy. We instead provide a general analysis to show that active feedback could indeed provide an optimal solution for the joint optimization problem, where  $C$  scales as  $\mathcal{O}(1)$ . Following our theoretical result, we empirically demonstrate the effectiveness of an intuitive feedback approach in the experimental sections (Section 4.3, Appendix C).
Proof overview. We apply some generic generalization bound (e.g., [17] for CNN or similar models) to the learning objective (I) in the joint optimization problem given by (8), which gives  $\mathcal{O}(1 / \sqrt{n})$ . We then leverage the confidence interval to get a high probability bound for the testing objective (II), which also gives  $\mathcal{O}(1 / \sqrt{n})$  [4, 9, 28]. We use the formalized results on the convergence of the estimate as introduced in [20]. With that, we continue to show that both the learning and testing objectives share the same dependency on  $n$ . These common dependencies on  $n$  give us the foundation to further analyze the feedback process. We offer an intuitive justification of active feedback as follows. The risk estimators are importance weighted estimates of the true risk. The estimate converges to the true risk asymptotically, so fewer samples might hurt the quality of the estimate (due to a large variance), but does not change the fact that the expected average of the estimate is still the true risk. With the confidence interval conversions, we can see that except for the change of constants, the objective's dependency on the number of samples does not change. (This also provides guidance for the feedback proposal later: if we can keep the change of the estimate to the minimum, meanwhile using the samples discarded from the test set to improve the AL model as much as possible, it would be the ideal use of available labels.) Following these high-level ideas as described above, we present the detailed proof below.
Proof. We first break the joint (I) learning-(II) testing objective into two parts and approach each part separately:
$$
\begin{array}{l} R \left(f _ {\theta | \left(\mathcal {S} _ {L} \cup \mathcal {S} _ {\mathrm {F B}}\right)}\right) \leq R _ {C N N} \left(f _ {\theta | \left(\mathcal {S} _ {L} \cup \mathcal {S} _ {\mathrm {F B}}\right)} ^ {*}\right) + \mathcal {O} \left(1 / \sqrt {N _ {L} + N _ {\mathrm {F B}}}\right) \\ \lesssim R _ {C N N} \left(f _ {\theta | \left(S _ {L}\right)} ^ {*}\right) + \mathcal {O} \left(1 / \sqrt {N _ {L} + N _ {\mathrm {F B}}}\right) \tag {26} \\ \end{array}
$$
$$
\begin{array}{l} \| R - \tilde {R} _ {\left(\left\{Q _ {1}, \dots , Q _ {T} \right\} \backslash S _ {\mathrm {F B}}\right)} \| \leq | | \tilde {R} _ {T} \left(\left\{Q _ {1}, \dots , Q _ {T} \right\}\right) - \tilde {R} _ {T} \left(\left\{Q _ {1}, \dots , Q _ {T} \right\} \backslash S _ {\mathrm {F B}}\right) | | \\ + \left\| \tilde {R} _ {T} \left(\left\{Q _ {1}, \dots , Q _ {T} \right\}\right) - R \right\| \tag {27} \\ \end{array}
$$
The learning objective. As mentioned earlier, (26) is a common generalization error bound for CNN or similar models. For example, given a training set  $S_{L}$  with  $N_{L}$  samples, we can draw from the basic bound (e.g., according to Theorem 2.1 in [17]):
$$
R \left(f _ {\theta | \mathcal {S} _ {L}}\right) = \mathbb {E} _ {\mathcal {D}} \left[ l _ {f _ {\theta | \mathcal {S} _ {L}}} (\cdot) \right] \leq \mathbb {E} _ {\mathcal {S} _ {L}} \left[ l _ {f _ {\theta | \mathcal {S} _ {L}}} (\cdot) \right] + C ^ {\prime} \left(\beta^ {\prime} \lambda^ {\prime} \sqrt {\frac {| \theta |}{N _ {L}}} + \sqrt {\frac {\log (1 / \delta)}{N _ {L}}}\right) \tag {28}
$$
with probability of at least  $1 - \delta$ , where  $C'$ ,  $\beta'$ , and  $\lambda'$  are constants and  $|\theta|$  is the total number of trainable parameters in the network. In our case, we do not make further assumptions about the constants and  $|\theta|$  is fixed for evaluating a certain model. Similarly, we can substitute  $N_L$  with  $N_L + N_{\mathrm{FB}}$  and arrive at:
$$
\begin{array}{l} R (f _ {\theta | (\mathcal {S} _ {L} \cup \mathcal {S} _ {\mathrm {F B}})}) = \mathbb {E} _ {\mathcal {D}} [ l _ {f _ {\theta | (\mathcal {S} _ {L} \cup \mathcal {S} _ {\mathrm {F B}})}} (\cdot) ] \leq \mathbb {E} _ {(\mathcal {S} _ {L} \cup \mathcal {S} _ {\mathrm {F B}})} [ l _ {f _ {\theta | (\mathcal {S} _ {L} \cup \mathcal {S} _ {\mathrm {F B}})}} (\cdot) ] \\ + C ^ {\prime} \left(\beta^ {\prime} \lambda^ {\prime} \sqrt {\frac {| \theta |}{N _ {L} + N _ {\mathrm {F B}}}} + \sqrt {\frac {\log (1 / \delta)}{N _ {L} + N _ {\mathrm {F B}}}}\right) \tag {29} \\ \end{array}
$$
We notice that in both (28) and (29), we include the expected loss which is slightly different from the best possible AL model risks  $R(f_{\theta |(\mathcal{S}_L\cup \mathcal{S}_{\mathrm{FB}})}^*)$  and  $R(f_{\theta | \mathcal{S}_L}^*)$ . However, the difference is usually