SlowGuess/NeurIPS_2023_OCR huggingface dataset:
on a smaller scale than  $(1 / \sqrt{N_L} - 1 / \sqrt{N_L + N_{\mathrm{FB}}})$ . In general, we assume that  $R(f_{\theta |(\mathcal{S}_L \cup \mathcal{S}_T)}^*) \lesssim R(f_{\theta |(\mathcal{S}_L \cup \mathcal{S}_{\mathrm{FB}})}^*) \lesssim R(f_{\theta | \mathcal{S}_L}^*)$  since more labeled samples can benefit learning (we do not need to assume a strictly monotonic case for the sake of this analysis). For most AL strategies, the difference between the expected empirical risk and the optimal risks given the learning set size is on a higher order of dependency on  $n$  than the learning bound itself. If we ignore the higher order terms, we can simplify the results as shown in (26). Then, the term and more importantly the change in the learning objective that is related to the assumed feedback  $\mathcal{S}_{\mathrm{FB}}$  is only dependent on  $N_{\mathrm{FB}}$  through  $\mathcal{O}(1 / \sqrt{N_L + N_{\mathrm{FB}}})$ .
The testing objective. The relation in (27) can be further analyzed by taking a probabilistic view. If we assume the risks are bounded in the third moment, w.l.o.g., the two risk-difference terms can both be generalized to a slightly more specific high-probability confidence interval [4, 9, 28] than the plain central limit theorem result itself: with probability of at least  $1 - \alpha$ , we have
$$
\begin{array}{l} \left| \left| \tilde {R} _ {T} \left(\left\{Q _ {1}, \dots , Q _ {T} \right\}\right) - \tilde {R} _ {T} \left(\left\{Q _ {1}, \dots , Q _ {T} \right\} \backslash \mathcal {S} _ {\mathrm {F B}}\right) \right| \right. \\ \leq 2 \left[ F _ {N _ {T}} ^ {- 1} \left(1 - \frac {\alpha}{2}\right) \frac {\tilde {\sigma} _ {N _ {T}}}{\sqrt {N _ {T}}} - F _ {N _ {T} - N _ {\mathrm {F B}}} ^ {- 1} \left(1 - \frac {\alpha}{2}\right) \frac {\tilde {\sigma} _ {N _ {T} - N _ {\mathrm {F B}}}}{\sqrt {N _ {T} - N _ {\mathrm {F B}}}} \right] \tag {30} \\ \end{array}
$$
$$
\left| \left| \tilde {R} _ {T} \left(\left\{Q _ {1}, \dots , Q _ {T} \right\}\right) - R \right| \right| \leq 2 \left[ F _ {N _ {T}} ^ {- 1} \left(1 - \frac {\alpha}{2}\right) \frac {\tilde {\sigma} _ {N _ {T}}}{\sqrt {N _ {T}}} \right] \tag {31}
$$
where  $F^{-1}$  is the inverse cumulative distribution function of the Student-  $t$  distribution and  $\tilde{\sigma}^2$  is the empirical variance. For the active feedback analysis, we only care about how  $N_{\mathrm{FB}}$  affects the testing objective, thus also obtaining an  $\mathcal{O}(1 / \sqrt{N_T - N_{\mathrm{FB}}})$  dependency.
The detailed balancing between the two objectives (I) and (II) requires specific knowledge about the constants involved in the bounds. However, if we only focus on terms involving  $N_{\mathrm{FB}}$ , both dependencies on the sample numbers are on the  $1 / \sqrt{n}$  level, making it possible to be balanced by a constant factor  $C$ . Combining these results, we get the  $N_{\mathrm{FB}}$  term as  $\mathcal{O}(1 / \sqrt{N_L + N_{\mathrm{FB}}} + \mathcal{O}(1 / \sqrt{N_T - N_{\mathrm{FB}}})$  (absorbing  $\mathcal{O}(1)$  terms that do not depend on  $N_{\mathrm{FB}}$ ). The next key factor is that throughout the entire ATL process, we either keep  $N_{\mathrm{FB}}$  fixed or only change it at a linear rate (flexible  $N_{\mathrm{FB}}$  should be an interesting future direction). Combining with our previous assumption that  $N_L$  and  $N_T$  are of similar magnitudes, we know that an optimal balance could be achieved between (I) and (II) to minimize the joint learning-testing objective given in (8).
C Additional Experiment Results
In this section, we present the detailed experimental settings and additional experimental results.
C.1 Synthetic Experiment
Figure 4 shows how the proposed feedback strategy helps to encourage exploration. The background color shows the model's predictive distribution. For each quiz, we display all the training samples obtained by an active learner (red and blue circles representing 2 classes) but only the current quiz (triangles) and feedback samples (squares, then added to circles in later AL rounds) from the active tester to make the visualization clear. Figure 4a shows that ATL selects a feedback sample in the bottom right corner because it is not included in the current knowledge base of the AL model. The AL model predicts it poorly in the quiz. In Figure 4b, we see that the AL model is guided by the feedback samples and starts to explore the bottom right corner. Once the AL model collects samples from the bottom right area, ATL stops to provide guidance for that region. In this way, the proposed feedback strategy manages to find the minority cluster at the other corner shortly as shown in Figure 4c.
In Figure 4d, to further demonstrate the effectiveness of the proposed feedback strategy, we compare it with the feedback samples selected using two other baselines: random feedback (in Figure 4f) and AL based feedback (in Figure 4e), when the samples at the bottom right corner are first discovered. First, we notice that those data samples are found by the AL model rather than through the feedback strategies. As a result, it happens at a much later quiz time compared with ATL. Therefore, they result in a less efficient learning process. Second, we observe that when an AL model discovers a
(a) ATL Feedback Quiz 2
(b) ATL Feedback Quiz 3
(c) ATL Feedback Quiz 4
(d) ATL Feedback Quiz 10
(e) AL Feedback Quiz 9
(f) Random Feedback Quiz 10
Figure 4: Effectiveness of active feedback for improving model training
Table 7: Test classification accuracy
Feedback Strategy	Quiz 2	Quiz 3	Quiz 4	Quiz 9	Quiz 10
ATL	0.75	0.87	0.90	0.96	0.97
AL	0.75	0.83	0.84	0.84	0.88
Random	0.74	0.78	0.81	0.83	0.91
ATL - NF	0.74	0.76	0.79	0.81	0.86
new area to learn, both baseline feedback strategies fail to provide support even though they have some test samples (i.e., the red point at the bottom right corner) available in the interesting region. Last, we can see that at around quiz 10, the AL model with the proposed ALT converges to a better decision boundary that captures the entire data distribution while the two other baselines both fail to correctly discover the predictive distribution at the two corners. As a result, ALT leads to a more accurate model (shown in Table 7) while maintaining lower estimation error in the end.
C.2 Real-world Experiments
In this section, we provide more results on the real-world datasets including MNIST, FashionMNIST and CIFAR10, mainly to demonstrate different feedback approaches and how we can implement early stopping in ATL.
C.2.1 Experimental Settings
In all experiments, we use a CNN model and standard data transformation for each dataset. In each AL training round, we run 10 epochs for MNIST and FashionMNIST and 50 epochs for CIFAR10. A threshold of  $1 \times 10^{-5}$  is used for probability outputs as required for the proposal  $q(\mathbf{x})$  computation [14] to avoid 0 denominators.
An important detail to note is that for ATL-NF results, we sample 50 additional data points during AL for fair comparison (550 in each round), which is actually very similar to ATL-RF. The results with only 500 data points per round will be shown in the following section C.2.2. Another detail worth mentioning is that although we set the initial budget to be 500 labels and add 500 training
samples plus 100 testing samples in each round, the final total budget is 12,450 on average instead of 12,500 because we allow replacement while sampling.
C.2.2 Random Hold-out Test and Random Feedback
In this section, we discuss the issues with traditional hold-out validation/testing procedures during AL and compare the results using random sampling for both test sample selection and feedback selection with the proposed learning-testing-feedback process.
Table 8: Risk estimation error comparison with random methods
Dataset	AL round Method	4	8	12	16	20
MNIST	Random Test	7.80 ± 13.4	6.61 ± 3.72	5.94 ± 7.25	3.15 ± 5.83	6.18 ± 2.41
Random Test & Feedback	30.4 ± 42.2	16.8 ± 16.1	5.34 ± 6.33	11.8 ± 10.6	5.62 ± 2.73
Random Test & Weighted	71.3 ± 31.9	19.1 ± 16.5	12.3 ± 12.2	10.0 ± 11.7	5.64 ± 1.64
ATL-NF	2.57 ± 1.17	0.79 ± 1.15	0.17 ± 0.15	0.56 ± 0.30	1.32 ± 0.37
Fashion MNIST	Random Test	6.97 ± 11.2	5.29 ± 3.56	10.9 ± 6.55	5.40 ± 2.76	8.56 ± 7.57
Random Test & Feedback	12.7 ± 13.4	15.4 ± 16.0	12.7 ± 6.93	18.8 ± 20.9	32.7 ± 15.4
Random Test & Weighted	6.85 ± 14.8	3.01 ± 11.3	2.80 ± 11.1	4.58 ± 29.0	9.51 ± 9.13
ATL-NF	3.64 ± 1.61	0.67 ± 0.38	0.96 ± 0.16	0.98 ± 0.43	3.04 ± 1.37
CIFAR10	Random Test	20.5 ± 6.50	15.8 ± 10.3	13.0 ± 10.1	9.99 ± 6.58	9.89 ± 9.61
Random Test & Feedback	44.7 ± 36.4	16.4 ± 15.4	31.0 ± 12.8	11.7 ± 8.65	55.3 ± 19.0
Random Test & Weighted	43.5 ± 14.8	14.6 ± 11.3	15.1 ± 11.1	11.2 ± 29.0	48.7 ± 9.13
ATL-NF	8.83 ± 7.79	3.06 ± 5.04	4.95 ± 7.12	7.94 ± 5.22	6.20 ± 5.79
Synthetic	Random Test	4.17 ± 0.78	8.00 ± 0.71	11.2 ± 0.19	5.06 ± 0.10	8.96 ± 0.04
Random Test & Feedback	67.8 ± 2.7	75.0 ± 3.95	35.6 ± 1.02	8.29 ± 1.29	0.51 ± 0.72
Random Test & Weighted	5.26 ± 2.04	14.10 ± 2.51	10.66 ± 1.85	8.71 ± 1.02	9.21 ± 0.07
ATL-NF	3.67 ± 0.11	2.86 ± 0.63	1.48 ± 0.14	1.20 ± 0.02	0.26 ± 0.07
In Table 8, random test is referring to randomly sampling 100 test samples after each 550-sample (additional 50 for fair comparison, same as ATL-NF) AL round and simply averaging the loss over these test samples. Random test & feedback is referring to sampling 100 test samples after each 500-sample AL round and then randomly selecting 50 for feedback. Random test & weighted is referring to the same process but the quizzes are weighted by  $1 / R_{t}$ . From Table 8, we can see that in the small-data regime, random sampling may not provide an accurate estimate of the true risk. However, in later AL rounds, the no feedback case (Random Test) can maintain an unbiased estimate, and we do see that some results are comparable with active risk estimation baselines without the ATL-integrate estimator. This is probably because existing active risk estimation baselines (ARE-quiz, AT-integrate, ASE-integrate) do not consider the biased selection and model change through the AL process. The methods that use surrogate models also suffers from the insufficient training of the surrogate model. However, random testing selection does not work well with the active feedback process. For Random Test & Feedback and Random Test & Weighted, we often see much worse estimation due to the feedback process involved.
C.2.3 ATL with Various AL Strategies
In this section, we show additional ATL estimation error results and active feedback results using two different AL strategies: margin sampling [27] and BALD [12].
From Table 9, we can see that the integrated risk estimation performs similarly regardless of the AL strategy. The relative scales of different risk estimations are also similar to the entropy case in the main paper. From Table 10, we do see some different effects. Depending on the AL strategy, random feedback performs unstably. However, our proposed feedback still mostly outperforms the fair comparison baseline. The study on AL-specific feedback strategies could be an interesting future direction.
C.2.4 Additional Active Feedback Comparisons
In this section, we show a more complete comparison between different feedback approaches. The feedback comparison consists of two parts: (1) baseline comparison including no feedback (ATL-NF), random feedback (ATL-RF), entropy-based feedback (ATL-EN) and (2) ablation study including loss-based feedback (ATL-LF), weighted loss-based feedback (ATL-WL) and the proposed weighted loss plus diversity feedback (ATL).
Table 9: Squared difference between the estimate and the true risk over 20 AL rounds  $\left( {\times {10}^{-3}}\right)$
Dataset	AL strategy	AL round Method	4	8	12	16	20
MNIST	Margin	ARE quiz	16.9	2.30	3.86	6.91	10.1
AT integrate	11.7	18.9	15.3	8.35	2.64
ASE integrate	3.96	3.78	5.04	11.1	17.6
ATL	3.40	3.49	3.94	1.93	2.46
Fashion MNIST	Margin	ARE quiz	16.3	16.7	11.5	11.8	9.92
AT integrate	6.29	5.66	6.98	7.04	2.55
ASE integrate	18.4	12.5	19.9	13.4	5.38
ATL	1.14	0.95	0.87	0.79	0.62
CIFAR10	Margin	ARE quiz	11.1	3.31	1.32	1.24	5.64
AT integrate	3.97	7.05	7.48	6.74	8.33
ASE integrate	9.15	12.8	3.99	9.50	4.04
ATL	2.49	0.87	1.88	0.55	2.96
MNIST	BALD	ARE quiz	16.99	9.16	5.75	6.91	10.1
AT integrate	8.48	35.9	11.7	28.5	19.1
ASE integrate	16.4	5.52	7.88	2.16	4.81
ATL	3.83	4.41	5.24	1.54	2.82
Fashion MNIST	BALD	ARE quiz	11.1	12.2	10.1	13.1	9.92
AT integrate	6.09	4.04	7.33	8.86	8.75
ASE integrate	9.00	10.7	4.73	5.23	5.69
ATL	4.25	3.92	4.50	4.28	4.59
CIFAR10	BALD	ARE quiz	3.04	8.93	3.11	19.4	13.2
AT integrate	12.5	22.3	21.9	15.6	14.9
ASE integrate	2.47	3.94	5.20	12.0	19.7
ATL	1.09	3.82	4.65	7.21	10.6
Table 10: Hold-out test risk using different AL strategies and feedback methods over 20 AL rounds
Dataset	AL strategy	AL round Method	4	8	12	16	20
MNIST	Margin	ATL-NF	0.97	0.61	0.49	0.29	0.14
ATL-RF	0.89	0.52	0.32	0.22	0.14
ATL	0.88	0.53	0.35	0.17	0.14
Fashion MNIST	Margin	ATL-NF	1.30	1.05	0.90	0.82	0.69
ATL-RF	1.23	1.06	0.88	0.73	0.70
ATL	1.14	0.95	0.87	0.79	0.62
CIFAR10	Margin	ATL-NF	1.68	1.57	1.50	1.55	1.37
ATL-RF	1.61	1.58	1.56	1.57	1.34
ATL	1.62	1.55	1.49	1.54	1.34
MNIST	BALD	ATL-NF	1.03	0.75	0.49	0.36	0.29
ATL-RF	0.91	0.58	0.44	0.35	0.25
ATL	0.89	0.59	0.46	0.35	0.25
Fashion MNIST	BALD	ATL-NF	1.18	1.04	0.91	0.89	0.85
ATL-RF	1.40	1.06	0.95	0.88	0.81
ATL	1.19	1.07	0.90	0.83	0.80
CIFAR10	BALD	ATL-NF	1.95	1.92	1.78	1.72	1.72
ATL-RF	1.93	1.91	1.79	1.75	1.73
ATL	1.88	1.80	1.76	1.56	1.57
First, we show the hold-out test risk of the AL model throughout AL using different active feedback approaches as the indicator of the model performance. From Table 11, we see that in most occasions, all active feedback approaches can reduce the test risk compared to ATL-NF.
In Table 12, we show a full comparison of the squared error of risk estimation. All estimation results are based on the proposed ATL estimator  $\tilde{R}$ , where ATL-NF, ATL-RF, ATL-EN serve as baselines, meanwhile ATL-LF and ATL-WL serve as ablation studies since the proposed ATL utilizes the weighted loss as well. We see that all feedback approaches suffer from an increased estimation error, especially in the early stage when the number of test samples available is small. We see that
Table 11: Hold-out test risk using different feedback criteria over 20 AL rounds
Dataset	AL round Method	4	8	12	16	20
MNIST	ATL-NF	0.92 ± 0.06	0.55 ± 0.08	0.46 ± 0.06	0.32 ± 0.04	0.22 ± 0.02
ATL-RF	0.92 ± 0.12	0.54 ± 0.02	0.41 ± 0.05	0.29 ± 0.03	0.21 ± 0.02
ATL-EN	0.90 ± 0.12	0.55 ± 0.06	0.41 ± 0.02	0.34 ± 0.06	0.23 ± 0.03
ATL-LF	0.89 ± 0.10	0.56 ± 0.04	0.41 ± 0.02	0.32 ± 0.07	0.20 ± 0.02
ATL-WL	0.86 ± 0.06	0.53 ± 0.06	0.40 ± 0.05	0.32 ± 0.07	0.22 ± 0.03
ATL	0.88 ± 0.07	0.53 ± 0.04	0.39 ± 0.03	0.26 ± 0.01	0.19 ± 0.03
Fashion MNIST	ATL-NF	0.75 ± 0.03	0.69 ± 0.02	0.61 ± 0.02	0.57 ± 0.04	0.56 ± 0.03
ATL-RF	0.75 ± 0.04	0.68 ± 0.02	0.61 ± 0.01	0.58 ± 0.06	0.56 ± 0.04
ATL-EN	0.76 ± 0.02	0.67 ± 0.05	0.58 ± 0.02	0.59 ± 0.03	0.56 ± 0.02
ATL-LF	0.76 ± 0.04	0.65 ± 0.03	0.63 ± 0.01	0.56 ± 0.02	0.56 ± 0.04
ATL-WL	0.76 ± 0.03	0.65 ± 0.02	0.62 ± 0.01	0.56 ± 0.02	0.53 ± 0.02
ATL	0.74 ± 0.03	0.65 ± 0.04	0.59 ± 0.02	0.56 ± 0.03	0.51 ± 0.01
CIFAR10	ATL-NF	1.91 ± 0.04	1.76 ± 0.05	1.72 ± 0.01	1.66 ± 0.02	1.55 ± 0.03
ATL-RF	1.91 ± 0.03	1.77 ± 0.04	1.69 ± 0.03	1.60 ± 0.04	1.54 ± 0.07
ATL-EN	1.92 ± 0.09	1.76 ± 0.04	1.70 ± 0.03	1.66 ± 0.04	1.54 ± 0.02
ATL-LF	1.94 ± 0.04	1.75 ± 0.03	1.65 ± 0.01	1.59 ± 0.03	1.54 ± 0.01
ATL-WL	1.94 ± 0.04	1.75 ± 0.03	1.63 ± 0.01	1.63 ± 0.03	1.54 ± 0.01
ATL	1.90 ± 0.05	1.76 ± 0.02	1.65 ± 0.03	1.58 ± 0.02	1.53 ± 0.02
Table 12: Squared difference between the estimate and the true risk over 20 AL rounds  $(\times 10^{-3})$
Dataset	AL round Method	4	8	12	16	20
MNIST	ATL-NF	2.57 ± 1.17	0.79 ± 1.15	0.17 ± 0.15	0.56 ± 0.30	1.32 ± 0.37
ATL-RF	26.8 ± 21.4	21.4 ± 17.0	3.54 ± 4.01	5.54 ± 3.21	7.62 ± 4.41
ATL-EN	23.6 ± 24.8	14.0 ± 15.8	13.8 ± 11.7	29.5 ± 21.7	21.8 ± 12.8
ATL-LF	15.6 ± 12.6	42.4 ± 36.9	48.5 ± 25.8	15.7 ± 14.8	10.9 ± 7.44
ATL-WL	16.5 ± 19.4	21.0 ± 24.3	7.36 ± 8.44	11.4 ± 12.7	7.59 ± 4.45
ATL	14.6 ± 22.1	16.9 ± 13.7	3.19 ± 2.63	4.15 ± 3.20	1.87 ± 1.41
Fashion MNIST	ATL-NF	3.64 ± 1.61	0.67 ± 0.38	0.96 ± 0.16	0.98 ± 0.43	3.04 ± 1.37
ATL-RF	10.2 ± 9.30	4.41 ± 3.77	2.19 ± 5.53	5.69 ± 4.52	11.6 ± 7.51
ATL-EN	93.2 ± 23.4	50.2 ± 10.2	78.5 ± 32.4	76.2 ± 59.6	85.8 ± 25.9
ATL-LF	9.36 ± 10.2	27.2 ± 26.0	22.6 ± 28.3	14.6 ± 12.0	11.0 ± 15.2
ATL-WL	8.39 ± 8.97	7.52 ± 6.09	4.89 ± 6.50	7.29 ± 4.45	11.1 ± 7.02
ATL	2.50 ± 2.93	1.94 ± 2.25	1.78 ± 1.07	6.32 ± 5.41	5.03 ± 4.41
CIFAR10	ATL-NF	8.83 ± 7.79	3.06 ± 5.04	4.95 ± 7.12	7.94 ± 5.22	6.20 ± 5.79
ATL-RF	20.6 ± 17.6	19.1 ± 13.7	9.82 ± 8.03	33.6 ± 30.5	24.8 ± 32.4
ATL-EN	30.3 ± 17.0	45.8 ± 24.4	20.3 ± 17.4	36.8 ± 31.7	27.0 ± 27.1
ATL-LF	35.0 ± 27.9	45.8 ± 28.5	20.3 ± 10.1	57.2 ± 33.6	40.5 ± 34.0
ATL-WL	22.7 ± 19.7	25.0 ± 13.2	12.9 ± 21.5	52.2 ± 45.9	28.7 ± 16.3
ATL	11.6 ± 13.4	5.11 ± 3.45	8.81 ± 6.51	11.9 ± 16.7	6.57 ± 6.29
the baseline methods suffer from increased estimation error. However, ATL can usually maintain a similar level of estimation error after 20 AL rounds. For ATL-LF, there is usually a larger variance of the estimation error. The potential reason for the unstable behavior of ATL-LF is that it only selects samples with larger losses in the feedback process. Although the importance mechanism can make up for some of the difference, there is still the potential risk of the estimate being biased. Further combining with the diversity metric, we achieve the best results with ATL.
Concluding from both the risk results and the estimation error results, we show that the proposed feedback approach achieves a good balance in the performance-estimation trade-off. This is because we consider both the loss  $L$  and the importance weight  $q$  in the selection criterion. Overall, ATL achieves a similar model test risk as ATL-LF/ATL-WL, both of which are much better than ATL-NF and ATL-RF. ATL also achieves a much lower estimation error than ATL-RF, ATL-EN, and ATL-LF.
C.2.5 Single Feedback Round Comparison
In previous experiments, we add additional training points for the no feedback case (ATL-NF) to make fair comparison for the model risk. However, if we look at the risk change before and after a single feedback round, the difference is even more obvious as shown in Table 13.
C.2.6 Early Stopping in AL
In this section, we show how the ATL-based risk estimation can be readily used for early stopping in AL. In the above experiments, we observe a steady decrease of the estimated risk most of the
Table 13: Hold-out test risk before and after a specific feedback round
Dataset	AL round Method	4	8	12	16	20
MNIST	ATL-before	0.91 ± 0.09	0.54 ± 0.04	0.41 ± 0.08	0.29 ± 0.02	0.21 ± 0.03
ATL-after	0.88 ± 0.07	0.53 ± 0.04	0.39 ± 0.03	0.26 ± 0.01	0.19 ± 0.03
Fashion	ATL-before	0.77 ± 0.03	0.66 ± 0.03	0.61 ± 0.02	0.57 ± 0.03	0.53 ± 0.03
MNIST	ATL-after	0.74 ± 0.03	0.65 ± 0.04	0.59 ± 0.02	0.56 ± 0.03	0.51 ± 0.01
CIFAR10	ATL-before	1.97 ± 0.07	1.82 ± 0.05	1.70 ± 0.03	1.67 ± 0.03	1.57 ± 0.04
ATL-after	1.90 ± 0.05	1.76 ± 0.02	1.65 ± 0.03	1.58 ± 0.02	1.53 ± 0.02
times. However, we do find the decrease becomes more insignificant near the end of the 20 rounds of learning, especially for the MNIST and Fashion MNIST datasets. We observe that after a certain amount of AL rounds, the risk decrease is significantly small, and the corresponding test accuracy is also stabilized (MNIST around  $94\%$ , Fashion MNIST around  $80\%$ , CIFAR around  $54\%$ ). This gives us the opportunity to apply early stopping in real-life AL applications. We here show the average stopping iteration and model performance (hold-out test accuracy) of the compared methods in Table 14. Following the same threshold value, by augmenting the moving average of active risk estimation given by (10) with stabilized prediction (SP), the combined method can stop at a similar testing accuracy as compared with the SP method, but with much lower variance in test accuracy. Based on the threshold setting, it is also possible to stop AL much earlier, saving the overall labeling budget.
Table 14: Average early stopping iteration and final test accuracy comparison (with variance)
Dataset	Method	Iteration	Variance	Test Accuracy	Variance
MNIST	SP	15	6.8	94.52%	6.0e-5
Combined	11	1.2	94.08%	3.1e-5
Fashion	SP	16	4.4	81.32%	3.7e-5
MNIST	Combined	12.4	1.04	80.12%	2.4e-5
CIFAR10	SP	12	2.8	53.87%	1.4e-4
Combined	12.8	0.16	54.43%	8.9e-5
D Details of Hardware for Experiments
All experiments were run on clusters with either NVIDIA A6000 or NVIDIA A100 graphic cards and Intel Xeon Gold 6150 CPU processors. The runtime of the experiments varies depending on the number of repeat runs, but is usually on the scale of a few hours. For example, to get the 5 runs results of one ATL setting for 20 AL rounds on MNIST or Fashion MNIST may take about 6 to 8 hours. The CIFAR10 experiments may take slightly longer.
E Limitation, Future work, and Social Impact
In this section, we first discuss some limitation of the proposed framework and identify some important future direction. We then discuss some potential social impact of our work.
E.1 Limitation and Future Directions
In this paper, we propose an integrated framework that combines active learning and testing. In the interactive framework, the exchange of training and testing information should be carefully guided. Although the proposed testing selection is statistically unbiased and the active feedback is backed by the high-level analysis, we still have room for improving the specification of methods in applicable settings, which we will introduce here as future directions:
- From the learning perspective, we can improve upon the general setting in this paper. In this paper, we focus on introducing a general framework and working under the agnostic setting. However, using specific AL strategies can potentially provide advantages in certain use cases. There have been works that analyze AL label complexity bounds using either importance weighting mechanism in stream-based settings [5, 8] or other methods in pool-based settings [11].
- Continuing on the results from the feedback size analysis in Section 4.3 and the discussion above, the feedback size is a very important factor in the process, especially if we allow the
size to change during AL. Further investigating the relationship between the sample size and the combined learning-testing objective can potentially improve the framework.
- We also propose the ATL framework under an AL-agnostic assumption. Given specific AL strategies, we might be able to also incorporate the learning or testing proposal in the construction of feedback proposal.
E.2 Social Impact
The proposed ATL framework considers the practical challenges of applying active learning in real-world settings, where both model training and evaluation require labeled data. It is a critical step towards realizing label-efficient learning in practice, which can benefit many critical domains where data annotation is highly costly. To this end, the proposed ATL framework has the potential to fundamentally address the data annotation crisis and further broaden the usage of AI to benefit the entire society.
F Algorithm and Source Code
Here we present the pseudo code for the ATL framework using an algorithm block below: