SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The visualizations for the process of TBAL on CIFAR-10 using the small network (a small CNN network with 2 convolution layers followed by 3 fully connected layers [48]) with energy scores and soft-max scores for confidence functions are shown in Figures 10 and 11 respectively. We note that both the energy scores and soft-max scores do not seem to be calibrated to the correctness of the predicted labels which makes it difficult to identify subsets of unlabeled data where the current hypothesis in each round could have potentially auto-labeled. We also note that the test accuracies of the trained models were around  $50\%$  for most of the rounds of TBAL even though the small network model is not a powerful enough model class for this dataset. Note that CIFAR-10 has 10 classes, so the accuracy of  $50\%$  is much better than random guessing and one would expect to be able to auto-label a reasonably large chunk of the data with such a model if accompanied by a good confidence function. This highlights the important role that the confidence function plays in a TBAL system and more investigation is needed which is left to future work.
Note that, in our auto-labeling implementation we find class specific thresholds. In these figures, we show the histograms of scores for all classes for simplicity. We want to emphasize that the visualization figures in this section are 2D representations (approximation) of the high-dimensional features (either of the penultimate layer or the raw features).
\(N_v\)	Error (%)	Coverage (%)
TBAL	AL+SC	PL+SC	AL	PL	TBAL	AL+SC	PL+SC	AL	PL
100	3.10 ±1.80	0.68 ±0.81	1.45 ±0.73	1.23 ±0.99	2.87 ±0.57	71.43 ±8.86	96.95 ±1.01	92.29 ±3.27	98.52 ±0.16	96.88 ±0.00
400	1.97 ±0.76	0.59 ±0.18	1.19 ±0.53	0.81 ±0.26	2.87 ±0.57	93.99 ±2.39	97.89 ±0.50	91.73 ±2.86	98.44 ±0.00	96.88 ±0.00
800	1.64 ±0.50	0.66 ±0.19	1.21 ±0.41	0.81 ±0.26	2.87 ±0.57	96.26 ±1.33	98.06 ±0.53	92.25 ±2.31	98.44 ±0.00	96.88 ±0.00
1200	1.39 ±0.39	0.67 ±0.19	1.11 ±0.30	0.81 ±0.26	2.87 ±0.57	96.67 ±0.84	98.10 ±0.45	91.98 ±2.20	98.44 ±0.00	96.88 ±0.00
1600	1.33 ±0.30	0.70 ±0.19	1.11 ±0.26	0.81 ±0.26	2.87 ±0.57	97.13 ±0.45	98.16 ±0.44	92.01 ±2.09	98.44 ±0.00	96.88 ±0.00
2000	1.28 ±0.34	0.71 ±0.21	1.07 ±0.25	0.81 ±0.26	2.87 ±0.57	97.15 ±0.54	98.20 ±0.34	91.86 ±2.17	98.44 ±0.00	96.88 ±0.00
Table 10: Unit Ball. Effect of variation of validation data size  $(N_v)$ , without using a UCB (i.e.,  $C_1 = 0$ ) on error estimates. We keep training data size  $N_q$  fixed at 500 and use error threshold  $\epsilon_a = 1\%$ . We report the mean and std. deviation over 10 runs with different random seeds.
\(N_v\)	Error (%)	Coverage (%)
TBAL	AL+SC	PL+SC	AL	PL	TBAL	AL+SC	PL+SC	AL	PL
100	3.10 ±1.80	0.68 ±0.81	1.45 ±0.73	1.23 ±0.99	2.87 ±0.57	71.43 ±8.86	96.95 ±1.01	92.29 ±3.27	98.52 ±0.16	96.88 ±0.00
400	1.65 ±0.65	0.32 ±0.15	0.52 ±0.32	0.81 ±0.26	2.87 ±0.57	93.27 ±2.50	96.91 ±0.99	87.86 ±3.73	98.44 ±0.00	96.88 ±0.00
800	1.08 ±0.47	0.24 ±0.16	0.31 ±0.17	0.81 ±0.26	2.87 ±0.57	96.01 ±1.16	96.31 ±1.36	86.21 ±3.55	98.44 ±0.00	96.88 ±0.00
1200	0.78 ±0.27	0.17 ±0.11	0.18 ±0.14	0.81 ±0.26	2.87 ±0.57	96.82 ±0.84	95.96 ±1.40	84.65 ±4.14	98.44 ±0.00	96.88 ±0.00
1600	0.65 ±0.20	0.13 ±0.08	0.12 ±0.09	0.81 ±0.26	2.87 ±0.57	96.93 ±0.57	95.70 ±1.38	83.76 ±3.93	98.44 ±0.00	96.88 ±0.00
2000	0.54 ±0.16	0.21 ±0.11	0.21 ±0.10	0.81 ±0.26	2.87 ±0.57	97.23 ±0.42	96.36 ±1.13	85.72 ±3.47	98.44 ±0.00	96.88 ±0.00
Table 11: Unit-Ball. Effect of variation of validation data size  $(N_v)$ , without using a UCB (i.e.,  $C_1 = 0.25$ ) on error estimates. We keep training data size  $N_q$  fixed at 500 and use error threshold  $\epsilon_a = 1\%$ . We report the mean and std. deviation over 10 runs with different random seeds.
\( {\mathrm{N}}_{\mathrm{q}} \)	Error (%)	Coverage (%)
TBAL	AL+SC	PL+SC	AL	PL	TBAL	AL+SC	PL+SC	AL	PL
100	\( {1.53} \pm {0.27} \)	\( {1.16} \pm {0.35} \)	\( {1.14} \pm {0.29} \)	\( {16.93} \pm {2.48} \)	\( {12.53} \pm {2.24} \)	\( {75.31} \pm {7.06} \)	\( {31.69} \pm {10.78} \)	\( {51.47} \pm {10.46} \)	\( {99.69} \pm {0.00} \)	\( {99.38} \pm {0.00} \)
200	\( {1.25} \pm {0.21} \)	\( {1.04} \pm {0.25} \)	\( {0.98} \pm {0.17} \)	\( {7.85} \pm {1.69} \)	\( {7.08} \pm {1.82} \)	\( {96.24} \pm {0.88} \)	\( {73.27} \pm {8.68} \)	\( {76.05} \pm {8.19} \)	\( {99.38} \pm {0.00} \)	\( {98.75} \pm {0.00} \)
400	\( {1.20} \pm {0.20} \)	\( {0.94} \pm {0.17} \)	\( {1.03} \pm {0.15} \)	\( {1.81} \pm {0.58} \)	\( {3.25} \pm {0.81} \)	\( {97.70} \pm {0.19} \)	\( {96.48} \pm {1.74} \)	\( {91.08} \pm {2.63} \)	\( {98.75} \pm {0.00} \)	\( {97.50} \pm {0.00} \)
600	\( {1.21} \pm {0.26} \)	\( {0.44} \pm {0.13} \)	\( {1.09} \pm {0.18} \)	\( {0.44} \pm {0.14} \)	\( {2.25} \pm {0.56} \)	\( {97.56} \pm {0.20} \)	\( {98.11} \pm {0.02} \)	\( {93.19} \pm {1.60} \)	\( {98.12} \pm {0.00} \)	\( {96.25} \pm {0.00} \)
800	\( {1.13} \pm {0.20} \)	\( {0.12} \pm {0.05} \)	\( {1.02} \pm {0.18} \)	\( {0.12} \pm {0.05} \)	\( {1.76} \pm {0.50} \)	\( {97.25} \pm {0.19} \)	\( {97.49} \pm {0.01} \)	\( {93.23} \pm {1.01} \)	\( {97.50} \pm {0.00} \)	\( {95.00} \pm {0.00} \)
1000	\( {1.08} \pm {0.19} \)	\( {0.04} \pm {0.02} \)	\( {1.00} \pm {0.20} \)	\( {0.04} \pm {0.02} \)	\( {1.37} \pm {0.34} \)	\( {97.02} \pm {0.25} \)	\( {96.87} \pm {0.01} \)	\( {92.90} \pm {0.79} \)	\( {96.88} \pm {0.00} \)	\( {93.75} \pm {0.00} \)
Table 12: Unit-Ball. Effect of variation of  $N_{q}$ , the maximum number of samples the algorithm can use for training, without using a UCB (i.e.,  $C_1 = 0$ ) on error estimates. We keep validation data size  $N_{v}$  fixed at 4000 and use error threshold  $\epsilon_{a} = 1\%$ . We report the mean and std. deviation over 10 runs with different random seeds.
\( {\mathrm{N}}_{\mathrm{q}} \)	Error (%)	Coverage (%)
TBAL	AL+SC	PL+SC	AL	PL	TBAL	AL+SC	PL+SC	AL	PL
100	0.78 ±0.19	0.29 ±0.25	0.24 ±0.15	16.93 ±2.48	12.53 ±2.24	71.40 ±5.64	19.31 ±8.77	35.90 ±10.66	99.69 ±0.00	99.38 ±0.00
200	0.49 ±0.13	0.16 ±0.09	0.17 ±0.08	7.85 ±1.69	7.08 ±1.82	95.28 ±1.29	59.41 ±10.32	63.77 ±9.15	99.38 ±0.00	98.75 ±0.00
400	0.40 ±0.12	0.15 ±0.07	0.15 ±0.08	1.81 ±0.58	3.25 ±0.81	97.63 ±0.24	92.06 ±2.63	83.64 ±4.76	98.75 ±0.00	97.50 ±0.00
600	0.34 ±0.13	0.12 ±0.05	0.17 ±0.08	0.44 ±0.14	2.25 ±0.56	97.36 ±0.20	97.07 ±0.57	87.92 ±2.68	98.12 ±0.00	96.25 ±0.00
800	0.28 ±0.10	0.07 ±0.03	0.15 ±0.06	0.12 ±0.05	1.76 ±0.50	97.10 ±0.23	97.36 ±0.11	88.96 ±2.14	97.50 ±0.00	95.00 ±0.00
1000	0.25 ±0.10	0.03 ±0.02	0.19 ±0.08	0.04 ±0.02	1.37 ±0.34	96.90 ±0.21	96.84 ±0.03	89.61 ±1.49	96.88 ±0.00	93.75 ±0.00
Table 13: Unit-Ball. Effect of variation of  $N_{q}$ , the maximum number of samples the algorithm can use for training, using a UCB (i.e.,  $C_1 = 0.25$ ) on error estimates. We keep validation data size  $N_v$  fixed at 4000 and use error threshold  $\epsilon_a = 1\%$ . We report the mean and std. deviation over 10 runs with different random seeds.
\(N_v\)	Error (%)	Coverage (%)
TBAL	AL+SC	PL+SC	AL	PL	TBAL	AL+SC	PL+SC	AL	PL
2000	0.0 ±0.0	0.0 ±0.0	0.0 ±0.0	0.0 ±0.0	0.0 ±0.0	0.0 ±0.0	0.0 ±0.0	0.0 ±0.0	0.0 ±0.0	0.0 ±0.0
4000	13.88 ±5.42	13.31 ±10.79	12.55 ±7.23	34.34 ±0.32	31.46 ±0.20	0.72 ±0.55	0.48 ±0.04	0.69 ±0.39	95.00 ±0.00	90.00 ±0.00
6000	14.18 ±0.76	11.52 ±0.82	12.21 ±1.49	34.42 ±0.34	31.46 ±0.20	17.29 ±0.72	8.18 ±1.12	11.81 ±2.70	95.00 ±0.00	90.00 ±0.00
8000	13.97 ±0.14	11.31 ±0.51	12.22 ±0.61	34.42 ±0.34	31.46 ±0.20	36.36 ±1.78	23.40 ±1.15	29.99 ±0.97	95.00 ±0.00	90.00 ±0.00
10000	13.42 ±0.29	11.14 ±0.54	12.12 ±0.40	34.42 ±0.34	31.46 ±0.20	43.79 ±0.93	33.38 ±0.72	39.40 ±0.50	95.00 ±0.00	90.00 ±0.00
Table 14: Tiny-ImageNet. Effect of variation of validation data size  $(N_v)$ , without using a UCB (i.e.,  $C_1 = 0$ ) on error estimates. We keep training data size  $N_q$  fixed at  $10K$  and use error threshold  $\epsilon_a = 10\%$ . We report the mean and std. deviation over 10 runs with different random seeds.
\(N_v\)	Error (%)	Coverage (%)
TBAL	AL+SC	PL+SC	AL	PL	TBAL	AL+SC	PL+SC	AL	PL
2000	0.0 ±0.0	0.0 ±0.0	0.0 ±0.0	0.0 ±0.0	0.0 ±0.0	0.0 ±0.0	0.0 ±0.0	0.0 ±0.0	0.0 ±0.0	0.0 ±0.0
4000	10.50 ±6.01	7.37 ±4.57	6.04 ±1.85	34.20 ±0.32	31.52 ±0.27	0.47 ±0.05	0.48 ±0.06	0.43 ±0.01	95.00 ±0.00	90.00 ±0.00
6000	10.61 ±0.62	7.71 ±1.03	8.53 ±1.70	34.42 ±0.34	31.46 ±0.20	10.16 ±1.10	4.31 ±1.10	7.13 ±1.18	95.00 ±0.00	90.00 ±0.00
8000	9.90 ±0.63	6.80 ±0.77	7.81 ±0.85	34.42 ±0.34	31.46 ±0.20	25.84 ±1.57	14.43 ±2.01	19.23 ±1.43	95.00 ±0.00	90.00 ±0.00
10000	8.97 ±0.36	6.87 ±0.48	7.32 ±0.49	34.42 ±0.34	31.46 ±0.20	32.19 ±1.34	21.96 ±1.35	27.01 ±0.98	95.00 ±0.00	90.00 ±0.00
Table 15: Tiny-ImageNet. Effect of variation of validation data size  $(N_v)$ , using a UCB (i.e.,  $C_1 = 0.25$ ) on error estimates. We keep training data size  $N_q$  fixed at  $10K$  and use error threshold  $\epsilon_a = 10\%$ . We report the mean and std. deviation over 10 runs with different random seeds.
\( {\mathrm{N}}_{\mathrm{q}} \)	Error (%)	Coverage (%)
TBAL	AL+SC	PL+SC	AL	PL	TBAL	AL+SC	PL+SC	AL	PL
2000	14.02 ±0.26	11.49 ±0.80	12.17 ±0.35	52.34 ±1.16	42.94 ±0.26	24.34 ±0.86	14.41 ±1.00	25.13 ±0.58	99.00 ±0.00	98.00 ±0.00
4000	14.10 ±0.77	11.58 ±0.26	11.92 ±0.39	43.14 ±0.33	36.07 ±0.41	34.16 ±1.00	21.84 ±1.36	33.41 ±0.65	98.00 ±0.00	96.00 ±0.00
6000	13.55 ±0.17	11.33 ±0.35	12.31 ±0.16	38.73 ±0.59	33.51 ±0.19	37.80 ±1.05	28.59 ±1.53	38.14 ±0.85	97.00 ±0.00	94.00 ±0.00
8000	13.79 ±0.27	11.72 ±0.32	12.36 ±0.30	36.06 ±0.30	32.33 ±0.32	42.00 ±1.71	32.00 ±1.12	39.64 ±1.07	96.00 ±0.00	92.00 ±0.00
10000	13.26 ±0.35	11.42 ±0.28	12.14 ±0.45	34.27 ±0.21	31.47 ±0.17	43.63 ±0.38	33.80 ±0.82	39.23 ±0.37	95.00 ±0.00	90.00 ±0.00
Table 16: Tiny-ImageNet. Effect of variation of  $N_{q}$ , the maximum number of samples the algorithm can use for training, without using a UCB (i.e.,  $C_1 = 0$ ) on error estimates. We keep validation data size  $N_{v}$  fixed at  $10K$  and use error threshold  $\epsilon_{a} = 10\%$ . We report the mean and std. deviation over 5 runs with different random seeds.
\( {\mathrm{N}}_{\mathrm{q}} \)	Error (%)	Coverage (%)
TBAL	AL+SC	PL+SC	AL	PL	TBAL	AL+SC	PL+SC	AL	PL
2000	\( {9.22} \pm {1.04} \)	\( {7.42} \pm {0.71} \)	\( {7.48} \pm {0.32} \)	\( {52.34} \pm {1.16} \)	\( {42.94} \pm {0.26} \)	\( {17.51} \pm {1.16} \)	\( {9.33} \pm {0.66} \)	\( {17.02} \pm {1.32} \)	\( {99.00} \pm {0.00} \)	\( {98.00} \pm {0.00} \)
4000	\( {9.30} \pm {0.38} \)	\( {6.97} \pm {0.39} \)	\( {7.37} \pm {0.21} \)	\( {43.14} \pm {0.33} \)	\( {36.07} \pm {0.41} \)	\( {25.01} \pm {1.20} \)	\( {14.25} \pm {1.71} \)	\( {22.29} \pm {0.61} \)	\( {98.00} \pm {0.00} \)	\( {96.00} \pm {0.00} \)
6000	\( {9.12} \pm {0.22} \)	\( {6.85} \pm {0.26} \)	\( {7.49} \pm {0.35} \)	\( {38.73} \pm {0.59} \)	\( {33.51} \pm {0.19} \)	\( {28.06} \pm {0.75} \)	\( {17.51} \pm {0.36} \)	\( {25.60} \pm {0.34} \)	\( {97.00} \pm {0.00} \)	\( {94.00} \pm {0.00} \)
8000	\( {9.21} \pm {0.14} \)	\( {7.38} \pm {0.53} \)	\( {7.71} \pm {0.25} \)	\( {36.06} \pm {0.30} \)	\( {32.33} \pm {0.32} \)	\( {30.88} \pm {0.64} \)	\( {21.18} \pm {0.90} \)	\( {27.26} \pm {0.78} \)	\( {96.00} \pm {0.00} \)	\( {92.00} \pm {0.00} \)
10000	\( {8.95} \pm {0.23} \)	\( {7.10} \pm {0.26} \)	\( {7.42} \pm {0.36} \)	\( {34.27} \pm {0.21} \)	\( {31.47} \pm {0.17} \)	\( {32.31} \pm {1.21} \)	\( {22.34} \pm {0.61} \)	\( {27.36} \pm {0.59} \)	\( {95.00} \pm {0.00} \)	\( {90.00} \pm {0.00} \)
Table 17: Tiny-ImageNet. Effect of variation of  $N_{q}$ , the maximum number of samples the algorithm can use for training, using a UCB (i.e.,  $C_1 = 0.25$ ) on error estimates. We keep validation data size  $N_{v}$  fixed at  $10K$  and use error threshold  $\epsilon_{a} = 10\%$ . We report the mean and std. deviation over 5 runs with different random seeds.
Figure 9: Auto-labeling MNIST data using linear classifiers. Validation size = 12k. Maximum training samples = 1600. Each round algorithm queries 160 samples. Coverage of auto-labeling is  $62.9\%$  with  $98.0\%$  accuracy. For the rounds we show, the test error rates are  $21.4\%$ ,  $13.9\%$ ,  $12.5\%$ ,  $10.2\%$ , and  $9.8\%$ , respectively. For four columns (left to right), we show: a) The samples that are labeled by TBAL in this round. b) The embeddings for training samples. c) The embeddings for validation data points. d) The score distribution for the validation dataset.
Figure 10: Auto-labeling CIFAR-10 data using a small network and energy scores. Validation size = 10k. Maximum training samples = 25k. Each round algorithm queries 2500 samples. Coverage of auto-labeling is  $5.3\%$  with  $90.0\%$  accuracy. For the rounds we show, the test error rates are  $56.6\%$ ,  $55.2\%$ ,  $55.6\%$ ,  $53.0\%$ , and  $49.3\%$  respectively. For four columns (left to right), we show: a) The samples that are labeled by TBAL in this round. b) The embeddings for training samples. c) The embeddings for validation data points. d) The score distribution for the validation dataset.
Figure 11: Auto-labeling CIFAR-10 data using a small network and softmax scores. Validation size = 10k. Maximum training samples = 25k. Each round algorithm queries 2500 samples. Coverage of auto-labeling is  $2.3\%$  with  $91.0\%$  accuracy. For the rounds visualized here in each row, the test error rates of the trained classifiers are  $56.6\%$ ,  $59.1\%$ ,  $52.8\%$ ,  $50.5\%$ , and  $51.7\%$  respectively. For four columns (left to right), we show: a) The samples that are labeled by TBAL in this round. b) The embeddings for training samples. c) The embeddings for validation data points. d) The score distribution for the validation dataset.