SlowGuess/ACL_2020_OCR huggingface dataset:
Figure 12: Accuracy of trade-off sampling for annealed and fixed  $\lambda$  using  $U_{\mathrm{soft}}$  for the interrupted learner.
Figure 9: Comparing  $U_{\mathrm{ent}}$  and  $U_{\mathrm{soft}}$  for the interrupted learner.
A.4 Stability of system objective
To provide estimates how stable our approaches are across different randomly initialized weights, we compute the upper and lower quartiles for each sampling strategy across all runs. Figures 10 and 11 show our results for the interrupted learner.
Overall, we observe that user-oriented sampling has lower deviations across different runs for both our aggregation functions  $U_{\mathrm{ent}}$  and  $U_{\mathrm{soft}}$ . One reason for this may be that in contrast to uncertainty sampling, we query instances with highly certain predictions in our user-oriented sampling approach. This leads to sampled instances which are easier to learn resulting in a higher training stability with small data. Comparing the user-oriented against our joint sampling strategies shows that especially in the earlier iterations, our proposed sampling strategies perform better and provide more stable training.
A.5 Further investigation of  $\lambda$
To further validate our findings for an annealed  $\lambda$ , we conduct the same experiments with our novel aggregation function  $U_{\mathrm{soft}}$ . As with  $U_{\mathrm{ent}}$ , we obtain significant improvements for our trade-off sampling strategy (figure 12) for the motivated and interrupted learner, but also a significant decrease for the static and decreasing learner. With respect to the user objective, we do not see any significant differences at all, indicating that  $U_{\mathrm{soft}}$  does not benefit at all from the emphasised user objective in later iterations.
Table 5 (including the previous results for better comparability) shows the results for all learner behaviours and both our aggregation functions  $U_{\mathrm{ent}}$
Figure 10: Upper and lower quartiles for the interrupted learner using  $U_{\mathrm{ent}}$ .
Uent	Usoft
Accuracy	MAE	Accuracy	MAE
STAT	MOT	INT	DEC	STAT	MOT	INT	DEC	STAT	MOT	INT	DEC	STAT	MOT	INT	DEC
tosλ	.333	.346	.347	.314	0.85	0.53	0.48	0.53	.331	.345	.347	.316	0.86	0.64	0.59	0.70
tos	.334	.338	.339	.327	0.98	0.65	0.93	0.75	.345	.336	.338	.327	0.91	0.64	0.62	0.70
comb	.343	.340	.341	.327	0.98	0.63	0.88	0.65	.344	.338	.340	.326	0.93	0.63	0.62	0.66
usr	.338	.331	.334	.328	0.85	0.58	0.65	0.75	.337	.331	.334	.328	0.92	0.63	0.59	0.70
unc	.332	.331	.331	.331	1.17	1.33	1.35	1.72	.336	.336	.336	.335	1.24	1.32	1.31	1.72
rand	.325	.325	.325	.325	1.16	1.22	1.82	1.24	.325	.325	.325	.325	1.16	1.22	1.82	1.24
Table 5: Averaged accuracy and MAE for all strategies (including the annealed  $\lambda$  strategy) for  ${U}_{\text{ent }}$  and  ${U}_{\text{soft }}$  .
and  $U_{\mathrm{soft}}$ . As can be seen, using an annealed  $\lambda$ $(\mathrm{tos}_{\lambda})$  leads to the best results with respect to the user objective for  $U_{\mathrm{ent}}$  but fails to improve the results for  $U_{\mathrm{soft}}$ .
4246
Figure 11: Upper and lower quartiles for the interrupted learner using  $U_{\mathrm{soft}}$ .