SlowGuess/ACL_2020_OCR huggingface dataset:
For a language learning platform, it is likely that motivated learners who continually practice improve their proficiency over time. Less motivated learners or learners who suffer from distractions, interruptions, or frustration, however, may show different paces in their learning speed or even deteriorate in their proficiency. Therefore, we study four prototypical types of learner behavior:
- Static learners (STAT) do not improve their skills over the course of our experiments. Instead, they provide answers constantly at the
same, pre-defined proficiency level. This models learners with a slow progress or with little motivation overall.
Motivated learners (MOT) continually improve their language proficiency throughout our experiments with a fixed step size of  $t_1$  C-tests. That is, we simulate that their proficiency level  $\phi(v)$  increases by one every  $t_1$  iterations.
- Interrupted learners (INT) experience a drop in their proficiency during our experiments. Such cases occur, for example, if a learner has to interrupt their learning process for a longer time. For our simulation, we start with the motivated learner setup, constantly increasing the proficiency every  $t_1$  iterations. However, this learner experiences a sudden increase  $(t_2)$  and drop  $(t_3)$  in the proficiency level by one. After recovering from the drop  $(t_4)$  the proficiency will again increase according to the motivated learner  $(t_5)$ .
- Artificially decreasing learner. (DEC) Finally, our last group of simulated learners displays a constant drop in their proficiency during our simulation. Although such cases rarely occur in the real world, we use this learner to evaluate all sampling strategies in the case of constant drop. Similar to the motivated learner, we start with the highest possible proficiency and decrease it by one every  $t_1$  iterations.
For our experiments, we assume a static learner that remains at proficiency level  $\phi(v) = 3$ . For motivated learners, we set the initial proficiency level to 1 and use a step size of  $t_1 = 8$ , so that they traverse all proficiency levels throughout a single run. For interrupted learners, we also use  $t_1 = 8$  with an additional increase after  $t_2 = 12$ , a drop after  $t_3 = 16$ , and a recovery (increase) after  $t_4 = 20$ . Starting from  $t_5 = 24$ , interrupted learners behave the same as motivated learners.
Like Beinborn (2016), we cannot publish the C-test data due to data privacy reasons, but we provide our code and simulated learner models on GitHub.
We present and discuss our results for  $U_{\mathrm{ent}}$  and  $A$  as defined in section 4. For each strategy we run our experiments ten times with different weight
Henceforth, we use learner to refer to the users of an educational application rather than to a machine learning system.
$^{6}$ https://github.com/UKPLab/ acl2020-empowering-active-learning
initializations and report the averaged scores. For random sampling, we do ten runs with different random seeds for each weight initialization to provide more stable results. We set  $\lambda = 0.5$  for our trade-off sampling strategy.
As our system and user objectives have different scopes (gap-level vs. exercise-level), we quantify both differently. To measure the system objective, we report the accuracy of our model for predicting the individual gap difficulties of the test data after each iteration. As our training data increases by 20 gaps after each iteration, we provide plots for all experiments from the first to the last (40th) iteration. For quantifying the user objective, we evaluate all sampling strategies across all 40 iterations, i.e., how well our sampling strategies were able to satisfy the user's needs after the whole set of exercises. Instead of accuracy, we take the distance-based metric mean absolute error (MAE). As users explicitly query a C-test of a specific proficiency level at each iteration, suggesting a C-test which deviates by two levels from the requested proficiency has a worse impact on the user's learning experience than a C-test which only deviates by one level. For better interpretability, we do not normalize the MAE as we do for our error function err, i.e., a MAE of 1 means that on average, the difficulty of the sampled instances was off by a whole proficiency level from the queried ones.
Since the interrupted learner experiences both a drop and increase in proficiency in a less constant manner than the motivated or decreasing learners, we conduct further analysis of our sampling strategies for the interrupted learner.
System objective. Figure 3 shows the system objective for  $U_{\mathrm{ent}}$  after each iteration. Vertical blue lines indicate increases in the learner's proficiency whereas the vertical yellow line indicates a drop. We observe that although random sampling performs rather well in the early iterations, all our proposed strategies as well as the uncertainty sampling baseline are able to outperform it in the later iterations. Moreover, all proposed strategies perform similar to uncertainty sampling. This is surprising, especially for the user-oriented sampling strategy as it inherently does not optimize the system objective. One reason for this may be the similarity
Figure 3: Accuracy on the test data for  $U_{\mathrm{ent}}$ .
STAT	MOT	INT	DEC
tos	.344	.338	.339	.327
comb	.343	.340	.341	.327
usr	.338	.331	.334	.328
unc	.332	.331	.331	.331
rand	.325	.325	.325	.325
Table 2: Averaged accuracy over all iterations for  ${U}_{\text{ent }}$
of the user-oriented sampling strategy to curriculum learning (Bengio et al., 2009), which opts to organize model training in a meaningful way. As we sample instances the model is most confident in (i.e., have the highest prediction confidence) this leads to instances which are easier to learn and may especially be helpful in low-data scenarios.
To better quantify our results, we compare the averaged accuracy scores across all iterations, shown in table 2 and conduct Wilcoxon signed-rank tests (Wilcoxon, 1992) on the active learning curves for system and model objectives to test for statistical significance. We can observe that for the static, motivated, and interrupted learners both our joint sampling strategies outperform all baselines significantly  $(p < 0.05)$ , but show no significant difference between each other. Only for the decreasing learner all strategies show no significant difference at all. In concordance with our observations for the user-oriented sampling which may benefit from first sampling easy-to-learn instances, jointly optimizing system and user objective seems to benefit from curriculum learning and active learning paradigms.
User objective. Table 3 shows the MAE for all strategies using  $U_{\mathrm{ent}}$ . We can observe that all strate
The system performance of random sampling remains the same for all learner types as it is averaged across all runs.
STAT	MOT	INT	DEC
tos	0.98	0.65	0.93	0.75
comb	0.98	0.63	0.88	0.65
usr	0.85	0.58	0.65	0.75
unc	1.17	1.33	1.35	1.72
rand	1.16	1.22	1.82	1.24
Figure 4: Sampled instances for the interrupted learner.
gies which consider a separate user objective sample instances which significantly better fit the current user proficiency. $^{8}$  Furthermore, the combined sampling approach which puts more emphasis on the user objective outperforms our trade-off sampling for all learner behaviors and even manages to outperform the user-oriented sampling strategy for the decreasing learner.
We further investigate how well our approaches react to changes in the user objective by plotting the mean difficulty  $\phi(v)$  of sampled instances after each step for all our strategies modeling the user objective. As figure 4 shows, all sampling strategies are able to match the queried C-test difficulties well, as they do not deviate much from the queried difficulty (in black).
Adaptive choice of  $\lambda$ . We furthermore investigate how the choice of  $\lambda$  affects our trade-off sampling strategy. As the system predictions may not be very accurate in early iterations, it is reasonable to put more emphasis on the system objective in the beginning, but focus on providing suited C-tests (user objective) in later iterations. We thus define  $\lambda$  as an adaptive function  $\lambda = f(i) = \frac{1}{\sqrt{i}} = i^{-0.5}$  which highly emphasizes the system objective in early stages and anneals with an increasing number
Figure 5: Accuracy of tos for annealed and fixed  $\lambda$ .
Table 3: MAE for  ${U}_{\text{ent }}$
Acc	STAT	MOT	INT	DEC
tosλ	.333	.346	.347	.314
tos	.334	.338	.339	.327
MAE	STAT	MOT	INT	DEC
tosλ	0.85	0.53	0.48	0.53
tos	0.98	0.65	0.93	0.75
Table 4: Averaged accuracy scores and MAE with an annealed  $\lambda$  for  ${U}_{\text{ent }}$  .
of iterations  $i$
Figure 5 shows the system performance of our trade-off sampling strategy averaged across ten different runs. The colored areas show the corresponding upper and lower quartiles. As shown in table 4, we can see that our annealed  $\lambda$  leads to considerable improvements for system and user objective, leading to a significant increase in average accuracy from 0.339 to 0.347 and a decrease in the MAE from 0.93 to 0.48 for the interrupted learner, outperforming all other sampling strategies.
Further findings. We observe similar results for system and user objectives for the other learner types. Investigating the stability of all sampling approaches furthermore shows that our joint optimization strategies perform better and more stable in early iterations.
Due to averaging,  $U_{\mathrm{ent}}$  cannot distinguish between C-tests with only a few highly uncertain gaps and C-tests which have a higher number of less uncertain gaps. However, in preliminary experiments with a different aggregation function which is more robust to C-tests with only a few highly uncertain gaps, we come to similar findings across all sampling strategies and learner types. Detailed results for our other learner behaviors, the stability of our sampling strategies, and the results of our pre
Statistical testing was again conducted using a Wilcoxon signed-rank test for  $p < 0.05$ .
liminalry experiments with a different aggregation function are provided in the paper's appendix.
Limitations. Although our setup with simulated learners may seem artificial compared to an evaluation study with real-world learners, to conduct such a study in an ethical way, we need to ensure that participants are not hurt in their learning process. Thus, strategies which can be evaluated in user studies are limited to those which consider the user objective. In contrast, the use of simulated learners allows us to compare our proposed strategies against common active learning strategies which do not consider the user objective at all.
Another limitation is how to estimate a learner's current proficiency given that we do not know the true difficulty of a C-test. This raises the general question of using relative or absolute difficulties for the selection of suited exercises. In this work, we assumed absolute proficiency levels and implemented according learner behaviors to provide a more controlled environment for our experiments. In the case of absence of any absolute (true) difficulty estimations for C-tests, we see several directions for future work:
a) As a simple baseline, a normalized version of  $\psi(x)$  may be applied on a learner's previously filled-out C-tests. However, this assumes that all C-tests are equally difficult which may lead to unsuited C-tests.
b) Training an additional model for assessing a learner's proficiency given their results on a C-test with the gap-difficulty predictions from our model serving as additional input.
c) Instead of using the absolute difficulty, one may define an optimal error margin as a zone of proximal development (Vygotsky, 1978). This requires an adaptation of the user objective to the relative difficulties of exercises for individual learners, but may be an important step in achieving highly personalized user models without any absolute labels.
In this work, we investigated how we can incorporate user feedback into existing active learning approaches without hurting the user's actual needs. We formalize both system (active learning) and user objectives and propose two novel sampling strategies which aim to maximize both objectives jointly.
We evaluate our sampling strategies for the task of selecting suited C-tests, a type of fill-the-gap exercise, which fit the current proficiency of a human learner. We create simulated learners for five different proficiency levels from real-world data and use them to define different learning behaviors. Our experiments show that both our novel sampling strategies are successfully selecting instances which lead to a better model training while not hurting a learner's progress by selecting too easy or too difficult C-tests. Although system and user objective at first seem counteracting, our experiments indicate that they complement each other as jointly optimizing them outperforms optimizing only one of the goals. Additional experiments with an adaptive  $\lambda$  for our trade-off sampling strategy show that properly balancing system and user objective can lead to considerable improvements in performance for both objectives.
Our findings open up new opportunities for training models on low-resource scenarios with implicitly collected user feedback while jointly serving the user's actual needs. Additional use cases like the training of personalized recommendation models as well as the use of reinforcement learning to find a good trade-off between system and user objective remain to be investigated in future work.
This work has been supported by the German Research Foundation with the ArguAna project (GU 798/20-1) and the Evidence project (GU 798/27-1). We thank the anonymous reviewers for their detailed and helpful comments as well as Edwin Simpson and Yevgeniy Puzikov for the insightful discussions about our work. We especially thank the language center of the Technische Universität Darmstadt for providing us with the data and Dr. Lisa Beinborn for providing us with the code to extract her proposed features.
Lisa Marina Beinborn. 2016. Predicting and Manipulating the Difficulty of Text-Completion Exercises for Language Learning. Ph.D. thesis, Technische Universität Darmstadt, Darmstadt.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, pages 41-48, New York, NY, USA. ACM.
Nhabhat Chaimongkol, Shotiga Pasiphol, and Sirichai Kanjanawasee. 2016. Computerized Adaptive Testing with Reflective Feedback: A Conceptual Framework. Procedia-Social and Behavioral Sciences, 217:806-812.
C. A. Chapelle. 1994. Are C-tests valid measures for L2 vocabulary research? Second Language Research, 10(2):157-187.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL/HLT), pages 4171-4186, Minneapolis, MI, USA.
Meng Fang, Jie Yin, and Dacheng Tao. 2014. Active learning for crowdsourcing using knowledge transfer. In Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence (AAAI), pages 1809-1815, Qubec, Canada.
Mariano Felice and Paula Buttery. 2019. Entropy as a proxy for gap complexity in open cloze tests. In Proceedings of the International Conference Recent Advances in Natural Language Processing (RANLP), pages 323-327, Varna, Bulgaria.
Yang Gao, Christian M Meyer, and Iryna Gurevych. 2018. APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4120-4130, Brussels, Belgium.
Peter Hastings, Simon Hughes, and M. Anne Britt. 2018. Active learning for improving machine learning of student explanatory essays. In Carolyn Penstein Rose, Roberto Martinez-Maldonado, H. Ulrich Hoppe, Rose Luckin, Manolis Mavrikis, Kaska Porayska-Pomsta, Bruce McLaren, and Benedict du Boulay, editors, Artificial Intelligence in Education (AIED), volume 10947 of Lecture Notes in Computer Science, pages 140-153. Cham: Springer.
Andrea Horbach and Alexis Palmer. 2016. Investigating Active Learning for Short-Answer Scoring. In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications (BEA), pages 301-311, San Diego, CA, USA.
Diederik P. Kingma and Jimmy Lei Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR), San Diego, CA, USA.
Christine Klein-Braley and Ulrich Raatz. 1982. Der C-Test: ein neuer Ansatz zur Messung allgemeiner Sprachbeherrschung. AKS-Rundbrief, 4:23-37.
Florian Laws, Christian Scheible, and Hinrich Schütze. 2011. Active Learning with Amazon Mechanical Turk. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1546-1556, Edinburgh, UK.
Ji-Ung Lee, Erik Schwan, and Christian M. Meyer. 2019. Manipulating the difficulty of c-tests. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 360-370, Florence, Italy.
David D. Lewis and William A. Gale. 1994. A sequential algorithm for training text classifiers. In Proceedings of the 17th Annual International ACM Conference on Research and Development in Information Retrieval (SIGIR), pages 3-12, Dublin, Ireland.
Olana Missura and Thomas Gartner. 2011. Predicting dynamic difficulty. In J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 24, pages 2007-2015. Curran Associates.
Avinesh P.V.S and Christian M. Meyer. 2017. Joint optimization of user-desired content in multi-document summaries by learning from user feedback. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1353-1363, Vancouver, Canada.
Ishan Rastogi, Aditya Kanade, and Shirish Shevade. 2018. Active learning for efficient testing of student programs. In Carolyn Penstein Rose, Roberto Martnez-Maldonado, H. Ulrich Hoppe, Rose Luckin, Manolis Mavrikis, Kaska Porayska-Pomsta, Bruce McLaren, and Benedict du Boulay, editors, Artificial Intelligence in Education (AIED), volume 10947 of Lecture Notes in Computer Science, pages 296-300. Cham: Springer.
Tobias Scheffer, Christian Decomain, and Stefan Wrobel. 2001. Active Hidden Markov Models for Information Extraction. In Frank Hoffmann, David J. Hand, Niall Adams, Douglas Fisher, and Gabriela Guimaraes, editors, Advances in Intelligent Data Analysis (IDA), volume 2189 of Lecture Notes in Computer Science, pages 309-318. Berlin/Heidelberg: Springer.
Burr Settles. 2012. Active Learning, volume 18 of Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool.
Burr Settles and Mark Craven. 2008. An Analysis of Active Learning Strategies for Sequence Labeling Tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1070-1079, Honolulu, HI, USA.
Claude Elwood Shannon. 1948. A mathematical theory of communication. Bell system technical journal, 27(3):379-423.
Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y Ng. 2008. Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 254-263, Honolulu, HI, USA.
Huong May Truong. 2016. Integrating learning styles and adaptive e-learning system: Current developments, problems and opportunities. Computers in human behavior, 55(Part B):1185-1193.
Lev Vygotsky. 1978. Mind in society: The development of higher psychological processes. Cambridge: Harvard University Press.
Shiyu Wang, Haiyan Lin, Hua-Hua Chang, and Jeff Douglas. 2016. Hybrid Computerized Adaptive Testing: From Group Sequential Design to Fully Sequential Design. Journal of Educational Measurement, 53(1):45-62.
Frank Wilcoxon. 1992. Individual comparisons by ranking methods. In *Breakthroughs in statistics*, pages 196-202. Springer.
Yi Zheng and Hua-Hua Chang. 2015. On-the-fly assembled multistage adaptive testing. Applied Psychological Measurement, 39(2):104-118.
A.1 Results of  $U_{\mathrm{ent}}$  for other learner types
Figure 6 shows our results for the static, motivated, and artificially decreasing learner. As with the interrupted learner, blue (yellow) vertical lines indicate an increase (drop) in the learner's proficiency. Similar to the results for the interrupted learner, all strategies outperform random sampling in later iterations.
A.2 An outlier-invariant variation of  $U$
Due to averaging,  $U_{\mathrm{ent}}$  cannot distinguish between C-tests with only a few highly uncertain gaps and C-tests which have a higher number of less uncertain gaps. We investigated another aggregation function  $U_{\mathrm{soft}}$  in preliminary experiments, which measures the entropy across all gaps and thus, is more robust to C-tests with only a few highly uncertain gaps.
Formulation. For our second formulation of  $U$ , we use a different aggregation method. Due to the mean,  $U_{\mathrm{ent}}$  is unable to distinguish between C-tests where the system is highly uncertain for only a few gaps and C-tests where all gaps are less, but more equally uncertain. We propose to use the softmax function  $\sigma$  for normalizing  $H(g_i)$  and then to compute the entropy across all gaps  $g_i$ .  $U_{\mathrm{soft}}$
thus considers the distribution of gap-uncertainties and favours C-tests with equally distributed gap-uncertainties over C-tests with only a few highly uncertain gaps.
$$
U _ {\text {s o f t}} (x) = \gamma \left[ - \sum_ {i = 1} ^ {n} \sigma_ {i} \left(H \left(g _ {i}\right)\right) \log \sigma_ {i} \left(H \left(g _ {i}\right)\right) \right] \tag {13}
$$
As the squashing of the individual gap entropy values removes the information about their magnitude, we furthermore scale the resulting value by the normalized mean entropy
$$
\gamma = \frac {1}{n \log n} \sum_ {i = 1} ^ {n} \frac {H \left(g _ {i}\right)}{H _ {\max }} \tag {14}
$$
for all gaps  $g_{i}$  in the C-test.
Results. Figure 11 shows similar tendencies as we already found for  $U_{\mathrm{ent}}$  in section 6. Again, we can observe that random sampling performs better in early iterations, while the other sampling strategies outperform it in latter iterations. Averaging the accuracy across all iterations (table 5) shows that both our joint sampling strategies tos and comb again perform in average better than the other sampling strategies for the static, motivated, and interrupted learners. However, conducting a Wilcoxon signed-rank test with  $p < 0.05$  shows that the active learning curves only significantly differ for the static learner.
Figure 8: Sampled instances for the interrupted learner using  $U_{\mathrm{soft}}$ .
For the user objective (also shown in table 5) we observe that all strategies which include a user objective significantly outperform rand and unc, but there is no clear favorite amongst them. This can also be seen in figure 8 where all strategies manage to sample instances close to the queried difficulty (in black).
Figure 6: Accuracy scores for the static, motivated, and artificially decreasing learners using  $U_{\mathrm{ent}}$ .
Figure 7: Accuracy on the test data for  $U_{\mathrm{soft}}$ .
A.3 Impact of the aggregation function
Figure 9 compares both our aggregation functions  $U_{\mathrm{ent}}$  and  $U_{\mathrm{soft}}$  against each other on the interrupted learner for uncertainty, combined, and trade-off sampling. Although  $U_{\mathrm{ent}}$  and  $U_{\mathrm{soft}}$  differ to some regard, directly comparing both aggregation functions and the respective aggregated scores (cf., table 5 shows that there is no clear favourite between both. Extensive work with respect to both aggregation functions as well as additional aggregation strategies remains to be investigated in future work.