SlowGuess/ACL_2020_OCR huggingface dataset:
Ji-Ung Lee
Christian M. Meyer
{lastname}@ukp.informatik.tu-darmstadt.de
Ubiquitous Knowledge Processing (UKP) Lab
Technische Universität Darmstadt, Germany
https://www.ukp.tu-darmstadt.de
Existing approaches to active learning maximize the system performance by sampling unlabeled instances for annotation that yield the most efficient training. However, when active learning is integrated with an end-user application, this can lead to frustration for participating users, as they spend time labeling instances that they would not otherwise be interested in reading. In this paper, we propose a new active learning approach that jointly optimizes the seemingly counteracting objectives of the active learning system (training efficiently) and the user (receiving useful instances). We study our approach in an educational application, which particularly benefits from this technique as the system needs to rapidly learn to predict the appropriateness of an exercise to a particular user, while the users should receive only exercises that match their skills. We evaluate multiple learning strategies and user types with data from real users and find that our joint approach better satisfies both objectives when alternative methods lead to many unsuitable exercises for end users.<sup>1</sup>
State-of-the-art machine learning approaches require huge amounts of training data. But for many NLP applications, there is little to no training data available. Interactive NLP systems are a viable solution to alleviate the cost of creating large training datasets before a new application can be used. Such systems start with no or few labeled instances and acquire additional training data based on user feedback for their predictions. Active learning (Settles, 2012) is a frequently used technique to quickly maximize the prediction performance, as the system acquires user feedback in each iteration for
those instances that likely yield the highest performance improvement (e.g., because the system is yet uncertain about them). Active learning has been shown to reduce the amount of user feedback required while improving system performance for interactive NLP systems (P.V.S and Meyer, 2017; Gao et al., 2018) and to reduce the annotation costs in crowdsourcing scenarios (Fang et al., 2014). However, outside the typical annotation setup, it can be boring or frustrating for users to provide feedback on ill-predicted instances that hardly solve their needs. Consider a newly launched web application for learning a foreign language, which aims at suggesting exercises that match the user's proficiency according to Vygotsky's Zone of proximal development (Vygotsky, 1978). The underlying machine learning system starts without any data, but employs active learning to select an exercise the system cannot confidently predict. Then, it adjusts its model interactively based on the user's feedback. While the system is still uncertain, the users often receive inappropriate (e.g., too hard or too easy) exercises. Thus, they get the impression that the system does not work properly, which is especially harmful during the inception phase of an application, as the community opinion largely defines its success.
In this paper, we distinguish the system objective of maximizing the prediction performance with minimal labeled instances and the user objective of providing useful instances for the user's current needs. For the first time, we propose an active learning approach that jointly optimizes these seemingly counteracting objectives and thus trades off the demands of system and user.
The users of educational applications can particularly benefit from this, as they can learn most if they receive appropriate learning material while the underlying system requires considerable training to reach acceptable performance. We employ our
$^{1}$ Our code and simulated learner models are available on Github: https://github.com/UKPLab/acl2020-empowering-active-learning
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4233-4247 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: Overview of our interactive approach. We go beyond previous work on optimizing the system objective (blue) by modeling the user objective (green) and jointly optimizing these seemingly counteracting goals (gold).
new approach in a language learning platform for C-tests (i.e., cloze tests, in which the second half of every second word is replaced by a gap). Our system successfully learns how to predict the difficulty of a C-test gap (system objective) and how to provide a C-test that is neither too easy for the current user, which would cause boredom, nor too hard, which would create frustration (user objective). Predicting the difficulty of an exercise and correspondingly selecting exercises that match a user's proficiency are important steps towards self-directed language learning and massive open online courses (MOOCs) on language learning. Though we focus on this educational use case in this paper, our approach may also yield new insights for other problems that suffer from seemingly counteracting system and user objectives, for example, interactively trained recommender systems for books, movies, or restaurants.
Active learning. Active learning aims to reduce the amount of training data by intelligently sampling instances that benefit the model most (Settles, 2012). A distinct characteristic of active learning is that labels for sampled instances are unknown and provided by an oracle after sampling. Various works investigate the use of active learning for crowdsourcing, where the oracles (i.e., the crowd-workers) may provide noisy labels (Snow et al., 2008; Laws et al., 2011). Within the educational domain, active learning research is scarce. One example is the work by Rastogi et al. (2018), who propose a threshold-based sampling strategy utilizing the prediction probability and achieve a considerable speed-up without any significant performance drop. Hastings et al. (2018) find that ac
tive learning can be used to efficiently train a system for providing feedback on student essays using teachers as oracles. Horbach and Palmer (2016) report mixed results for employing active learning in short-answer grading. While all of these works focus on improvements of the proposed system, users only benefit after training. In contrast, our work explicitly models the user objective, such that users already benefit while labeling training instances.
Adaptive learning. Many systems provide user adaptation, and research has shifted from predefined sets of rules for adaptation to data-driven approaches. Several works investigate adaptive methods to provide exercises which are neither too hard nor too boring. For instance, Missura and Gartner (2011) model learning in a game-theoretic sense where the goal is to adjust the difficulty to neither being too easy nor too hard. Other works investigate adaptation in the context of testing (Zheng and Chang, 2015; Wang et al., 2016; Chaimongkol et al., 2016) and propose methods for an adaptive selection of appropriate tests for better assessing a student's proficiency. In a large survey, Truong (2016) discusses how to integrate different learning styles, modeling categorical student behavior, into an adaptive learning environment and emphasizes the need for more sophisticated methods.
Despite much research in adaptive and active learning, none of the previous works consider jointly modeling and optimizing both the system and user objectives which may retain a user's motivation and keep them from leaving the platform due to boredom or frustration.
Figure 1 shows our proposed interactive learning setup. The active learning component iteratively samples instances from a pool of unlabeled data and asks the user for a label that can be used to
2Note, that in education, active learning often refers to a teaching paradigm which is unrelated to active learning in machine learning.
train the machine learning system. Previous work on active learning focused on optimizing the system objective (blue). That is, only the system provides feedback to the active learning component (e.g., how certain it is about the predicted label of an instance). In our work, we first model the user objective (green) and propose sampling strategies that maximize the user satisfaction based on the user's feedback (e.g., the user's label for an instance). Finally, we study our novel joint optimization strategies (gold) that trade off the demands of the system and the users. Whereas we distinguish between the user's feedback (exercise-level) and labeled instances (gap-level) in our work, our proposed approach can easily be adapted to more specific cases where the (implicit) user feedback and the provided label are the same.
In the remainder of this section, we introduce sampling strategies that select which instance should be presented to the user next. We use the following notation: Let  $\mathcal{X}$  be the pool of unlabeled instances. In every iteration of the application (e.g., when a user requests a new exercise), the sampling strategy  $s(v)$  returns an instance  $x \in \mathcal{X}$  for user  $v$ . The user then provides a label  $y$  for instance  $x$ , potentially with additional feedback on the user's satisfaction. The active learning component finally removes  $x$  from its pool  $\mathcal{X}$  and adds  $(x, y)$  to the set of labeled instances, before the system is retrained with the increased labeled training set.
The simplest sampling strategy that we use as a baseline is random sampling  $s_{\mathrm{rand}}(v)$ , which selects an  $x \in \mathcal{X}$  uniformly at random, regardless of the user. In the following subsections, we discuss more advanced strategies that optimize the system or user objective as well as our new joint optimization strategies.
3.1 System optimization
To optimize the system objective, we consider uncertainty sampling (Lewis and Gale, 1994). Uncertainty sampling assumes that instances for which the model is least certain during prediction provide the most information for the model once their labels are known. The sampled instance is thus
$$
s _ {\text {u n c}} (v) = \arg \max  _ {x \in \mathcal {X}} U (x) \tag {1}
$$
where  $U \colon x \mapsto [0,1]$  returns the uncertainty of predicting a label for instance  $x$ . Like random sampling,  $s_{\mathrm{unc}}(v)$  is independent of the current user  $v$ . A model's uncertainty can be measured in multiple different ways, for example, by the prediction probability of the predicted label (Lewis and Gale, 1994), as the difference in probabilities between the first and second most probable labels (Scheffer et al., 2001), and based on the Shannon entropy (Shannon, 1948) that considers all possible labels (Settles and Craven, 2008). We instantiate  $U$  for our educational application in section 4.
3.2 User optimization
The objective of users is to receive instances that meet their demands. We therefore define a new user-oriented sampling strategy as
$$
s _ {\mathrm {u s r}} (v) = \arg \max  _ {x \in \mathcal {X}} A (x, v) \tag {2}
$$
where  $A\colon (x,v)\mapsto [0,1]$  returns the degree of appropriateness of instance  $x$  for the user  $v$ . In our educational application, we consider an exercise appropriate if it is neither too easy nor too difficult, as this maximizes the user's learning gain. To quantify  $A$ , we measure the error between the predicted label  $f(x)$  and the user's demand  $\phi (v)$  as
$$
A (x, v) = 1 - \operatorname {e r r} [ f (x), \phi (v) ] \tag {3}
$$
with an error function  $\operatorname{err} \in [0,1]$  (cf., section 4).
3.3 Joint optimization
We propose two novel strategies to jointly optimize the user and system objectives.
Combined sampling. Our first strategy
$$
s _ {\operatorname {c o m b}} (v) = \arg \max  _ {x \in \mathcal {X}} U (x) A (x, v) \tag {4}
$$
combines uncertainty sampling and user-oriented sampling by preferring appropriate instances for user  $v$  (as in  $s_{\mathrm{usr}}$ ), but among them returns the one the system is most uncertain about (as in  $s_{\mathrm{unc}}$ ).
Trade-off sampling. For our second strategy, we aggregate both objectives into a single function
$$
\begin{array}{l} s _ {\operatorname {t o s}} (v) = \arg \max  _ {x \in \mathcal {X}} \left\{\left(1 - \lambda\right) A (x, v) \quad \right. \tag {5} \\ + \lambda U (x) \} \\ \end{array}
$$
which is the weighted sum of user-oriented and uncertainty sampling. The weight parameter  $\lambda \in [0,1]$  can be used to adjust the learning towards the system objective or the user objective.
<sup>3</sup>Note, that from a single answer which is either correct or wrong, we cannot deduce a fine-grained gap label. To obtain these in a real-world setting, one either may assume querying groups of users or asking them for an explicit label.
4 Instantiation
We consider our jointly optimized active learning particularly beneficial for educational applications, since (1) the users of such a system may fail to achieve their learning goals with inappropriate exercises. Additionally, (2) it is difficult to acquire large difficulty-annotated datasets for training, as actual users are required for producing realistic training data and existing learner datasets can hardly be shared due to privacy concerns. We therefore instantiate our approach for a language learning platform that predicts the difficulty of exercises and learns to provide appropriate (neither too easy nor too hard) exercises to its users.
C-tests. For our experiments, we use the setup of the C-test difficulty prediction task as investigated by Beinborn (2016). C-tests are gap filling exercises proposed by Klein-Braley and Raatz (1982). In their proposed gap scheme, every second word is turned into a gap by removing the latter half of its characters. In contrast to cloze tests, C-tests do not require any distractors, since the first half of the word remains as a hint. Solving C-tests requires orthographic, morphologic, syntactic, and semantic competencies as well as general vocabulary knowledge (Chapelle, 1994). C-tests can be easily created automatically by choosing an arbitrary text and introducing the gaps as described above. Because of the context and the kept word prefixes, C-test gaps typically only allow for a single solution (given by the original text) and therefore do not require manual correction. The biggest challenge, however, lies in controlling the difficulty of the text and the derived C-test with its gaps as we have shown in previous work (Lee et al., 2019).
System objective. Given a large pool  $\mathcal{X}$  of C-tests  $x\in \mathcal{X}$  with  $n$  gaps  $g_{i}\in x$ ,  $1\leq i\leq n$ , the system objective is to learn a classifier  $d(g)\in L_{D}$  to judge the gap difficulty of gaps  $g\in x$  with minimal training data. As the difficulty classes  $L_{D}$ , we use the four labels very easy, easy, hard, and very hard proposed by Beinborn (2016). These four classes are based on the mean error rates  $e(g)$  of a gap  $g$  observed across all users. Figure 2 shows the mapping between the mean error rates  $e(g)$  and the four gap difficulty classes  $L_{D}$ .
Data. For our experiments, we obtained 3,408 solutions to English C-tests from our university's language center. Each participant solved five C-
Figure 2: Gap difficulty classes and error rate ranges
tests with 20 gaps each (i.e., 100 gaps per solution). The five C-tests vary across the participants based on a set of 74 different C-tests in total. We filter out answers from 22 participants who either did not provide any correct answer or only filled out the first of the five C-tests. Based on this dataset, we derive the ground-truth labels for the gap difficulty classification  $d(g)$  based on figure 2.
Aggregated instances. In contrast to Beinborn's (2016) work, a particular challenge of our setup is the need to aggregate instances. The active learning strategies  $s(v)$  always sample entire C-tests  $x \in \mathcal{X}$  and judge their appropriateness for a user  $v$  based on  $A(x,v)$ . The underlying classifier  $d(g)$ , however, operates at the level of gaps  $g \in x$  within a C-test. Similarly complex setups can be found in multiple other real-world tasks, including educational applications (e.g., providing reading recommendations at book or chapter level, but estimating appropriateness at word or sentence level) and product recommendation tasks (e.g., training a classifier for cast, plot, and action aspects, but recommending entire movies).
For our instantiation, we measure the classifier's uncertainty using the Shannon entropy
$$
H (g) = - \sum_ {\ell \in L _ {D}} P (\ell \mid g) \log P (\ell \mid g) \tag {6}
$$
across the four difficulty classes  $L_{D}$  of a gap  $g$ .  $P(\ell \mid g)$  denotes the probability of the classifier  $d$  to assign the difficulty class  $\ell$  to gap  $g$ . We then aggregate the resulting scores similar to the total token entropy proposed by Settles and Craven (2008):
$$
U _ {\text {e n t}} (x) = \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {H \left(g _ {i}\right)}{H _ {\max }} \tag {7}
$$
where  $H_{\mathrm{max}}$  is the maximum achievable Shannon entropy, which serves as a normalization term.  $H_{\mathrm{max}}$  can be pre-computed as:
$$
H _ {\max } = - \sum_ {i = 1} ^ {| L _ {D} |} \frac {1}{| L _ {D} |} \log \frac {1}{| L _ {D} |} \tag {8}
$$
User objective. To model the demands of the users, we define five proficiency levels  $L_{P} =$
Level	1	2	3	4	5
Score (%)	0–54	55–64	65–74	75–84	85–100
Users	814	607	724	769	472
Table 1: Proficiency levels, corresponding scores (\% correctly filled gaps), and number of users per level.
$\{1,2,3,4,5\}$  based on the users' ability to solve C-tests. The user representation  $\phi (v)\in L_P$  of user  $v$  thus returns a proficiency level between 1 and 5 with 5 indicating the highest proficiency.
In our experiments, we use the C-test dataset introduced above to obtain  $\phi(v)$ . Note that in this dataset, each user solved exactly five C-tests. We therefore map their score (i.e., the percentage of correctly filled gaps) to a proficiency level that roughly corresponds to the language courses offered by the university language center. Table 1 shows the five levels with their corresponding score ranges and the number of users in the dataset.
We estimate the proficiency level of a C-test  $x = g_{1},g_{2},\ldots ,g_{n}$  with
$$
f (x) = \psi \left(\frac {1}{n} \sum_ {i = 1} ^ {n} c \left(g _ {i}\right)\right) \tag {9}
$$
where  $c\colon g\mapsto \{0,1\}$  is an indicator function to predict if gap  $g_{i}$  will be correctly (1) or incorrectly (0) answered and  $\psi$  maps the percentage of correct answers to the corresponding proficiency level according to Table 1. For our experiments, we define
$$
c (g) = \left\{ \begin{array}{l l} 1 & \text {i f} k <   j \\ 0 & \text {o t h e r w i s e} \end{array} \right. \tag {10}
$$
where  $k\sim \mathcal{U}(\frac{\ell - 1}{|L_P|},\frac{\ell}{|L_P|})$  and  $j\sim \mathcal{U}(0,1)$  are uniformly sampled random variables and  $\ell = d(g)$ . Based on our estimation  $f(x)\in L_{P}$ , we can now define the error function err as the normalized distance of  $f(x)$  to the required proficiency:
$$
\operatorname {e r r} [ f (x), \phi (v) ] = \frac {1}{| L _ {P} |} | f (x) - \phi (v) | \tag {11}
$$
System setup. We initialize our system with an empty set of labeled instances. In every iteration, we sample a C-test consisting of 20 gaps from the pool of unlabeled instances  $\mathcal{X}$  using one of the sampling strategies introduced in the previous section. Then, we obtain labels based on how the user solved the test, which contributes (1) to the overall difficulty prediction for each gap and (2) to the representation of the current user's proficiency.
Our approach can be used with any underlying classifier  $d(g)$ . In this paper, we train a multilayer perceptron (MLP) to predict the four difficulty classes for a C-test gap. To represent the input of the MLP, we use the 59 features previously proposed by Beinborn (2016). We furthermore introduce two novel features computed from BERT (Devlin et al., 2019): We hypothesize that the masking objective of BERT which masks individual words during training is very similar to a gap filling exercise and thus, a model trained in such a way may provide useful signals for assessing the difficulty of a gap. For each gap, we generate a sentence where only the gap is replaced by the masking token and fetch its predictions from the BERT model. From these predictions we take the prediction probability of the solution as the first feature and the entropy of the prediction probabilities of the top-50 predicted words as the second feature in concordance with findings by Felice and Buttery (2019) who show that entropy strongly correlates with the gap difficulty. Adding both features to the 59 features proposed by Beinborn (2016) increases the accuracy of our MLP from 0.33 to 0.37.4
While Beinborn successfully used support vector machines (SVM) in her work, we find that MLPs perform on par with SVMs (for the old and new features) and that they are more robust regarding the choice of the first sampled instance. Moreover, in our initial experiments with little training data, SVMs and Logistic Regression classifiers were only able to predict the majority class.
Our MLP has a single hidden layer consisting of 61 hidden units. We train the neural network for 250 epochs with early stopping after 20 epochs without any improvement and use Adam (Kingma and Ba, 2015) as our optimizer. Note that our main interest is in the analysis of the novel active learning approach, which is why we do not systematically study the underlying classifier, but use a setup comparable to the state-of-the-art results reported by Beinborn (2016).
We run experiments for each of our sampling strategy. We select five C-tests without any overlap between users, texts, and their corresponding user answers to create an independent test set and put the remaining 69 C-tests into the pool of unlabeled data. In the first iteration, we use the randomly initialized weights of our neural network to select
4The results are averaged across ten runs with different random initializations.
the starting example. To provide comparable results between different runs, we keep the parameter initialization of our neural network fixed when comparing different sampling strategies. We limit each experimental run to  $8 \cdot 5 = 40$  iterations, as the five proficiency levels are not evenly distributed with the smallest class having only eight C-tests. At each iteration, we train our model on  $80\%$  of the already labeled data and use the remaining  $20\%$  as our validation set (split randomly). We use the best-performing model on the validation set for testing and store it as our model initialization for the next iteration. On an Intel Core i5-4590, a single run with 40 iterations takes less than four minutes.
Learner behavior. To study the benefit of our approach for different types of learners, we derive four prototypical learner behaviors from our C-test dataset. To prepare this, we first compile a probabilistic model for the learners of each proficiency group as described in Table 1 to obtain learner-specific gap error rates  $e(g,v)$ . The learner-specific gap error rates are computed by binning all learners into the specific groups and then computing the error rate by averaging for each gap. If there is no error rate for a given gap and learner in our dataset, we use the averaged gap error rate of the corresponding proficiency group to simulate an answer.
Using these learner-specific gap error rates, we predict whether an answer to a C-test gap  $g$  is correct or incorrect similar to Equation (10):
$$
\hat {c} (g) = \left\{ \begin{array}{l l} 1 & \text {i f} e (g, v) <   j \\ 0 & \text {o t h e r w i s e} \end{array} \right. \tag {12}
$$
In contrast to Equation (10), we do not sample  $k$  but use the learner-specific error rates  $e(g, v)$  for gap  $g_i$  from the proficiency level  $\phi(v)$ . Again,  $j \sim \mathcal{U}(0, 1)$  is a uniformly sampled random variable.