Impact of Batch Size on Stopping Active Learning for Text Classification

When using active learning, smaller batch sizes are typically more efficient from a learning efficiency perspective. However, in practice due to speed and human annotator considerations, the use of larger batch sizes is necessary. While past work has shown that larger batch sizes decrease learning efficiency from a learning curve perspective, it remains an open question how batch size impacts methods for stopping active learning. We find that large batch sizes degrade the performance of a leading stopping method over and above the degradation that results from reduced learning efficiency. We analyze this degradation and find that it can be mitigated by changing the window size parameter of how many past iterations of learning are taken into account when making the stopping decision. We find that when using larger batch sizes, stopping methods are more effective when smaller window sizes are used.

section_name: Introduction
paragraphs: The use of active learning has received a lot of interest for reducing annotation costs for text classification BIBREF0 , BIBREF1 , BIBREF2 .
Active learning sharply increases the performance of iteratively trained machine learning models by selectively determining which unlabeled samples should be annotated. The number of samples that are selected for annotation at each iteration of active learning is called the batch size.
An important aspect of the active learning process is when to stop the active learning process. Stopping methods enable the potential benefits of active learning to be achieved in practice. Without stopping methods, the active learning process would continue until all annotations have been labeled, defeating the purpose of using active learning. Accordingly, there has been a lot of interest in the development of active learning stopping methods BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 .
Another important aspect of the active learning process is what batch size to use. Previous work has shown that using smaller batch sizes leads to greater learning efficiency BIBREF2 , BIBREF7 . There is a tension between using smaller batch sizes to optimize learning efficiency and using larger batch sizes to optimize development speed and ease of annotation. We analyze how batch size affects a leading stopping method and how stopping method parameters can be changed to optimize performance depending on the batch size.
We evaluate the effect batch size has on active learning stopping methods for text classification. We use the publicly available 20Newsgroups dataset in our experiments.
For our base learner, we use the implementation of a Support Vector Machine from the scikit-learn Python library. For our sampling algorithm, we use the closest-to-hyperplane algorithm BIBREF2 , which has been shown in recent work to compare favorably with other sampling algorithms BIBREF8 . We use a binary bag of words representation and only consider words that show up in the dataset more than three times. We use a stop word list to remove common English words.
For analyzing the impact of batch size on stopping methods, we use a method that will stop at the first training iteration that is within a specified percentage of the maximum achievable performance. We denote this method as the Oracle Method, and we will set the percentage to 99 and denote this as Oracle-99. We set the percentage to 99 because it is typical for leading stopping methods to be able to achieve this level of performance (see Table 1 in BIBREF4 ). Although the Oracle Method cannot be used in practice, it is useful for contextualizing the stopping results of practical stopping methods.
section_name: Results
paragraphs: We considered different batch sizes in our experiments, based on percentages of the entire set of training data. The results for batch sizes corresponding to 1%, 5%, and 10% of the training data for the 20Newsgroups dataset are summarized in Table~ SECREF4 .
section_name: Oracle Results
paragraphs: Looking at Table~ SECREF4 , one can see that Oracle-99 needs more annotations with larger batch percents to reach approximately the same F-Measure as with smaller batch percents.
These results are consistent with past findings that learning efficiency is decreased with larger batch sizes BIBREF2 , BIBREF7 . However, an open question is whether changing the parameters associated with actual stopping methods can make them experience less degradation in performance when larger batch sizes are used. In particular, an important parameter of stopping methods is the window size of previous iterations to consider. The next subsection shows how decreasing the window size parameter can help to reduce the degradation in performance that stopping methods experience with larger batch sizes.
section_name: Comparing BV2009 with the Oracle Method
paragraphs: We denote the stopping method published in BIBREF4 as BV2009. This stopping method will stop the active learning process if the mean of the three previous kappa agreement values between consecutive models is above a threshold. For larger batch percents, note that BV2009 stops later than the optimal Oracle Method point.
We ran BV2009 with smaller window sizes for each of our different batch sizes. Our results are summarized for a window size of one in the row ``BV2009 (Window Size = 1)'' in Table~ SECREF4 . When using a window size of one, BV2009 is able to stop with a smaller number of annotations than when using a window size of three. This is done without losing much F-Measure. The next subsection provides an explanation as to why smaller window sizes are more effective than larger window sizes when larger batch sizes are used.
section_name: BV2009 Window Size Discussion
paragraphs: We set INLINEFORM0 to be the window size that the user has defined. Kappa is an agreement metric between two models. Therefore, BV2009 needs INLINEFORM1 models to be generated before it begins to check if the average is above the threshold. This does not necessarily mean it stops after INLINEFORM2 models have been generated. Rather, it represents the first point in the active learning process at which BV2009 even has a chance to stop.
When using larger batch percents, fewer models are generated than when using smaller batch percents. This gives any stopping method less points to test whether or not to stop. We also note that kappa agreement scores are generally low between the first few models trained. This, combined with fewer points to stop at, causes BV2009 to stop somewhat sub-optimally when using very large batch percents. Usage of very large batch sizes, such as 10% of the data, is not common so sub-optimal performance of stopping methods in those situations is not a major problem.
section_name: Conclusion
paragraphs: Active learning has the potential to significantly reduce annotation costs. Two important considerations in the active learning process are when to stop the iterative process of asking for more labeled data and how large of a batch size to use when asking for additional labels during each iteration. We found that stopping methods degrade in performance when larger batch sizes are used. The degradation in performance is larger than the amount that can be explained due to the degradation in learning efficiency that results from using larger batch sizes. An important parameter used by stopping methods is what window size of earlier iterations to consider in making the stopping decision. Our results indicate that making the window size smaller helps to mitigate the degradation in stopping method performance that occurs with larger batch sizes.
section_name: Acknowledgment
paragraphs: This work was supported in part by The College of New Jersey Support of Scholarly Activities (SOSA) program, by The College of New Jersey Mentored Undergraduate Summer Experience (MUSE) program, and by usage of The College of New Jersey High Performance Computing System.

Questions and Answers:
question: What downstream tasks are evaluated?
answers: answer: unanswerable: yes
answer: unanswerable: no
extractive_spans: text classification
evidence: We evaluate the effect batch size has on active learning stopping methods for text classification. We use the publicly available 20Newsgroups dataset in our experiments.
question: What is active learning?
answers: answer: unanswerable: no
free_form_answer: A process of training a model when selected unlabeled samples are annotated on each iteration.
evidence: Active learning sharply increases the performance of iteratively trained machine learning models by selectively determining which unlabeled samples should be annotated. The number of samples that are selected for annotation at each iteration of active learning is called the batch size.
answer: unanswerable: no
free_form_answer: Active learning is a process that selectively determines which unlabeled samples for a machine learning model should be annotated.
evidence: Active learning sharply increases the performance of iteratively trained machine learning models by selectively determining which unlabeled samples should be annotated. The number of samples that are selected for annotation at each iteration of active learning is called the batch size.

Figure and Table Captions:
caption: TABLE I STOPPING METHOD RESULTS ON 20NEWSGROUPS FOR DIFFERENT BATCH SIZES USING VARIOUS WINDOW SIZES. THE TOP NUMBER IN EACH ROW SHOWS THE NUMBER OF ANNOTATIONS AT THE STOPPING POINT AND THE BOTTOM NUMBER SHOWS THE F-MEASURE AT THE STOPPING POINT.