Re-weighting Tokens: A Simple and Effective Active Learning Strategy for Named Entity Recognition
Authors: Haocheng Luo
Wei Tan
Ngoc Dang Nguyen
Lan Du
Venue: EMNLP 2023 Findings (2023)
Keywords: Named entity recognition
active learning

Abstract: Active learning, a widely adopted technique for enhancing machine learning models in text and image classification tasks with limited annotation resources, has received relatively little attention in the domain of Named Entity Recognition (NER). The challenge of data imbalance in NER has hindered the effectiveness of active learning, as sequence labellers lack sufficient learning signals. To address these challenges, this paper presents a novel re-weighting-based active learning strategy that assigns dynamic smoothing weights to individual tokens. This adaptable strategy is compatible with various token-level acquisition functions and contributes to the development of robust active learners. Experimental results on multiple corpora demonstrate the substantial performance improvement achieved by incorporating our re-weighting strategy into existing acquisition functions, validating its practical efficacy. We will release our implementation upon the publication of this paper.

Re-weighting Tokens: A Simple and Effective Active Learning Strategy for Named Entity Recognition
Haocheng Luo, Wei Tan, Ngoc Dang Nguyen and Lan Du
Department of Data Science and AI, Monash University [Haocheng.Luo, Wei.Tan2, dan.nguyen2, Lan.Du) ) @ monash.edu
Abstract
2017: Shen al., 2017; Tan et al., 2021; Hacohen et al., 2022).
Active learning, a widely adopted technique for enhancing machine learning models in text and image classification tasks with limited an- notation resources, has received relatively little attention in the domain of Named Entity Recog- nition (NER). The challenge of data imbalance in NER has hindered the effectiveness of ac- tive learning, as sequence labellers lack suffi- cient learning signals. To address these chal- lenges, this paper presents novel reweighting- based active learning strategy that assigns dy- namic smoothed weights to individual tokens. This adaptable strategy is compatible with vari- ous token-level acquisition functions and con- tributes to the development of robust active learners. Experimental results multiple cor- pora demonstrate the substantial performance improvement achieved by incorporating our re-weighting strategy into existing acquisition functions, validating its practical efficacy.
The primary challenge in applying active learn- ing to sequence tagging lies in addressing data im- balance the entity level. Traditional methods of active sequence tagging. such as those refer- enced by (Shen et al., 2017; Zhang et al., 2020; Shelmanov et al., 2021). typically generate scores for sentences by summing or averaging the tokens within them, thereby treating each token equally. Radmard et al. (2021) attempted to address this by segmenting sentences for token-level selections, However this led to loss of context and seman- tic meaning, impairing human understanding. To effectively tackle this data imbalance issue, is imperative to focus on the token level within the sentences. By implementing a strategy that re- distributes sentence selection through assigning weights to the tags of the tokens, more balanced and efficient handling of the data can be achieved.
1 Introduction
In supervised learning (SL) and semi-supervised learning (SSL), re- weighting is a widely researched method to tackle data imbalance. However, they have limitations to be directly applied to active learning. For example, Wei al. (2021); Cao al. (2019) assume the labeled data and the unlabeled data have the same label distribution, while we usually have no prior information about the un- labeled data in active learning; Kim et al. (2020) use the confusion matrix on the labeled data to ap- proximate the label distribution of the unlabeled data. In active learning, the labeled data pool can be quite small, the approximation strong biases; Li al. (2018) suggest gradient- based method compute weights, while comput- ing gradients significantly increases the computa- tional cost for active learning.
The effectiveness of deep neural networks has been well estabblbabl on diverse sequence tagging tasks, including named entity recognition (NER) (Lam- ple et al., 2016). However, the evolving complexity of these models demands substantial labeled data, making annotation a resource-intensing endeavor. Active lcarning (AL) addresses this by astutely fo- cusing on information rich sequences, substantially reducing data that needs expert annotation. Such a strategy proves invaluable in NER, as it brings down costs and accelerates learning by honing in complex or dynamic entities. AL's iterative process smartly picks subsets from unlabeled pool, leveraging expert-annotated data to refine the model. Modern AL approaches primarily pivot on model uncertainty (Houlsby et al., 2011; Gal et al., 2017: Shen et al., 2017; Kirsch et al., 2019; Zhao et al., 2021) or data diversity (Sener and Savarese,
cause
In this paper, we focus on the feasibility of adapt- ing reweighting- methods from SI., SSL active sequence tagging, which has not been devel- oped in existing works. Our contributions can be
Corresponding Author

---

summarized follows:
mutual information (Houlsby et al., 2011) between outputs and model parameters.
We propose a flexible saweighting-baser method for active sequence tagging to tackle the data imbalance issue. It is easy to implement, and generic to any token-level acquisition function. We conduct extensive experiments to show our method can improve the performance for each baseline, across large range of datasets and query sizes. We also show that it can indeed mit- igate the class imbalance issue, which explains its success.
It is worth noting that we exclude subsequence- based active learning methods (Wanvarie et al., 2011; I; Radmard et al., 2021; Liu et al., 2023) from our comparison. In their setting, the query sequence instead of the whole sentence. As they admitted, it may be not practical in the real world because subsequences can be meaningless. Human experts usually need the context to label tokens.
sub-
Re-weighting methods for supervised learning and semi-supervised learning. Many real-world datasets exhibit "long-tailed" label distribution (Van Horn et al., 2018; 88; Liu et al., 2020; Chu et al.. 2020). Imbalanced supervised learning has been widely researched. Re-weighting the loss (Khan et al., 2017; Aurelio et al., 2019; Tan et al., 2020) is one of the popular methods on this topic. In recent semi-supervised learning methods, re-weighting is gradually gaining attention. Wei al. (2021) uti- lize the unlabeled data with pseudo labels accord- ing to an estimated class distribution, Kim et al. (2020) develop a framework to refine class distri- bution, Lai et al. (2022) use the effective number defined by Cui et al. (2019) to produce adaptive weights.
2 Related works
Active learning for NER. Active learning for NER has seen variety of methodologies being devel- oped to address its unique challenges (Settles and Craven, 2008; Marcheggiani and Artieres, 2014; Shelmanov al., 2021). The overarching goal is reduce the budget for labeling sequence data by selectively querying informative samples. The in- formation content of a sample is typically measured using an acquisition function, which quantifies the potential usefulness of unlabeled data.
Uncertainty-base acquisition functions have gained popularity due to their successful applica- tion and their compatibility with neural networks, which can generate a probability distribution using the softmax function over the model's output. Var- ious acquisition functions have been proposed to represent uncertainty. Least Confidence (LC) (Li and Sethi, 2006), for example, selects the sentence with the lowest model prediction confidence, while Sequence Entropy (SE) (Settles and Craven, 2008) employs the entropy of the output instead of the probability. Alternatively, Margin (Scheffer et al., 2001) defines uncertainty as the difference between the highest and the second-highest probability.
Methodology
3.1 Preliminaries
Consider multi-class token classification prob- lem with C' classes. An input is sequence of T tokens, denoted by (x+ ....T), and its cor- responding label is denoted by y = (y',...y'), The labeled data contains m samples, denoted by L = {x and the unlabeled data contains n samples, denoted by U = (x,]]=1- with m << n. In each active learning iteration, we select B data points form query Q, according some acquisition function q:
These traditional approaches, however, have their limitations. Shen et al. (2017) pointed out that summing over all tokens, as done in previ- ous methods, can introduce bias towards longer sentences. To overcome this, they proposed Maxi- mum Normalized Log-Probability (MNLP), a nor- malized version that mitigates this issue. In more recent development, Lowest Token Proba- bility (LTP) (Liu al., 2022) selects the sentence containing the token with the lowest probability. Additionally, Bayesian Active Learning by Dis- agreement (BALD) (Gal et al., 2017) uses Monte- Carlo Dropout to approximate the Bayesian pos- terior, which makes it feasible to approximate the
arg maggmaxpm g(x) XCU
3.2 Re-weighting Active learning for named entity recognition
Inspired by reweighting- methods for super- vised learning and semi-supervised learning, we propose a new re-weighting strategy for active se- quence tagging, which assigns a smoothed weight for each class, which is inversely proportional to

---

the class frequency in the labeled data pool:
we do not need make assumptions on the unla- beled data. Computationally lightweight. The time complexity of computing weights is (mm). Note that m << n in active learning, so it is more efficient compared to the effective number-based reweighting-base based method with time complexity (n) (Lai et al., 2022). Generic to any base ac- quisition function. Since our method is essentially weighted sum strategy, it can be combined with any base acquisition function for active learning mentioned in section 4.
1
Wk my + 3m for h = 1.2,....
where 3 is a hyperparameter controlling the degree of smoothing, my = (a.y)ec ly=k is the sample size of class k in the labeled data pool, m repre- sents the size of the entire labeled data pool. When 3 = 0, weights are strictly the inverse of the class frequencies; when 8 oc, weights follow a umi- form distribution, i.e. no re-weighting is activated. A reasonable value for 3 plays a significant role to combat sampling bias at the early stage of ac- tive learning. Based on this re-weighting function,
Experimental Setups
Datasets: In this section, we evaluate the efficacy of re-weighted active learning on three widely used Neural network (2; 0), labeled pool C, NER datasets, which are listed as follows, the sta- unlabeled pool U, size of queries B, a base loken- - level acquisilion f'unction q. computing weights using the labeled data
Algorithm 1 Re-wethtting active learning for NER
Require:
tistical data of each dataset is provided in the Ap- pendix B.
Conl12003 (Sang and De Meulder, 2003) is corpus for NER tasks, with four entity types in BIO format (LOC, MISC, PER, ORG), resulting = 9 classes
Uk my + 3m' Sor k = 1, 2,2, ....C
Initialize query set: =
WikiAnn (English) (Pan et al., 2017) is a corpus extracted from Wikipedia data, with three entity types in BIO format (LOC, PER, ORG), resulting
for x E U do
8' = arg max, ((ucl=t;0) 0)
(xx = E, (x')
C = 7 classes.
end for
BCSCDR (Wei et al., 2016) is biomedical dataset, with two entity types in BIO format (Dis- ease, Chemical), resulting C classes
for b- = 1.2... B do
x arg milx (acke(((((00 (x())
Q Q Ux
NER model: We finetuned BERT-base-cased model (Devlin et al., 2018) with an initialized lin- ear classification layer to perform sequence tagging. All experiments share the same settings: we used Adam (Kingma and Ba, 2014) with the constant learning rate 5e and the weight decay at 5e 5 as the optimizer; the number of training iterations is 30, All experiments were done with an Nvidia A40 GPU.
u 41 1x
end for return Q
we generate a sentence-level acquisition score by computing the weighted sum over all tokens. The algorithm is shown in Algorithm 1. It should be noticed that we do not have true labels of unlabeled data in active learning, therefore we use the pseudo label 9 instead. This pseudo label trick is widely used and has been verified as effective in many ac- tive learning works (Shen et al., 2017; Ash et al., 2019; Wang et al., 2022; Mohamadi et al., 2022).
Active learning settings: The initial size of the labeled data pool is 30, the number of active learning iterations is 10, the query size varies from {15, 30, 50). In each active learning iteration, we selected a batch of unlabeled data, queried their labels, added them to the labeled data pool, and then re-trained the model. We evaluated the perfor- mance of algorithms by their mean Fl-scores with 95% confidence interval for five trials on the test dataset.
Our method has the following advantages: In- dependent on the label distribution of the un- labeled data. Estimating the label distribution of the unlabeled data can be tricky, especially at the early stage of active learning or the unlabeled data does not have the same distribution as the la- beled data. Compared to reweighting- based meth- ods for SSL (Wei et al., 2021; Lai et al., 2022),
Baselines: We considered five baselines: (1) randomly querying, (2) Least Confidence (LC) (Li and Sethi, 2006), (3) Maximum Normalized Log-

---

Dataset: Conli2003, query size=15
0.8
F1-score
0.7
Random
Random
Random
Random
0.6
Entropy
BALD_m rw
LC_rv
Entropy_rv rw
MNLP_ .r
10
iterations
Dataset: WikiAnn, query size=15
F1-score
Random
Random
Random
Random
0.6
BALD
LC
Entropy
MNLP
BALD rw
LC_
Entropy rw
MNLP .W
10
2
8
10
10
10
AL iterations
Dataset: BCSCDR, query size=15
0.7
F1-score
Random
Random
Random
Random
BALD
LC
Entropy
MNLP
0.5
BALD
Entropy_nv
MNLP
10
10
10
2
10
AL fterations
Figure 1: Pairwise comparison between with and without weighting for each baseline with query size 15. 'rw' is an abbreviation for re-weighting. Orange dashed lines indicate randomly querying, blue solid lines represent the original baselines, red solid lines represent the reweighting-bett versions. The shaded area indicates the 95% confidence interval.
Dataset: Conli2003, query size=15
Entropy
Imbalance Ratio
25
Lc_rw
Entropy_nv
15
10
10
iterations
Figure 2: The variation of the imbalance ratio during active learning process Conl12003 with query size 15. Black dashed lines indicate the overall imbalance ratio of the entire dataset, blue solid lines represent original baselines. red solid lines represent reweighting-base versions. The shaded area indicates the 95% confidence interval.
Probability (MNLP) (Shen et al., 2017), (4) Se- quence Entropy (SE) (Settles and Craven, 2008), (5) Bayesian Active Learning by Disagreement (BALD) (Gal al., 2017). The details of the base- lines are mentioned in the Appendix
for all experiments. Please see Appendix C for the performances of all hyperparameters.
Experimental Results & Discussions
5.1 Main results
Hyperparameter setting: As our reweighting- based methods have a hyperparameter to be tuned, we first conduct grid search over {0.01, 0.2, 0.5, 1} in the re-weighting LC case on each dataset, and fix the optimal value 3 = 0.1
We evaluate the efficacy of reweighting-bant method by comparing the performance of active sequence tagging with and without re-weighting. Figure I shows the main results on three datasets with query size 15. We delay the results of other

---

query sizes to Appendix C. It is clear that each re-weighting method consistently outperforms the original method across different datasets and query sizes.
Iteration#
1
2
3
Smoothed
61.2+00.2
7..5+0.9
81.0+#.3
(3 0.1)
Non-
60.3
68.9
18.2+
smoothed
5.2 Analysis of the performance gain
Table 1: Comparison with non-smoothed re-weighting function. Mean Fi-scores and their 95% confidence intervals in the first three AL iterations on Conl12003 with query size 15 are reported.
As discussed, the label imbalance issue occurs commonly in NER datasets, and it potentially dam- ages the performance. We argue that the perfor- mance gain of the reweighting-base because can effectively decrease the imbalance ratio. To evaluate this, we define an imbalance ratio as
method is
Conclusion
In this work, we pioneered the use of re-weighting in active learning for NER to tackle the label im- balance issue. Our method uses the class sizes in the labeled data pool to generate a weight for each token. We empirically demonstrate that our reweighting-bast based method can consistently im- prove performance for each baseline, across a large range of NER datasets and query sizes.
N
7
Nmn
where Ne is the sample size of class and Nmin is the sample size of the class with the minimum number of samples in the labeled dataset. For balanced label distribution, the imbalance ratio close to 1. A larger imbalance ratio indicates a higher class imbalance.
Limitations
We acknowledge that the static nature of the hy- perparameter is a limitation of our work, par- ticularly when applying the algorithm to new or diverse datasets. Dynamically updating 3 line with Active Learning iterations can be a promising avenue for future research.
As shown in Figure 2, we plot the variation of the imbalance ratio on Conl12003 during the active learning process. Results on other datasets are put in Appendix C due to space limitations. In general, models have lower confidence samples from minority classes. As result, uncertainty-basee ac- quisition functions bias towards selecting samples from minority classes implicitly. This explains why they can achieve better performance than randomly querying. In contrast, reweighting-bald based methods further bias towards minority classes explicitly. leads to more balanced querying, which can cause better performance.
References
Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy. John Langford, and Alekh Agarwal. 2019. Deep batch active learning by diverse. uncertain gradient lower bounds. arXiv preprint 1rXiv:1906.0333
Yuri Sousa Aurelio, Gustavo Matheus De Almeida, Cris- tiano Leite de Castro, and Antonio Padua Braga. 2019. Learning from imbalanced data sets with weighted cross-entropy function. Neural processing letters, 50: 1937-1949.
5.3 Ablation study
Effect of smoothing. The first term in the denomi- nator of our re-weighting function 3m, which con- trols the degree smoothing, plays a significant role at the early stage of active learning. Not using the smoothed version may encounter sampling bias issues. To verify these views, we report fl scoress of the smoothed version and the non non-smoothed ver- sion in the first three AL iterations in table across five independent runs. For each experiment, set LC the base acquisition function. It can be observed that non-smoothed version has lower per- formance and higher variance at first. This is what we expect to see in our motivation.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. 2019. Learning imbalanced datasets with aibelldistribution-avvee margin loss. Advances in neural information processing systems, 32.
Peng Chu. Xiao Bian, Shaopeng Liu, and Haibin Ling. 2020. Feature space augmentation for long-tailed data. In Computer Aision-ECCv 2020: 16th Euro- pean Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIX 16, pages 694-710. Springer.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. 2019. Class-balanced loss based on effective number of samples. In Proceedings of

---

the IEEE/CVF conference on computer vision and pattern recognition, pages 9268-9277.
Jialun Liu, Yifan Sun, Chuchu Han, Zhaopeng Dou, and Wenhui Li. 2020. Deep representation learning on long-tailed data: A learnable embedding augmenta- tion perspective. In Proceedings of the IEEE/CVF conference on computer vision and patlern recogni- tion, pages 2970-2979.
Jacob Devlin, Ming-Wei, Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arviv preprint 1rXiv:1810.04805..
Mingyi Liu, Zhiying Tu, Tong Zhang, Tonghua Su, Xi- aofei Xu, and Zhongji Wang. 2022. Lip: a new active learning strategy for crf-based named entity recognition. Neural Processing Letters, 54(3):2433- 2454.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. 2017. Deep bayesian active learning with image data. In International conference on machine learning, pages 1183-1192. PMLR.
Yang Liu, Jinpeng Hu, Zhihong Chen, Xiang Wan, and Tsung-Hui Chang. 2023. Easal: entity-aware subsequence-basee active learning for named entity recognition. In Proceedings the AAAI Conference on Artifiiaal Intelligence, volume 37, pages 8897- 8905.
Guy Hacohen, Avihu Dekel, and Daphna Weinshall. 2022. Active learning on a budget: Opposite strate- gies suit high and low budgets. arXiv preprint arXiv:2202.02794.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. 2011. Bayesian active learning for classification and preference learning, arXiv preprint 11255:111..5745.
Diego Marcheggiani and Thierry Artieres. 2014. An experimental comparison of active learning strategies for partially labeled sequences. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 898-906.
Salman H Khan, Munawar Hayat, Mohammed Ben- namoun, Ferdous A Sohel, and Roberto Togneri. 2017. Cost-sensitive learning of deep feature repre- sentations from imbalanced data. IEEE transactions on neural networks and learning systems, 29(3):3573- 3587.
Mohamad Amin Mohamadi, Wonho Bae, and Danica J Sutherland. 2022. Making look-ahead active learning strategies feasible with neural tangent kernels. arXiv preprint arXiv: 220.1699.0....6.
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Noth- man, Kevin Kuight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Pro- ceedings of the 55th Annual Meeting of the Associa- tion for Computational Linguistics (Volune Long Papers), pages 1956-1958.
Jachyung Kim, Younghum Hur, Sejun Park, Funho Yang, Sung Ju Hwang, and Jinwoo Shin. 2020. Dis- tribution aligning refinery of pseudo-label for imbal- anced semi-supervised learning. Advances in neural information processing systems, 13:17467-44579
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv: 142.6914...680.
Puria Radmard, Yassir Fathullah, and Aldo Lipani. 2021. Subsequence based deep active learning for named entity recognition. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural I Language Processing (Volume 1: Long Papers), pages 4310 4321.
Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal. 2019. Batchbald: Efficient and diverse batch acquisi- tion for deep bayesian active learning. Advances in neural information processing systems, 32.
Erik Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. arXiv preprint cx/0306050.
Zhengfeng Lai, Chao Wang, Henrry Gunawan, Sen- Ching S Cheung, and Chen-Nee Chuah. 2022. Smoothed adaptive weighting for imbalanced semi- supervised learning: Improve reliability against un- known distribution data. In International Conference on Machine Learning, pages 11828-1 1843. PMLR.
Tobias Scheffer, Christian Decomain, and Stefan Wro- bel. 2001. Active hidden markov models for infor- mation extraction. In Advances in Intelligent Data Analysis: 4th International Conference, IDA 2001 Cascais, Portugal, September 13-15, 2001 Proceed- ings 4, pages 309-311 108. Springer.
Guillaume Lample, Miguel Ballesteros, Sandeep Sub- ramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. arXiv preprint arXiv: 1603.0130.
Ozan Sener and Silvio Savarese. 2017. Active learn- ing for convolutional neural networks: A core-set approach. arXiv preprint 1rXiv:1708.000 00489.
Buyu Li, Yu Liu, and Xiaogang Wang. 2018. Gradient harmonized single-stage detector.
Burr Settles and Mark Craven. 2008. An analysis of
active learning strategies for sequence laheling tasks. In proceedings of the 2008 conference on empirical methods natural language processing, pages 1070- 1079.
Mingkun Li and Ishwar K Sethi. 2006. Confidence- based active learning. IEEE transactions on pattern analysis and machine intelligence, 28(8): -121.

---

Artem Shelmanov, Dmitri Puzyrev, Lyubov Kupriyanova, Denis Belyakov, Daniil Larionov, Nikita Khromov, Olga Kozlova, Ekaterina Artemova, Dmitry V Dylov, and Alexander Panchenko. 2021. Active learning for sequence tagging with deep pre- trained models and bayesian uncertainty estimates. arXiv preprint arXiv: 101.0813.
A Baselines
For completeness, we list several commonly used acquisition functions below.
Least Confidence (LC) (Li and Sethi, 2006) uses the probability of the model's prediction to measure the uncertainty:
Yanyao Shen, Hyokun Yun, Zachary C Lipton, Yakov Kronrod, and Animashree Anandkumar. 2017. Deep active learning for named entity recognition. arXiv preprint 1777:1707.00922
(LC(x)= = => (lo(2")
(1)
= -E - p(001> 1))
(2)
Jingru Tan, Changbao Wang, Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan. 2020. Equalization loss for toiget object recognition. In Proceedings of the IEEE/CVF conference com- puter vision and pattern recognition, pages 11662- 11671.
Sequence Entropy (SE) (Settles and Craven, 2008) utilizes the probability distribution instead of the probability to measure the uncertainty:
ISE (x) = Σ UsE(a")
(3)
Wei Tan, Lan Du, and Wray Buntine, 2021. Diversity enhanced active learning with strictly proper scoring rules. Advances Neural Information Processing Systems, 34:10906-10918
ΣΣ p(vcla", O'logplulla" 0) (4)
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. 2018. The inaturalist species classification and detection dataset. In Pro- ceedings of the IEEE conference computer vision and pattern recognition, pages 8769-8775
Bayesian Active Learning by Disagreement (BALD) (Gal et al., 2017) utilizes MC-dropout samples to approximate the mutual information between outputs and model parameters:
(BALD(X) = Σ (BALD(a")
(5)
Haonan Wang. Wei Huang, Ziwei Wu, Hanghang Tong. Andrew J Margenot, and Jingrui He. 2022. Deep active learning by leveraging training dynamics. Ad- vances in Neural Information Processing Systems, 35:25171-25184.
-[y, 0|0r, L]
(6)
Myle Ep(oce) Hyy 0]]
(7)
E( Σ
1
Nog Σ
Po.)
Dittaya Wanvarie, Hiroya Takamura, and Manabu Oku- mura. 2011. Active learning with subsequence sam- pling strategy for sequence labeling tasks. Informa- tion and Media Technologies, 6(3):680-700.
M
M
1,0
(8)
ΣΣ C,
Per logg't
Th
(9)
M
Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille. and Fan Yang. 2021. Crest: A class-rebaland rebalancin self- training framework for imbalanced semi-supervised learning. In Proceedings of the IEEE/CVF confer- ence computer vision and pultern recognition, pages 10857-10866.
where ý is the prediction of the model, M denotes MC-dropout samples. Different from the acquisi- tion functions mentioned above, which sum over all tokens, Maximum Normalized Log-Probability (MNLP) (Shen et al., 2017) normalized by the length of the sequence:
Chih-Hsuan Wei, Yifan Peng. Robert Leaman, Allan Pe- ter Davis, Carolyn J Mattingly, Jiao Li, Thomas Wiegers, and Zhiyong Lu. 2016. Assessing the state of the art in biomedical relation extraction: overview of the biocreative chemical-disease relation (cdr) task. Database, 2016.
1 (MNLP(x) = Σ logp(0"(a".0)
B Dataset statisties
Rongzhi Zhang, Yue Yu, and Chao Zhang. 2020. Se- qmix: Augmenting active sequence labeling via se- quence mixup. arXiv preprini 22Xi::2010.03322.
See table 3 for the statistical data for each dataset.
C Additional results
Guang Zhao, Edward Dougherty, Byung-Jus Yoon, Francis Alexander, and Xiaoning Qian. 2021. anwer active learning for optimal bayesian classifier. In International Conference on Learning Representations (ICLR 2021).
In this section, we present all the remaining experi- mental results that could not showcased the main paper due to the space limitation.

---

Average
Imbalance
Dataset
Sentences
Tokens
Class proportion (B/T/O)
length
ratio
Conl12003
14041
203,621
14.5
11.5%
19.7
WikiAnn
20000
160394
8.0
17.4%//31.9%%50.799
2.35
BC5CDR
5228
109322
20.6
8.6%/2..9%/8..5%
38.4
Table 2: Statistics for the entire training dataset of the three datasets used in our experiments.
3
0.01
0.1
0.2
0.5
Conl12003
84.28
84.54
84.45
84.33
83.37
WikiAnn 78.18
78.53
77.32
77.60
76.15
BC5CDR
75.50
75.98
75.90
75.63
75.51
Table 3: Final Fi-scores of re-weighting I.C. with different hyperparameter on each dataset.
Dataset: Conli2003, query size=30
0.8
F1-score
Random
Random
Random
Random
BALD
Entropy
MNLP
BALD
Entropy_nv
MNLP_MV
0.6
10
10
Itteaaiions
Dataset: WikiAnn, query size=30
0.80
0.70
Random
Random
Random
0.65
Entropy
BALD_rW
Entropy_nw
MNLP_IW
Dataset: BC5CDR, query size=30
F1-score
0.7
Random
Random
Random Entropy
Random
0.6
BALD
MNLP
BALD IW
LC_ rw
Entropy_rv Lw
MNLP rw
10
2
AL iterations
Figure 3: Pairwise comparison between with and without re-weighting for each baseline with query size 30.

---

Dataset: Conli2003, query size=50
0.90
0.85
0.80
Random
Random
Random
Random
BALD
Entropy
MNLP
0.70
BALD
rw
Entropy_me nw
MNLP
10
AL iterationn
Dataset: WikiAnn, query size=50
0.80
F3-score
0.75
Random
Random
Random
Random
BALD
LC
Entropy
MNLP
0.65
BALD rw
LC_ rw
Entropy nw
MNLP rw
2
10
2
8
10
10
2
10
AL iterations
Dataset: BCSCDR, query size=50
F1-score
0.75
0.70
Random
Random
Random
Random
BALD
Entropy
MNLP
BALD
Entropy_nv
MNLP_NV
10
Itteaaiion
Figure Pairwise comparison between with and without re-weighting for each baseline with query size 50.
Dataset: WikiAnn, query size=15
2.5
BALD
Entropy Entropy nw
MNLP MNLP rw
Imbalance Ratio
BALD
2.0
LC
Lc_rw
1.5
10
AL iterationn
Dataset: BC5CDR, query size=15 15
Imbalance Ratio
BALD
LC
Entropy Entropy rw
MNLP
BALD TW
LC, rw
MNLP rW
40
20
2
10
2
8
10
6
10
2
8
10
AL iterations
Figure 5: The variation of the imbalance ratio during the active learning process on WikiAnn and BC5CDR with query size 15.

---

Dataset: Conli2003, query size=30
Imbalance Ratio
BALD BALD_IW
Entropy
MNLP
lc_rw
Entropy_rw
MNLP_NV
iterations
Dataset: WikiAnn, query size=30
Imbalance Ratio
2.5
BALD
Entropy
MNLP
BALD_rW
Entropy_rw
MNLP_TW
1.5
2
10
2
10
10
2
10
AL iterations
Dataset: BC5CDR, query size=30
Imbalance Rabio
BALD
BALD
rw
Entropy Entropy rw
MNLP TW
2
10
10
10
2
AL iterations
Figure 6: The variation of the imbalance ratio during the active learning process on three datasets with query size 30
Dataset: Conll2003, query size=50
Imbalance Rabio
BALD
Entropy Entropy_nv
MNLP
BALD
LC
MNLP
20
15
10
5
2
10
2
10
10
2
10
AL iterations
Dataset: WikiAnn, query size=50
3.0
Imbalance Rabio
Entropy
2.5
Entropy_nv
2.0
1.5
iteeationn
Dataset: BCSCDR, query size=50
Imbalance Ratio
30
BALD
LC
Entropy Entropy_nv
MNLP MNLP
BALD
Lc_rw
20
10
2
AL iterations
Figure 7: The variation of the imbalance ratio during the active learning process on three datasets with query size 50


Reviews:
[{"review_id": "6UuHsX1KTz", "reviewer": "Reviewer_9oCT", "summary": "", "questions": "The experimentation lacks diversity in the number of classes for the datasets. Introducing results for Twitter or Ontonote datasets would strengthen the argument. \n\nMore studies on the hyperparameter \"beta\" in the experimental section would provide better guidance in selecting an appropriate beta value.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer-9oCT for the constructive feedback. We provide a detailed response below.\n\n**Question 1: The experimentation lacks diversity in the number of classes for the datasets. Introducing results for Twitter or Ontonote datasets would strengthen the argument.**\n\nTo address this concern, we conducted additional experiments on Ontonotes5.0 with a query size of 15 and reported the average final F1-score for three independent runs after 10 active learning iterations. It can be observed that the re-weighting method can consistently improve the performance over different baselines. In the revised version of the paper, we are open to including the learning curves for different batch sizes on Ontonotes5.0, both in the main text and in the appendix, for a more comprehensive understanding.\n\n|         | Bald  | LC    | Entropy | MNLP  | Random |\n|---------|-------|-------|---------|-------|--------|\n| With rw | **76.34** | **76.73** | **77.38**   | **77.50** | 59.42  |\n| w/o rw | 73.65 | 76.44 | 76.45   | 76.76 |        |\n\n**Question 2: More studies on the hyperparameter \"beta\" in the experimental section would provide better guidance in selecting an appropriate beta value.**\n\nWe acknowledge the absence of study on hyperparameters as a limitation on our part, and we intend to incorporate this aspect in the final version. We suggest, for each dataset, tuning the hyperparameter with a specific base acquisition function and fixing it for other acquisition functions, using the final F1-score on the validation set as a metric. Because the optimal $\\beta$ is determined by the imbalanced nature of the current dataset, it can be shared across different acquisition functions. We report the final F1-score for three independent runs after 10 active learning iterations under various hyperparameter settings of $\\beta$ on different datasets with batch size=15. Based on the experiments, our method is not particularly sensitive to the choice of hyperparameters. In addition, we found $\\beta=0.1$ to be a solid default value, it can work even without excessive hyperparameter tuning.\n\nConll2003:\n| Beta  | 0.01  | 0.1   | 0.2   | 0.5   | 1     |\n|-------|-------|-------|-------|-------|-------|\n| LC_rw | 84.28 | **84.54** | 84.45 | 84.33 | 83.37 |\n\nBC5CDR:\n| Beta  | 0.01  | 0.1   | 0.2   | 0.5   | 1     |\n|-------|-------|-------|-------|-------|-------|\n| LC_rw | 75.50 | **75.98** | 75.90 | 75.63 | 75.51 |\n\nOntonotes5.0:\n| Beta  | 0.01  | 0.1   | 0.2   | 0.5   | 1     |\n|-------|-------|-------|-------|-------|-------|\n| LC_rw | 76.96 | 77.92 | **78.50** | 78.19 | 77.66 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The article is well-written, logically structured, and easily understandable. \n\nThe method is straightforward and has broad applicability. ", "reasons_to_reject": "The choice of hyperparameter \"beta\" is very important. The experiment lacks studies and guidelines for how to choose beta properly.", "questions_to_authors": "The experimentation lacks diversity in the number of classes for the datasets. Introducing results for Twitter or Ontonote datasets would strengthen the argument. \n\nMore studies on the hyperparameter \"beta\" in the experimental section would provide better guidance in selecting an appropriate beta value.", "paper_topic_and_main_contributions": "This paper proposes an active learning strategy, \"reweighting tokens,\" suitable for Named Entity Recognition (NER). The strategy is simple yet widely applicable. The authors conducted experiments to demonstrate how this approach effectively addresses class imbalance issues and improves NER model performance. ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hji48H25gD", "reviewer": "Reviewer_Mdb7", "summary": "", "questions": "a.\tThe hyper-parameter Beta is very important in the re-weighting method. Experimental results on the hyper-parameter beta also should be provided. It would be better to provide an automatic method to optimize the hyper-parameter Beta in the applications.  \n\nb.\tMore details on the experiments should be provided, such as the distribution of each NER type, the precision, recall and F of each type in the active learning process.  \n\nc.\tTable 1 only provides F1-scores in the first three AL iterations on Conll2003 with query size 15.  How about the performance in the other iterations?\n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer-Mdb7 for your constructive suggestions. We provide a detailed response below.\n\n**Question 1: The hyper-parameter Beta is very important in the re-weighting method. Experimental results on the hyper-parameter beta also should be provided. It would be better to provide an automatic method to optimize the hyper-parameter Beta in the applications.**\n\nWe acknowledge the absence of study on hyperparameters as a limitation on our part, and we intend to incorporate this aspect in the final version. We suggest, for each dataset, tuning the hyperparameter with a specific base acquisition function and fixing it for other acquisition functions, using the final F1-score on the validation set as a metric. Because the optimal $\\beta$ is determined by the imbalanced nature of the current dataset, it can be shared across different acquisition functions. We report the final F1-score for three independent runs after 10 active learning iterations under various hyperparameter settings of $\\beta$ on different datasets with batch size=15. Based on the experiments, our method is not particularly sensitive to the choice of hyperparameters. In addition, we found $\\beta=0.1$ to be a solid default value, it can work even without excessive hyperparameter tuning. \n\nWe acknowledge that an automated approach to determine the optimal value for Beta would likely yield better results, especially when extending the application of our model to diverse corpora with varying class distributions. We also believe that dynamically updating Beta in line with Active Learning iterations could be a promising avenue for future research.\n\nConll2003:\n| Beta  | 0.01  | 0.1   | 0.2   | 0.5   | 1     |\n|-------|-------|-------|-------|-------|-------|\n| LC_rw | 84.28 | **84.54** | 84.45 | 84.33 | 83.37 |\n\nBC5CDR:\n| Beta  | 0.01  | 0.1   | 0.2   | 0.5   | 1     |\n|-------|-------|-------|-------|-------|-------|\n| LC_rw | 75.50 | **75.98** | 75.90 | 75.63 | 75.51 |\n\nOntonotes5.0:\n| Beta  | 0.01  | 0.1   | 0.2   | 0.5   | 1     |\n|-------|-------|-------|-------|-------|-------|\n| LC_rw | 76.96 | 77.92 | **78.50** | 78.19 | 77.66 |\n\n**Question 2:  More details on the experiments should be provided, such as the distribution of each NER type, the precision, recall and F of each type in the active learning process.**\n\nWe report class frequencies for each NER type and the final F1-score for three independent runs for each class on Conll2003 with a query size of 15, using least confidence as our base acquisition function. It can be observed that the re-weighting version can significantly improve the performance of the minority classes without overly affecting the performance of the majority classes. This can explain why it improves the overall performance. We will include these details on the experiments in the final version.\n\n|            | O      | B-LOC  | I-LOC  | B-PER  | I-PER  | B-MISC | I-MISC | B-ORG  | I-ORG  |\n|------------|--------|--------|--------|--------|--------|---------|--------|--------|--------|\n| Class proportion | 0.833  | 0.035  | 0.0057 | 0.0324 | 0.0222 | 0.0169  | 0.0057 | 0.0310 | 0.0182 |\n| LC_rw      | **98.84** | **82.30** | **70.94** | 92.77 | 95.95 | 69.97  | **63.51** | **80.23** | **80.65** |\n| LC         | 98.67 | 79.66 | 64.82 | **93.56** | **97.16** | **71.39**  | 59.89 | 76.86 | 79.24 |\n\n**Question 3: Table 1 only provides F1-scores in the first three AL iterations on Conll2003 with query size 15. How about the performance in the other iterations?**\n\nWe report a comparison of the final F1-score for three independent runs between the smoothed and unsmoothed versions on Conll2003 with batch size=15, using least confidence as our base acquisition function. It can be observed that although the final performances are close, the unsmoothed version exhibited poorer performance and higher fluctuations during the early iterations of active learning, aligning with the purpose of introducing a smoothing function to counteract the high variance in the early stages of active learning.\n\n| iteration | 1     | 2     | 3     | 4     | 5     | 6     | 7     | 8     | 9     | 10    |\n|-----------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|\n| +smooth   | 67.43 | **77.02** | **79.93** | 80.86 | **81.64** | **82.64** | **81.56** | **83.22** | **82.47** | **83.46** |\n| no smooth   | **68.73** | 73.96 | 76.83 | **81.15** | 80.46 | 80.07 | 81.32 | 82.45 | 82.37 | 82.97 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Data imbalance is a very big challenge in NER applications.  \n\nThis paper presents a reweighting-based active learning strategy for active sequence tagging to tackle the data imbalance issue. The proposed method assigns dynamic smoothed weights to individual tokens. This adaptable strategy is compatible with various token-level acquisition functions.  \n\nExperimental results on several datasets demonstrate the performance enhancement on NER task.", "reasons_to_reject": "The hyper-parameter Beta is very important in the re-weighting method. Experimental results on the hyper-parameter beta also should be provided. It would be better to automatically optimize the hyper0parameter Beta in the applications.  \n\nMore details on the experiments should be provided, such as the distribution of each NER type, the precision, recall and F of each type in the active learning process.   \n\nTable 1 only provides F1-scores  in the first three AL iterations on Conll2003 with query size 15.  How about F1 in the other iterations (i.e. 4-10 iteration)?\n", "questions_to_authors": "a.\tThe hyper-parameter Beta is very important in the re-weighting method. Experimental results on the hyper-parameter beta also should be provided. It would be better to provide an automatic method to optimize the hyper-parameter Beta in the applications.  \n\nb.\tMore details on the experiments should be provided, such as the distribution of each NER type, the precision, recall and F of each type in the active learning process.  \n\nc.\tTable 1 only provides F1-scores in the first three AL iterations on Conll2003 with query size 15.  How about the performance in the other iterations?\n", "paper_topic_and_main_contributions": "Data imbalance is a very big challenge in NER applications.  This paper presents a reweighting-based active learning strategy for active sequence tagging to tackle the data imbalance issue. The proposed method assigns dynamic smoothed weights to individual tokens. This adaptable strategy is compatible with various token-level acquisition functions.  Experimental results on several datasets demonstrate the performance enhancement on NER task.\n\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5Is1pMbDZv", "reviewer": "Reviewer_xutz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer-xutz for the insightful comments and suggestions. We provide a detailed response below.\n\n**Question 1: The paper does not compare the proposed approach with other state-of-the-art active learning strategies for NER, which could provide a more thorough evaluation of its effectiveness.**\n\nAnswer 1: We would like to clarify that our study was designed to address sentence-level imbalances and has been compared with leading sentence-level active learning strategies for NER like LC, ENT, MNLP, and BALD. We mentioned other recent state-of-the-art active learning strategies for NER in the first section of the appendix, such as Lowest Token Probability (LTP) [1] and sub-sequence based methods [2] [3]. However, LTP requires the decoder to be conditional random fields, which limits its applicability; while [2] [3] query only a portion of the sentence. These facts make direct comparisons with them infeasible. Studying how to incorporate reweighting mechanisms into these algorithms could be a future direction.\n\n[1] Liu, M., Tu, Z., Wang, Z., & Xu, X. (2020). LTP: a new active learning strategy for BERT-CRF based named entity recognition. arXiv preprint arXiv:2001.02524.\n\n[2] Radmard, P., Fathullah, Y., and Lipani, A. Subsequence based deep active learning for named entity recognition. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4310–4321, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.332\n\n[3] Liu, Y., Hu, J., Chen, Z., Wan, X., & Chang, T. H. (2023, June). EASAL: Entity-Aware Subsequence-Based Active Learning for Named Entity Recognition. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 7, pp. 8897-8905).\n\n**Question 2: The authors do not provide a detailed discussion of the limitations or potential drawbacks of the proposed approach.**\n\nAnswer 2: We enumerate some limitations of our research here and will include them in the final version of the paper:\n- One of the limitations of our paper is related to the hyperparameter Beta in our reweighting algorithm. We agree that the static nature of Beta could be a drawback, particularly when applying the algorithm to new or different corpora. Furthermore, Beta may also be sensitive to the number of classes in the dataset.\n- We acknowledge that an automated approach to determine the optimal value for Beta would likely yield better results, especially when extending the application of our model to diverse corpora with varying class distributions. We also believe that dynamically updating Beta in line with Active Learning iterations could be a promising avenue for future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposes a novel active learning strategy for named entity recognition that addresses the challenge of data imbalance and demonstrates its effectiveness through experimental results. ", "reasons_to_reject": "- The paper does not compare the proposed approach with other state-of-the-art active learning strategies for NER, which could provide a more thorough evaluation of its effectiveness.\n- The authors do not provide a detailed discussion of the limitations or potential drawbacks of the proposed approach.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a novel active learning strategy that assigns dynamic smoothed weights to individual tokens, contributing to the development of robust active learners for named entity recognition. The paper addresses the challenge of data imbalance in NER and proposes a re-weighting-based active learning strategy that improves the performance of active sequence tagging. The experimental results demonstrate substantial performance improvement, validating its practical efficacy. ", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper is well written."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a reweighting-based active learning strategy for sequence tagging, with a focus on named entity recognition (NER) tasks. While the paper exhibits strengths, there are important aspects that need attention and further clarification. Below I summarize the most relevant aspects to be considered:
Reasons to Accept:
* 		Clarity and Readability: The paper is well-written, logically structured, and easily understandable. The presentation of the proposed method is clear, contributing to the overall comprehensibility of the research.
* 		Broad Applicability: The method presented in the paper is straightforward and exhibits potential for broad applicability. Its compatibility with various token-level acquisition functions is a valuable feature, demonstrating versatility.
* 		Innovative Active Learning Strategy: The paper introduces a novel active learning strategy for NER that effectively addresses the challenge of data imbalance. The proposed dynamic smoothed weights for tokens represent an adaptable approach that holds promise for improving sequence tagging tasks.
* 		Empirical Validation: The paper substantiates its claims through experimental results on multiple datasets, showcasing performance enhancements in the context of NER tasks. This empirical validation enhances the credibility of the proposed approach.
Reasons to Reject:
* 		Limited Hyperparameter Analysis: The absence of a thorough study on hyperparameters, specifically the importance of Beta in the re-weighting method, is a notable limitation. It is essential to explore the sensitivity of the method to hyperparameter variations and provide insights into the optimal choices for different scenarios.
* 		Lack of Hyperparameter Optimization: The paper could benefit from experimental results and discussions related to the hyperparameter Beta. Automatic optimization or at least a systematic exploration of this hyperparameter's impact on performance would strengthen the method's practical utility.
* 		Static Nature of Beta: The authors acknowledge the static nature of the Beta hyperparameter, which may pose limitations when applying the algorithm to new or diverse corpora. Further investigation into the adaptability of Beta and its sensitivity to dataset characteristics, such as the number of classes, is essential.
* 		Additional Experiment Details: While the paper presents experimental results demonstrating overall performance enhancement, it could be further strengthened by providing more detailed metrics, including the distribution of each NER type and precision, recall, and F1 scores for each type throughout the active learning process. Such details would offer a deeper understanding of the method's effectiveness.
In conclusion, while the paper introduces an active learning strategy for NER with several strengths, including clarity, applicability, and empirical validation, there are areas that require attention. These areas were discussed during the rebuttal period and there was convergence on the opinions. Thus, more specifically, addressing hyperparameter sensitivity, exploring automatic hyperparameter optimization, and providing additional experiment details would enhance the paper's overall quality and practicality.
Decision: Accept-Findings