Combining Counting Processes and Classification Improves a Stopping Rule for Technology Assisted Review
Authors: Reem Bin Hezam
Mark Stevenson
Venue: EMNLP 2023 Findings (2023)
Keywords: Technology assisted review
TAR
total recall
stopping criteria
counting processes
classification
TL;DR: Combining Counting Processes and Classification Improves a Stopping Rule for Technology Assisted Review

Abstract: Technology Assisted Review (TAR) stopping rules aim to reduce the cost of manually assessing documents for relevance by minimising the number of documents that need to be examined to ensure a desired level of recall. This paper extends an effective stopping rule using information derived from a text classifier that can be trained without the need for any additional annotation. Experiments on multiple data sets (CLEF e-Health, TREC Total Recall, TREC Legal and RCV1) showed that the proposed approach consistently improves performance and outperforms several alternative methods.

Combining Counting Processes and Classification Improves a Stopping Rule for Technology Assisted Review
Reem Bin-Hezam 3.5 and Mark Stevenson
*Department of Computer Science, University of Sheffield, United Kingdom Information Systems Department, Princess Nourah bint Abdulrahman University, Saudi Arabia rybinhezam@pnu. edu. sa, mark. www.ssseeensonseeeiillleccc ac.
Abstract
et al., 2014; Howard et al., 2020; Callaghan and Müller-Hansen 2020). However, these approaches may not account for the fact that most relevant documents appear early in the ranking, informa- tion shown to be useful for stopping algorithms (Cormack and Grossman, 201 Li and Kanoulas, 2020), so they may examine more documents than necessary.
Technology Assisted Review (TAR) stopping rules aim to reduce the cost of manually assess- ing documents for relevance by minimising the examined
number of documents that need ensure desired level of recall. This paper extends an effective stopping rule using infor- mation derived from text classifier that can be trained without the need for any additional annotation. Experiments on multiple data sets (CLEF e-Haalth, TREC Total Recall, TREC Legal and RCVI) showed that the proposed ap- proach consistently improves performance and outperforms several alternative methods.'
Counting Processes Counting processes, stochas- tic models of the number of occurrences of an event within some interval (Cox and Isham, 1980), can naturally model changes to the frequency of rele- vant document occurances within ranking. They have been used develop stopping rules that en- sure that a desired level of recall is reached while also minimising the number documents that need to be reviewed and found to outperform range of alternative approaches (Sneyd and Stevenson, 2019, 2021; Stevenson and Bin-Hezam, to appear). However, in these methods the estimate of the to- tal number of relevant documents is only based the examined initial portion of the ranking and no information is used from the remaining documents. Classification Another approach has been to text classifier estimate the total number of rele- vant documents (Yu and Menzies, 2019; Yang al., 2021a). This method has the advantage of using information about all documents in the ranking by observing the classifier's predictions for the docu- ments that have not yet been examined. However, the classifier alone does not consider model the occurrence rate at which relevant documents have already been observed. These methods were found to be effective, although each was only tested on a single dataset.
1 Background
Information Retrieval (IR) systems often return large numbers of documents in response user queries and screening them for relevance repre- sents a significant effort. This problem particu- larly acute in applications where it is important to identify most (or all) of the relevant documents, for example, systematic reviews of scientific literature (Higgins et al., 2019) and legal eDiscovery (Oard al., 2018). Technology Assisted Review (TAR) develops methods that aim to reduce the effort re- quired screen collection of documents for rele- vance, such as stopping rules that inform reviewers that a desired level of recall has been reached (and no further documents need to be examined) (Li and Kanoulas, 2020; Yang et al., 2021a).
One of the most commonly applicd approaches developing stopping rules is to identify when particular level of recall has been reached by estimating the total number of documents in the collection (either directly or indirectly). The ma- jority of algorithms using this approach operate by obtaining manual judgements for a sample of the unobserved documents and inferring the to- tal number of relevant documents, e.g. (Shemilt
This paper proposes extension to stopping algorithms based on counting processes by using text classifier to inform the estimate of the tal number of relevant documents. This approach makes use of information about the relevance of documents from the entire ranking without increas-
Coode to reported results available from https://githhb. Wtt.eeeeiiiieeeaaaTAAaSSeaee

---

ing the number of documents that need to be exam- ined for relevance, since the text classifier trained using information already available.
provides helps ensure that the rate function is appropriate and that the estimate of the total num- ber of relevant documents provided by the counting process accurate.
2
Approach
The text classifier is integrated into the stopping algorithm in a straightforward way. We begin by assuming that the ranking contains a total of doc- uments. As before, relevance judgements are ob- tained for the first n. These judgements are used to train a text classifier which is applied to all remain- ing documents without relevance judgements (i.e. ... N). The rate function is now fitted by examining all N documents in the ranking (rather than only the first n), using manual relevance judge- ments for the first and the classifier's predictions for the remainder (i.e. ni + ... N). As before, the algorithm stops has been estimated that enough relevant documents are contained within the first to achieve the desired recall. Otherwise, the value of is increased and the process repeated. See Figure 1.
We begin by describing the existing counting pro- cess approach and then explain how the classifier is integrated.
Counting Process Stopping Rule (Sneyd and Stevenson, 2019, 2021) The approach starts by obtaining relevance judgements for the n highest ranked documents. A rate function is then fitted to model the probability of relevant documents being encountered in the (as yet) unexamined documents later in the ranking. The counting process uses this rate function estimate the total number of rele- vant documents remaining. Based on this estimate, the algorithm stops if enough relevant documents have been found to reach a desired recall. If not then more documents are examined and the process repeated until either the stopping point is reached or all documents have been examined. Fitting an appropriate rate function is an important factor in the success of this approach. A set of sample points are extracted from the documents for which the rel- evance judgements are available (i.e. top n),, and the probability of document being relevant at each point is estimated. This set of points then provided to a non-limear least squares curve fitting algorithm to produce the fitted rate function. For each sample point, the probability of document being relevant is computed by simply examining window of documents around and computing the proportion that is relevant, i.c.
start
Obtain relovance judgements first (of N) ranked documents (1....n)
Train classifier using documents 1...n
Use classifier to predict relevance of unexamined documents (n+1...N)
Σ 1(d)
(1)
Fit rate function using all documents (1...N)
dEW
where W is the set of documents in the window and 1 (d) returns 1 if has been labelled as relevant. Integrating Text Classification Since the exist- ing counting process approach only examines the top-ranked documents, relies on the assumption that the examined portion of documents provides a reliable indication of the rate at which relevant documents occur within the ranking. However, this may not always be the case, particularly when only small number of documents have been manually examined when relevant documents unexpect- edly appear late the ranking. This problem can be avoided by applying a text classifier to the unex- amined documents at each stage. The information
Apply counting process to estimate total number relevant documents
sample relevance 2 recget
N
stop
Figure Counting Process with Classifier Approach

---

It is possible use information from the classi- fier in various ways when computing the probabil- ity of document relevance during the curve fitting process. Two were explored:
CLEF datasets, result consistent with previous lindings for TAR models (Yang al., 2022; Sadri and Cormack, 2022).
TAR problems are often highly imbalanced, with very few relevant documents. However, TAR meth- ods successfully place relevant documents early in the ranking (Cormack and Grossman, 2015, 2016b; Li and Kanoulas, 2020). Consequendy, the preva- lence of relevant documents in the training data is unlikely to be representative of the entire collection, particularly early the ranking. To account for this cost-sensitive learning was used during classifier training. The weight of the minority class within the training data was set to 1, while the weight of the majority class was set to the imbalance ratio (i.e. count of the minority over the majority class in training data) (Ling and Sheng, 2008)
ClassLabell the class labels output by the classi- fier are used directly, i.c. using eq. I with relevance determined by the classifier.
ClassScore: Alternatively, the scores output by the classifier is used, subject threshold of 0.5,
i.e. eq. 1 is replaced by
score(d) if score(d) 2 0.5
(2)
72
otherwise
deW
where score(d) the score the classifier as- signed to (Using the class score without thresh- olding was found to overestimate the number of relevant documents.)
3.1 Datasets
3 Experiments
Performance was evaluated on six diverse datasets widely used in previous work on TAR. CLEF e-Health (CLEF2017, CLEF2018, CLEF2019) (Kanoulas et al., 2017, 2018, 2019): A collection of systematic reviews from the Conference and Labs of the Evaluation Forum (CLEF) 2017, 2018, and 2019 e- e-Health lab Task 2: Technology- Assisted Re- views in Empirical Medicine. The datasets contain 42, 30, and 31 reviews respectively. TREC Total Recall (TR) (Grossman et al., 2016) A collection of 290,099 emails with 34 topies related to Jeb Bush's 's eight-year tenure as Governor of Florida (athome4). TREC Legal (Legal) (Cormack et al., 2010) A collection of 685,592 595 Enron emails made available by the Federal Energy Review Commis- sion during their investigation into the company's collapse. RCV1 (Lewis et al., 2004) A collection of Reuters news articles labelled with categories. Following previous work (Yang et al., 2021b,a), 45 categories were used to represent range of topies, and the collections downsampled to 20%.
Experiments compared three approaches. CP:
Baseline counting process approach without text classification, i.c. (Sneyd and Stevenson, 2021). CP+ClassLabel and ClaCsasScore:: Counting process combined with the text classifier (see $2). The counting process uses the most effective ap- proach previously reported (Sneyd and Stevenson, 2021). A power law (Zobel, 1998) was used as the rate function with an inhomogencous Poisson process since its performance is comparable with Cox process while being more computationally efficient. Parameters were set as follows: the confi- dence level was set to 0.95 while the sample size was initialised to 2.5% and incremented by 2.5% for each subsequent iteration. An existing reference implementation was used (Sneyd and Stevenson, 2021).
The text classifier uses logistic regression which has been shown be effective for TAR problems (Yang et al., 2021a; Li and Kanoulas, 2020). The classifier was based on scikit-leamd using TF-IDF scores of each document's 's content as features, the content title and abstract for CLEF datasets, email messages for TREC datasets (including title and message body), and news articles content for the RCVI dataset. The classification threshold was set to 0.5 (model default) by optimising F-measure for relevant documents using held-out portion of the dataset. Text classifiers based on BioBERT (Lee al., 2020) and PubMedBERT (Gu et al., 2020) were also developed. However, neither out- performed the linear logistic regression model on
The ranking used by the stopping model needs to be the same for all datasets used in the experi- ments to ensure fair and meaningful comparison. The rankings produced by AutoTAR (Cormack and Grossman, 2015) were used since they allow com- parison with a range of alternative approaches (Li and Kanoulas, 2020) and can be generated using their reference implementation.
3.2 Evaluation Metrics
Approaches were evaluated using the metrics com- monly used in previous work on TAR stopping

---

Table 1: Results for Counting Process and Classification Stopping Methods. (* indicate scores are Nor statistically significantly different from CP, indicate CP+ClassLabel and CP+ClassScore are statistically significantly different)
Desired recall
Desired recall
Desired recall 0.7
Dataset
Model
Recall
Rel.
Cost
Excess
Recall
Rel.
Cost
Excess
Rocall
Rel.
Cost
Excess
CP
1.000
1,000
0.281
0.238
1.000
1,000
0.265
0.232
1.000
1,000
0.255
0.229
CLEF2017
CP+ClastLabel
0.989
1,000
0.153
0.102
0.989
1.000
0.152
0.114
0.988
1,000
0.150
0.120
CP+ClasScore
0.989
1.000
0.152
0.101
0.989
1,000
0.152
0.114
0.988
1,000
0.147
0.117
CP 0.983
1.000
1,000
0.293
0.242
1,000
1.000
0.287
0.249
1.000
1.000
0.277
0.245
CLEF2018
CPoClastL.abel
1,000
0.137
0.075
0.983
1,000
0.137
0.091
0.982
1,000
0.135
0.097
CPCClasSoore
0.983
1,000
0.137
0.075
0.982
1,000
0.136
0.090
0.981
1,000
0.134
0.096
CP 0.999
1,000
0.283
0.228
0.999
1,000
0.279
0.235
0,999
1.000
0.276
0.240
CLEF2019
Ca+elassLabel
0.996
1,000
0.221
0.161
0.996
1,000
0.216
0.169
0.996
1.000
0.212
0.173
Cl+ClasScore
0.996
1.000
0.221
0.161
0.996
1,000
0.213
0.165
0.994
1.000
0.207
0.168
CP 1.000
1,000
0.425
0.401
1,000
1,000
0.338
0.320
1.000
1.000
0.287
0.273
Legal
ClaClastLabel
0.972
1,000
0,088
0.050
0.972"
1,000
0.088
0.0664
0.972
1.000
0.088
0.070
CP+ClasScore
0.972
1,000
0,088
0.050
0.972
1,000
0.088
0.064
0.963'
1,000
0.075
0.057
CP 1,000
1,000
0.059
0.054
1,000
1,000
0.056
0.052
1,000
1.000
0.052
0.049
TR
CP+ClassLabel
0,999
1,000
0.028
0.023
0.999
1,000
0.027
0.023
0.999'
1.000
0.027
0.024
CloClasScore
0.959
1,000
0.028
0.023
0.999'
1,000
0.027
0.023
0.999'
1.000
0.027
0.024
CP 0.999
1,000
0.193
0.180
0.998
1,000
0.154
0. 0.145
0.998
1.000
0.134
0.127
RCVI
CP+ClassLabel
0,972
0.956
0,038
0.022
0.969
1.000
0.036
0.026
0.969
1.000
0.036
0.028
CP+ClasScore
0.969
0.933*
0.036
0.020
0.969
1,000
0.036
0.026
0.969
1.000)
0.036
0.028
criteria, e.g. (Yang et al., 2021a; Li and Kanoulas, 2020), and calculated using the tar_eval open- source evaluation script. 2 Average scores across all topics in each collection are reported.
Results and Analysis
Experiments were carried out using a range of de- sired recalls (0.9, 0.8, 0.7} with results shown in Table 1. Results show that combining the classi- fier with the counting process (CP+ClassLahel) and CP+ClassScore) is more effective than using the counting process alone. The improvement in the performance of both approaches compared with CP is statistically significant across topics (p < 0.05, paired t-test with Bonferroni correction). There is a significant reduction in cost which is often sub- stantial (c.g. Legal collection with desired recall of 0.9). This is achieved with a minimal reduction to the number of relevant documents identified, although the reliability remains unaffected in the majority of cases, and when it is affected (RCV1 collection with desired recall of 0.9), the reduc- tion is minimal and not statistically significant.
proportion of relevant documents identified by the method. Following Li and Kanoulas (2020), results closest to the desired recall are considered best, rather than the highest overall recall.
Recall:
Reliability (Rel.): percentage of topics where the desired recall was reached (or exceeded). For each topic, reliability is 1 if the desired recall was reached, and otherwise.
Cost: percentage of documents examined.
Excess cost (Exeess): we introduce this measure which quantifies the additional documents that have to be examined compared with optimal stopping (i.e. stopping immediately when the desired recall has been reached). It is computed as follows:
The performance of CP+ClassScore and LaP are comparable. The scores for CP+ClassScore may be marginally better, although the difference is only significant in limited circum- stances. (Differences in the cost and excess scores were significant for the CLEF 2019 collection with desired recalls of 0.8 and 0.7.) Overall, the way in which the classifier output is integrated into the approach seems less important than the fact it is used all. The average recall for all approaches ex- ceeds the desired recall, indicating a tendency to be somewhat conservative in proposing the stopping
ezcess cost- cost (oethmo)ood) cost 1 - cost 1--oos(optimmg coptimpti
(3) where (metho))h and (optimaa)))) are the cost of the method being evaluated and stopping at the optimal point, respectively. This metric in- dicates the proportion of the documents that need to be examined after the desired recall has been reached.

---

Table 2: Proposed Method vs, Baselines (TR collection)
Desired recall 0.9
Desired recall 0.8
Model
Recall
Rel.
Cost
Excess
Recall
Rel.
Cost
Excess
CP+Classcoore
0.999
1.000
0.028
0.023
0.999
1.000
0.027
0.023
SCAL (Cormack and Grossman, 2016b)
0,903
0.647
0.144
0.140
0.761
0.676
0.107
0.103
SD-training (Hollmann and Fickhoff, 2017)
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
SD-sampling (Hollmann and Eickhoff, 2017)
0,936
0.794
0.779
0.778
0.896
0.735
0.690
0.689
AutoStop (Li and Kanoulas, 2020)
0,953
0.941
0.766
0.765
0,885
0.912
0.754
0.753
Target (Cormack and Grossman, 60)
0.900
0.706
0.069
0.064
0.844
0.882
0.069
0.065
point. However, the classifier allows the algorithm to identify this point earlier, presumably because it has indicated a low probability of relevant docu- ments being found later in the ranking.
target method (Cormack and Grossman, 2016a) are also reported. Table 2 compares the proposed approach against these baselines for the TR col- lection for the desired recall levels for which re- sults are available. (Results for other datasets were similar and available in Li and Kanoulas (2020).) Cl+ClassSoore has lower cost than the baselines, requiring only fraction of the documents to be ex- amined, while achieving higher recall. Although the average recall of some baselines is closer the desired recall than the proposed approach, their reliability is also lower which indicates that they failed to identify enough relevant documents to achieve the desired recall for multiple topics.
4.1 Effect of Cost-sensitive Learning
The effect of using cost-sensitive learning during classifier training was explored by comparing it with the performance obtained when it was not used, see Table 3. These results demonstrate the importance of accounting for class imbalance when training the classifier. The low prevalence of rel- evant documents can lead to the classifier being unable to identify them resulting in the approach stopping too early, particularly when the desired recall is high.
5 Conclusion
Table 3: Effect of Cost-Sensitive Learning (CSL) for CP+ClassLabel Model applied to Legal collection
This paper explored the integration of a text classi- fier into an existing TAR stopping algorithm. Ex- periments on six collections indicated that the pro- posed approach was able to achieve the desired re- call level with a statistically significant lower cost than the existing method based on counting pro- cesses alone. Integrating the classifier output in the form of labels or scores made little difference, with improvements observed using either approach. The text classifier provides the stopping algorithm with information about the likely relevance of the docu- ments that have not yet been manually examined, allowing it to better estimate the number remaining.
Desired recall
CSL
Recall
Rel.
Cost
Excess
0.9
x
0.793
0.000
0.025
-0.016
v
0.972
1.000
0.088
0.050
X 0.793
0,500
0.025
-0,001
0.8
v 0.972
1.000
0.088
0.064
0.7
x
0.793
1.000
0.025
0.006
v
0.972
1.000
0.088
0.070
4.2 Baseline Comparison
Direct comparison of stopping methods is made difficult by the range of rankings and evaluation metrics used previous work. However, Li and Kanoulas (2020) reported the performance of sev- eral approaches using the same rankings as our ex- periments, allowing benchmarking against several methods. SCAL (Cormack and Grossman, 2016b), and AutoStop (Li and Kanoulas, 2020) sample doc- uments from the entire ranking to estimate the number of relevant documents, SD-training/SD- sampling (Hollmann and Eickhoff, 2017) use the scores assigned by the ranking. Results for the
Limitations
We presented stopping rule for ranked lists. Com- parison against multiple datasets requires the same ranking model for all datasets, which may not al- ways be available. In addition, reported results have been limited single rate function (power law) and confidence level (0.95). However, the pat- tern of results for other hyperparameters is similar to those reported.

---

Ethics Statement
Systematic Reviews of Interventions. John Wiley & Sons.
This work presents a TAR method designed to re- duce the amount of effort required to identify in- formation of interest within large document col- lections. The most common applications of this technology are literature reviews and identification of documents for legal disclosure. We do not fore- see any ethical issues associated with these uses.
Noah Hollmann and Carsten Eickhoff. 2017. Rank- ing and Feedback-based Stopping for Recall-Centric Document Retrieval. Working Notes of 2017 Conference and Labs of the Evaluation Fo- rum, pages 7-8.
CLEF
Brian E. Howard, Jason Phillips, Arpit Tandon, Adyasha Maharana, Rebecca Elmore, Deepak Mav, Alex Sedykh, Kristina Thayer, B. Alex Merrick, Vickie Walker, Andrew Rooney, and Ruchir R. Shah. 2020. Swift-Active Screener: Accelerated Docu- ment Screening Through Active Learning and Inte- grated Recall Estimation. Environment International. 138:105623.
References
Max W Callaghan and Finn Müller-Hansen. 2020. Sta- tistical Stopping Criteria for Automated Screening in Systematic Reviews. Systematic Reviews, 9(1):1-14.
Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker. 2017. CLEF 2017 Technologically Assisted Reviews in Empirical Medicine Overview. In CEUR workshop proceedings, volume 1866.
Gordon Cormack and Maura Grossman. 2015. Au- tonomy and Reliability of Continuous Active Learn- ing for Technology-Assinnnee Assisted Review. arXiv preprint arXiv: Sodi:v/00400866.
Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker. 2018. CLEF 2018 Technologically Assisted Reviews in Empirical Medicine Overview. In CEUR workshop proceedings. volume 2125.
Gordon V. Cormack and Maura R. Grossman. 2016a. Engineering quality and reliability in technology- assisted review. Proceedings of the 39th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, pages 75-84, Association for Computing Machinery, Inc.
Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker. 2019. CLEF 2019 Technology Assisted Re- views in Empirical Medicine Overview. In CEUR workshop proceedings, volume 2380.
Gordon V Cormack and Maura R Grossman. 2016b. Scalability of Continuous Active Learning for Reli- able Bigh-eecall Text Classification. In Proceedings of the 25th ACM International Conference on Infor- mation and Knowledge Management, pages 1039- 1048.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jacwoo Kang, 2020. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4): 1234-120..
Gordon V. Cormack, Maura R. Grossman, Bruce Hedin, and Douglas W. Oard, 2010. Overview the TREC 2010 Legal Track. In Proceedings of The Nineteenth Text REtrieval Conference, TREC 2010, Gaithers- burg, Maryland, USA, November 16-19, 2010, vol- ume 500-294 of NIST Special Publication. National Institute of Standards and Technology (NIST).
David D Lewis, Yiming Yang, Tony Russell- -ose,, and Fan Li. 2004. RCVI: A New Benchmark Collec- tion for Text Categorization Research. Journal of Machine Learning Research, 5:361-397.
Dan Li and Evangelos Kanoulas. 2020. When Stop Reviewing in Technology- Reviews. ACM Transactions on Information Systems, 3-(4):1-36.
David R. Cox and Valerie Isham. 1980. Point Processes, volume 12. CRC Press.
Charles X Ling and Victor S Sheng. 2008. Cost- Sensitive Learning and the Class Imbalance Problem. Encyclopedia of machine learning. 2011 1:231-22.35
Maura R. Grossman, Gordon V. Cormack, and Adam Roegiest. 2016. TREC 2016 Total Recall Track Overview. In Proceedings of The Twenty-Fifth Text REtrieval Conference, TREC 2016, volume 500-321 of NIST Special Publication. National Institute of Standards and Technology (NIST).
Douglas Oard, Fabrizio Sebastiani, and Jyothi K Vinjumur. 2018 Jointly Minimizing the Expected Costs of Review for Responsiveness and Privilege in E-discovery. ACM Transactions on Information Systems (TOIS), 37(1):1-35.
Yu Gu, Robert Tinn, Hao Cheng. Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Lan-
Gao, and Hoifung Poon. 2020. Domain-Spectie guage Model Pretraining for Biomedical Natural Lan- guage Processing. ACM Transactions on Computing for Healthcare, 3-):1123.
Nima Sadri and Gordon V Cormack. 2022. Continu- ous Active Learning Using Pretrained Transformers. arXiv preprint arXiv:2208.0695.
Ian Shemilt, Antonia Simon, Gareth J Hollands, Theresa M Marteau, David Ogilvie, Alison O' O'Mara- Eves, Michael P Kelly, and James Thomas. 2014. Pinpointing Needles in Giant Haystacks: Use of Text
Julian PT Higgins, James Thomas, Jacqueline Chandler, Miranda Cumpston, Tianjing Li, Matthew Page, and Vivian A Welch. 2019. Cochrane Handbook for

---

Mining Reduce Impractical Screening Workload Extremely Large Scoping Reviews. Research Syn- thesis Methods, 5(1):31-49.
Alison Sneyd and Mark Stevenson. 2019, Modelling Stopping Criteria for Search Results using Poisson Processes. In Proceedings of the 2019 Conference on Empirical Methods in Natural Longuage Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-JJCNLP) pages 3484-3489.
Alison Sneyd and Mark Stevenson, 2021. Stopping Criteria for Technology Assisted Reviews Based on Counting Processes. In Proceedings of the 44th In- termational ACM SIGIR Conference Research and Development Information Retrieval. SIGIR '21, page 2293-2297.
Mark Stevenson and Reem Bin-Hezam. to appear. Stop- ping Methods for Technology Assisted Reviews based on Point Processes. ACM Transactions on Information Systems (TOIS).
Eugene Yang, David Lewis, and Ophir Frieder. 202 la. Heuristic Stopping Rules for Eechnology-Assister Review. ACM Symposium on Ducument Engineer- ing 2021 (DocEng '21). pages 1-10.
Eugene Yang, David D Lewis. and Ophir Frieder. 2021b. On Minimizing Cost in Legal Document Review Workflows, DocEng 202/ Proceedings of the 2021 ACM Symposium on Document Engineering, pages 1-10.
Eugene Yang. Sean MacAvaney, David D. Lewis, and Ophir Frieder. 2022. Goldilocks: Just-Right Tun- ing of BERT for Technology-Assister Review. In Proceedings of the 44th European Conference Information Retrieval, ECIR 2022, page 502-517. Springer International Publishing.
Zhe Yu and Tim Menzies. 2019. FAST2: An Intelli- gent Assistant for Finding Relevant Papers. Expert Systems with Applications, 120:57-71.
Justin Zobel. 1998. How Reliable Are the Results of Large-Scale Information Retrieval Experiments? In Proceedings of the 2t Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 307-314.


Reviews:
[{"review_id": "FuPaVToF20", "reviewer": "Reviewer_yCws", "summary": "", "questions": "How it may happened that …relevant documents unexpectedly appear late in the ranking.?\nIn the fragment: “A set of sample points are extracted from the documents…”, do you mean by point a document? \nWhy you experimented with three values of desired recall {0.9, 0.8, 0.7} while in the Table with the results Recall always is higher than 0.9?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their constructive comments and feedback on our work. We want to address their questions and comments below.\n\n**Questions For The Authors:**\n\n\n* **1A: How it may happened that …relevant documents unexpectedly appear late in the ranking.?**\n\n* This depends on the ranker and how it perfectly ranked the topic’s relevant documents. For most topics, the ranker placed the relevant documents early in the ranking, but for some topics, the ranker failed to do so, and some of those relevant documents appeared after a big portion of non-relevant documents, which is unexpected for the curve fitting, and hence the classifier predictions would help to spot those documents.\n\n---\n\n* **1B: In the fragment: “A set of sample points are extracted from the documents…”, do you mean by point a document?**\n\n* Yes, it is a document selected within a specified window of documents, and its relevance judgement is calculated using equation 1. The whole ranking is divided into windows, and a point (document) from each window is selected. This set of points, which is a subset of the complete document collection, is then used to fit the curve.\n\n---\n\n* **1C: Why you experimented with three values of desired recall {0.9, 0.8, 0.7} while in the Table with the results Recall always is higher than 0.9?**\n\n* Our model goal is to make sure that the desired recall is achieved while minimising cost and maintaining high reliability. Therefore, although it is overshooting, the cost is reduced with lower desired recall levels.\n\n---\n\n**Typos Grammar Style And Presentation Improvements:**\n\nThanks a lot, we’ll fix this.\n* 040: most relevant documents appear early in the ranking\n* 069: the classifier alone does not consider or model the occurrence rate at which relevant documents have already been observed\n* 118:  it relies on the assumption that the examined portion of documents provides a reliable indication of the rate at which relevant documents occur within the ranking", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper address an actual problem in information retrieval domain. It proposed a solution and demonstrated its effectivity on six benchmark datasets.\n The paper has proper organization and contains all necessary sections.  The methodology and experiments are described in details. \n", "reasons_to_reject": "The problem addressed in the paper is a particular one. It may be important for the Information Retrieval domain; then it should better presented at the specific conference.\n The solution is cumbersome and the improvement is relatively small. \n", "questions_to_authors": "How it may happened that …relevant documents unexpectedly appear late in the ranking.?\nIn the fragment: “A set of sample points are extracted from the documents…”, do you mean by point a document? \nWhy you experimented with three values of desired recall {0.9, 0.8, 0.7} while in the Table with the results Recall always is higher than 0.9?\n", "paper_topic_and_main_contributions": "The paper proposes a solution for optimizing the stopping rule in Technology Assisted Review. The main aim of the stopping rule is to cut the list of documents returned by the retrieval system as an answer to a query. The challenge here is to find the optimal cut with the best recall and minimum non relevant documents that can be filtered manually. The main contribution of the presented work is integration of the text classifier into the existing approach. The text classifier is trained on the most relevant documents and classifies the rest of the documents in order to select the most relevant and remove non relevant more effectively.  \nThe presented experiments on six widely used datasets demonstrated the reduction of additional non relevant documents with minimum drop in recall.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "The paper has enough references."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Some sentences are difficult to comprehend. For example,\n040  …occurrences of relevant documents are likely to decrease higher in the ranking…\n069 …they do not model the rate at which relevant documents have already been observed.\n118 …this provides a reliable indication of the rate at which relevant documents occur within the ranking.\n\n"}}, {"review_id": "UxyJKb8gtm", "reviewer": "Reviewer_Mr4c", "summary": "", "questions": "A: Is it possible to just use the text classification label predictions to estimate recall and avoid the counting processes and curve fitting entirely?   If not, why not? If so, I think it would be beneficial to provide information on how such approaches behave. \n\nB: Lines 198-202 discuss the imbalance ratio and state it is set based on the count of the minority class over the majority class \"in training data\". Is the training data changing at each iteration of TAR? If so, does this means that the imbalance ratio can change each iteration after more human-provided relevancy labels are provided?  If this is not right, then can you explain how it is set?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their constructive comments and feedback on our work. We want to address their questions and comments below.\n\n**Questions For The Authors:**\n\n\n* **2A: Is it possible to just use the text classification label predictions to estimate recall and avoid the counting processes and curve fitting entirely? If not, why not? If so, I think it would be beneficial to provide information on how such approaches behave.**\n\n* Yes, it is possible to use the classification only (classify-and-count quantification: CC-Quant), and we have already explored this and have the results, but we did not include results mainly because of the space limit, and the limitations of this approach.\n\n* One limitation of the classification-only approach is that it does not guarantee the confidence level of the results, unlike the counting process approach which estimates probability distribution with a defined confidence level. Another limitation is that it does not model or consider the occurrence rate of relevant documents within the ranking, which is usually decreasing.\n\n* Our experiments on using the classification only as a stopping rule reached lower reliability levels than combining it with the counting process  (i.e. its reliability ranges from 0.96 to 0.80 for CLEF datasets, and 0.93 for RCV1 dataset). Moreover, it also undershoots desired recall for the CLEF2019 dataset on average. \n\n---\n\n\n* **2B: Lines 198-202 discuss the imbalance ratio and state it is set based on the count of the minority class over the majority class \"in training data\". Is the training data changing at each iteration of TAR? If so, does this mean that the imbalance ratio can change each iteration after more human-provided relevancy labels are provided? If this is not right, then can you explain how it is set?**\n\n* Yes, this is right. The training data changes at each iteration as more examined documents are included, and thus, the imbalance ratio is also updated and re-calculated based on the majority/minority class distribution of the training dataset at each iteration. \n\n---\n\n**Typos Grammar Style And Presentation Improvements:**\n\nThanks a lot, we’ll fix this.\n\nBelow are answers to the following raised points:\n* **How are the sample points chosen? For example, are they equally spaced in the relevancy ranking list and systematically chosen, or are they randomly chosen? How many sample points are used? What is the window size W and how is it chosen? Is it fixed or can it vary?**\n\n* sample points are equally spaced across the ranking and selected from within each window, at the same position. The ranking is divided into 10 windows, and the total points == total number of windows == 10. \n* Window size are fixed for each topic, but not fixed across topics, since it varies depending on the topic size (total number of documents), but the total number of windows is fixed for all topics (i.e. 10 windows).\n * *window size = total number of documents / total number of windows = total number of documents / 10*\n\n\n* **BioBERT and PubMedBERT:**\n\n* BioBERT and PubMedBERT were tested on CLEF datasets only. Other more general BERT models would be more appropriate for TREC Total Recall, TREC Legal, and RCV1 datasets. \n\n\n* **Document Content:**\n\n* The classifier was based on scikit-learn using TF-IDF scores of each document’s content as features, the content is title and abstract for CLEF datasets, email messages for TREC datasets (including title and message body), and news articles content for the RCV1 dataset.\n\n---\n\n**Other Comments**\n* The main reason behind the counting process not fitting well is that it fits the curve based on the explored portion of the documents only, which may not always be representative of the remaining un-explored documents, and here comes the benefit of adding the classifier to predict the labels of the un-explored documents and fit the curve on the whole ranking. We already have figures that compare real data points vs. fitted rate curves, and we could add them in the extra page space provided.\n\n* We have already experimented using other rate functions e.g. Exponential, which while reaching lower recall levels, the reliability was also lower for some datasets, that is why we reported the Power Law results. Also, regarding comparing the benefit of adding the classifier, they all follow the same patterns i.e. results of the counting process improved with the classifier integration for all rate functions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A novel method for stopping TAR based on desired levels of recall is proposed. The new method combines ideas from previously published TAR stopping methods that use counting processes and that use text classification.  \n\nExperiments cover a range of datasets covering a range of different genres and three different desired recall levels were used. The results seem to relatively consistently show that the new proposed integrated method improves stopping performance for the experimental setup used when compared with using the existing counting process methods.", "reasons_to_reject": "The new proposed method is not compared with previously published methods that use text classification, such as those methods mentioned on lines 063-065. It's mentioned at lines 288-290 that direct comparison with previous work is difficult due to the range of rankings and evaluation metrics used in previous work. While this could make it difficult to compare performance numbers across different papers, it seems like it should still be feasible to re-implement previous methods and run them directly with the same rankings and evaluation metrics as the new proposed method was run with in the experiments reported. Currently the paper shows results for the counting process approach with no use of text classification and for the combined approach of using counting process with text classification, but it is missing information on how the text classification approaches without using the counting process approach behave. \n\nThe paper doesn't seem to dive deeper into understanding what's going on other than that an approach was tried and it improved performance. It seems that the improvement means that the counting process rate fitting was not working so well, which is why there was so much room for improvement, however, there is not any analysis to illuminate why it wasn't working well. It could help to show some of the curves that were fit versus the reality of the observed data to show how the observed data differ from what the rate curves were indicating. From the final results it seems to me that the counting process rules are stopping too conservatively in the sense mentioned earlier in this review, which would seem to imply that they're indicating that stopping should not occur because they're predicting that there are more relevant documents left to be found in the remainder of the collection than there actually are left in the collection. It might be helpful to analyze this overestimation more and shed some light on why this overestimation occurs. It seems possible that functions different from the one power law function that was used in the paper could be worth exploring.\n\nThe impact of integrating the text classification component seems like it has the potential to be highly dependent on how the counting process stopping rule is implemented. The paper could be stronger if it experimented with and provided substantial information on the behavior when using different rate functions, confidence levels, sampling of points for curve fitting, window sizes, etc. The paper, to its credit, does mention some of these aspects, although not all of them, as limitations in the limitations section. The mentioning of some of these limitations is better than not mentioning them, however, it does not remove the fact that they remain as limitations.  There is a statement at lines 329-330 that \"the pattern of results is similar to those included\", but more details would be helpful in clarifying what was tested and what the results were.", "questions_to_authors": "A: Is it possible to just use the text classification label predictions to estimate recall and avoid the counting processes and curve fitting entirely?   If not, why not? If so, I think it would be beneficial to provide information on how such approaches behave. \n\nB: Lines 198-202 discuss the imbalance ratio and state it is set based on the count of the minority class over the majority class \"in training data\". Is the training data changing at each iteration of TAR? If so, does this means that the imbalance ratio can change each iteration after more human-provided relevancy labels are provided?  If this is not right, then can you explain how it is set?", "paper_topic_and_main_contributions": "This paper deals with information retrieval, in particular for Technology Assisted Review (TAR) settings where high recall is important. The TAR setting considered is one where an iterative process is set up that aims to reduce the work needed to scan a set of documents for relevance to some topic. The iterative process will keep choosing batches of documents that are ranked as most relevant by the AutoTAR system to provide to human annotators to mark as relevant or not until some stopping criterion is met. This paper focuses on how to detect when to stop the TAR process by building on counting process approaches published previously in (Sneyd and Stevenson, 2019, 2021). The counting process stopping methods work by fitting a relevancy rate function to model the chances of relevant documents occurring lower in the ranking. The counting process uses the fitted rate function to estimate the total number of relevant documents remaining in the collection. Based on this estimate, then the process is stopped when a desired level of estimated recall is reached. The main contribution of the paper is to improve the stopping rule by integrating text classification in order to improve the modeling of relevance rates. The intuition is that the counting process approach relies on the assumption that the data from the top-ranked documents is reliable for predicting the rate of relevant documents lower in the ranking, an assumption which may not be true. The paper proposes that a text classifier be built that can classify documents lower in the ranking and then the predicted class labels (named the ClassLabel approach) or class scores thresholded by 0.5 (named the ClassScore approach) on those lower ranked documents can then also be used for fitting the rate function. Experiments are conducted using a power law as the rate function with an inhomogeneous Poisson process and a logistic regression text classifier using tf-idf features. Cost-sensitive learning was used with an imbalance ratio obtained by using the count of the minority class over the count of the majority class in the training data. Experiments were conducted on CLEF e-Health, TREC Total Recall, TREC Legal, and RCV1 datasets. Recall, Reliability, Cost, and excess cost were used as evaluation metrics. Desired recall levels of 0.9, 0.8, and 0.7 were used in the experiments. The proposed integration of text classification improves stopping performance over the existing counting process approach, with the ClassLabel and ClassScore variations not having much difference in performance between each other, but both performing better than the existing counting process approach without text classification integrated. If we use the conservative and aggressive terminology to describe the behavior of the TAR stopping methods in the same sense the terms are used in the active learning stopping methods literature (https://aclanthology.org/W09-1107.pdf), the results of the paper show that the existing counting process stopping methods tend to stop too conservatively and the newly proposed stopping method that integrates the counting process and text classification ideas tends to stop more aggressively, thereby saving annotation effort while not harming recall and reliability.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Consider whether it might be more intuitive to replace 'n' with |W| in equation 1. \n\nIn general, I sense that the paper might be a little difficult for readers to follow if they don't already know quite a bit about TAR and the (Sneyd and Stevenson, 2019, 2021) papers. Consider if it's possible to make the paper more self-contained. \n\nFigure 1 and Table 2 are too small to read easily when the paper is printed out. Consider if there is a way to make it easier to read from a hard copy of the paper.\n\nThere are some spots in the paper where it's possible that providing some more details might be helpful for readers. I found that I think I was able to understand the gist of how the counting process stopping rules work, but I struggled to understand or to find in the paper precise details about how they work. In particular, how are the sample points chosen? For example, are they equally spaced in the relevancy ranking list and systematically chosen, or are they randomly chosen? How many sample points are used? What is the window size W and how is it chosen? Is it fixed or can it vary? \n\nIt's mentioned that BioBERT and PubMedBERT models were developed, but that they didn't perform better than the logistic regression model at lines 183-186. It seems an odd choice and a bit confusing as to why one would use BioBERT and PubMedBERT on the TREC Total Recall, TREC Legal, and RCV1 datasets since these datasets don't seem to be from a biomedical domain. Some explanation of this and possibly providing results in a supplement might be helpful. \n\nLines 178-180 mention that \"each document's title and abstract\" were used to provide tf-idf features. This could be a bit confusing for some readers since for several of the datasets it seems that the documents in the dataset might not contain titles and abstracts, or at least it's not clear what one would consider as the title and abstract. For example, for the email datasets, what would be considered as the title and abstract for each document? Consider whether there might be a way to rephrase or clarify this.\n\nUpdate after Author Response: Thank you for the author responses. Answer 2B about the imbalance ratios being updated after each iteration of active learning was interesting. The authors might find it useful to consult this NAACL paper ( https://aclanthology.org/N09-2035.pdf ), which compares the approach of updating the imbalance ratios after each iteration with a modified approach that might work better when active learning is being used."}}, {"review_id": "TBILdQOHUJ", "reviewer": "Reviewer_jZAo", "summary": "", "questions": "There are some aspects unclear:\n\n- In line 114 you write 'if d has been labelled as relevant', and then in line 122 you say that the counting process might not provide reliable results when e.g. only a small number of documents have been manually examined. So how do you define in your setup something as relevant? Is this a manual process in general? Somebody would analyse the first set of documents and label them? And in your case, would this be given, as the data is already labelled?\n\n- About the classifier: You use the first n samples with their labels and train a classifier on them. Generally, it would be interesting to know how well the classifier performs and how it performs then on the unexamined documents. I would assume that the label distribution changes between examined to unexamined documents, which might strongly influence the classifier's performance. Also, is there some kind of correlation between your approach's performance and the classifier's reliability? These would certainly be nice aspects to explore in more detail, to improve the work....\n\n- Excess cost evaluation is not clear to me. Can you say a bit more about cost(method) and cost(optimal)?\n", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their constructive comments and feedback on our work. We want to address their questions and comments below.\n\n\n**Questions For The Authors:**\n\n\n* **3A: In line 114 you write 'if d has been labelled as relevant', and then in line 122 you say that the counting process might not provide reliable results when e.g. only a small number of documents have been manually examined. So how do you define in your setup something as relevant? Is this a manual process in general? Somebody would analyse the first set of documents and label them? And in your case, would this be given, as the data is already labelled?**\n\n* Yes, the examined documents are already labelled by human experts and the datasets we used are already labelled. However, while using the classifier, we are training the classifier on the examined set of documents with true labels, and apply it to the unexamined part to get the classifier predictions (labels or scores). Therefore, for the examined part relevance is labelled by human experts, and for the unexamined part, relevance is predicted by the classifier.\n\n---\n\n* **3B (1): About the classifier: You use the first n samples with their labels and train a classifier on them. Generally, it would be interesting to know how well the classifier performs and how it performs then on the unexamined documents.** \n\n* Classifier performance improved as more documents are included in the training set at each iteration. Without any imbalance handling, the initial classifier performance was biased towards the majority class, which was fixed later with proper imbalance handling tehcnique (cost-sensitve learning in our experiments).\n\n---\n\n* **3B (2): I would assume that the label distribution changes between examined to unexamined documents, which might strongly influence the classifier's performance.** \n* Yes the label distribution changes between examined and unexamined documents, since TAR problems are usually highly imbalanced, that's why we handled this using cost-sensitive learning. With ranked list, the majority of the training dataset would be relevant, while the majority of the test set would be non-relevant. The training data changes at each iteration as more examined documents are included, and thus the imbalance ratio is also updated and re-calculated based on the training dataset majority/minority class at each iteration.  \n\n---\n\n* **3B (3): Also, is there some kind of correlation between your approach's performance and the classifier's reliability? These would certainly be nice aspects to explore in more detail, to improve the work....** \n\n* Yes, definitely the stopping decision is affected by the classifier's performance. We have explored the classifier performance without using any imbalance handling, which causes the learning to be biased toward the majority class at each iteration, and the stopping decision would be either too early (causes lower reliability as low as 0 for some datasets), where the majority in the training was the non-relevant class, or the stopping would be too late (cause higher cost), where the majority in the training was the relevant class.\n\n* However, applying the cost-sensitive learning improved the classifier's performance and hence the stopping decision became more accurate. Handling the imbalance with cost-sensitive learning was selected over using oversampling techniques because, although they are a good approach, in some cases where the minority class is too small in the sample, the generated synthetic instances by SMOTE are not going to be representative of the class, and hence results would not be accurate. Moreover, cost-sensitive was selected over using undersampling because it may cause the loss of some potentially useful instances of the relevant class during the learning phase, since labelling relevant documents correctly and not missing any of them by the classifier is important for many TAR problems.\n\n* The table below shows the results of how the classifier performance drops while not using any imbalance handling techniques, which affects the overall stopping decisions and causes lower reliability and undershooting of the desired recall. \n\nTable 1: CP+ClassLabel Model Results with Original Data without any Imbalance Handling\n\n| dataset  | des_recall | recall | reliability | cost  | excess |\n|----------|------------|--------|-------------|-------|--------|\n| CLEF2017 | 0.9        | 0.967  | 0.867       | 0.133 | 0.081  |\n|          | 0.8        | 0.967  | 0.967       | 0.133 | 0.094  |\n|          | 0.7        | 0.966  | 1           | 0.131 | 0.100  |\n| Legal    | 0.9        | 0.793  | 0           | 0.025 | -0.016 |\n|          | 0.8        | 0.793  | 0.5         | 0.025 | -0.001 |\n|          | 0.7        | 0.793  | 1           | 0.025 | 0.006  |\n\n\n---\n\n* **3C: Excess cost evaluation is not clear to me. Can you say a bit more about cost(method) and cost(optimal)?**\n\n* cost(method) is the cost of the method being evaluated (e.g. CP+ClassLabel), and cost(optimal) is the cost of stopping at the optimal point once the desired recall is achieved. The cost here is calculated as the percentage of examined documents from the whole collection.\n* The Excess metric indicates the proportion of the documents that are examined after the desired recall has been reached (positve: overshooting), or that need to be examined until the desired recall cost is reached (negative: undershooting). \n\n---\n\n**Typos Grammar Style And Presentation Improvements:**\n\nThanks a lot, we’ll fix this.\n\n\n---\n\n\n**Other Comments**\n\n* Results for baselines with all CLEF & TREC datasets are already available but were not included because of the space limit. They follow the same pattern and could be included when the extra page is provided. The table below summarises our approaches with the baselines on more datasets. \n\nTable 2: Proposed Methods vs. Baseline\n\n|          |               | des_recall = 0.9 |             |       |        | des_recall = 0.8 |             |       |        |\n|----------|---------------|:----------------:|:-----------:|:-----:|:------:|:----------------:|:-----------:|:-----:|:------:|\n| Dataset  | model         | recall           | reliability | cost  | excess | recall           | reliability | cost  | excess |\n| CLEF2017 | CP            |            1.000 |       1.000 | 0.281 |  0.238 |            1.000 |       1.000 | 0.265 |  0.232 |\n|          | CP+ClassLabel |            0.989 |       1.000 | 0.153 |  0.102 |            0.989 |       1.000 | 0.152 |  0.114 |\n|          | CP+ClassScore |            0.989 |       1.000 | 0.152 |  0.101 |            0.989 |       1.000 | 0.152 |  0.114 |\n|          | SCAL          |            0.914 |       0.667 | 0.496 |  0.466 |            0.888 |       0.800 | 0.451 |  0.426 |\n|          | SD-training   |            0.955 |       0.833 | 0.691 |  0.672 |            0.881 |       0.767 | 0.417 |  0.391 |\n|          | SD-sampling   |            0.902 |       0.567 | 0.506 |  0.476 |            0.798 |       0.433 | 0.350 |  0.321 |\n|          | AutoStop      |            0.884 |       0.500 | 0.421 |  0.386 |            0.787 |       0.367 | 0.335 |  0.305 |\n|          | Target        |            0.960 |       1.000 | 0.498 |  0.468 |            0.922 |       1.000 | 0.468 |  0.444 |\n| Dataset  | model         | recall           | reliability | cost  | excess | recall           | reliability | cost  | excess |\n| CLEF2018 | CP            |            1.000 |       1.000 | 0.293 |  0.242 |            1.000 |       1.000 | 0.287 |  0.249 |\n|          | CP+ClassLabel |            0.983 |       1.000 | 0.137 |  0.075 |            0.983 |       1.000 | 0.137 |  0.091 |\n|          | CP+ClassScore |            0.983 |       1.000 | 0.137 |  0.075 |            0.982 |       1.000 | 0.136 |  0.090 |\n|          | SCAL          |            0.902 |       0.667 | 0.493 |  0.457 |            0.862 |       0.767 | 0.428 |  0.397 |\n|          | SD-training   |            0.972 |       0.967 | 0.701 |  0.680 |            0.886 |       0.833 | 0.414 |  0.383 |\n|          | SD-sampling   |            0.855 |       0.367 | 0.379 |  0.334 |            0.753 |       0.367 | 0.258 |  0.218 |\n|          | AutoStop      |            0.892 |       0.600 | 0.441 |  0.401 |            0.781 |       0.467 | 0.347 |  0.312 |\n|          | Target        |            0.934 |       0.833 | 0.431 |  0.390 |            0.908 |       0.967 | 0.421 |  0.390 |\n| Dataset  | model         | recall           | reliability | cost  | excess | recall           | reliability | cost  | excess |\n| CLEF2019 | CP            |            0.999 |       1.000 | 0.283 |  0.228 |            0.999 |       1.000 | 0.279 |  0.235 |\n|          | CP+ClassLabel |            0.996 |       1.000 | 0.221 |  0.161 |            0.996 |       1.000 | 0.216 |  0.169 |\n|          | CP+ClassScore |            0.996 |       1.000 | 0.221 |  0.161 |            0.996 |       1.000 | 0.213 |  0.165 |\n|          | SCAL          |            0.893 |       0.516 | 0.621 |  0.592 |            0.887 |       0.903 | 0.577 |  0.551 |\n|          | SD-training   |            0.940 |       0.774 | 0.713 |  0.691 |            0.826 |       0.613 | 0.421 |  0.386 |\n|          | SD-sampling   |            0.893 |       0.508 | 0.517 |  0.480 |            0.787 |       0.475 | 0.366 |  0.328 |\n|          | AutoStop      |            0.878 |       0.387 | 0.479 |  0.439 |            0.791 |       0.452 | 0.397 |  0.361 |\n|          | Target        |            0.968 |       0.935 | 0.673 |  0.648 |            0.945 |       1.000 | 0.648 |  0.627 |\n| Dataset  | model         | recall           | reliability | cost  | excess | recall           | reliability | cost  | excess |\n|   Legal  | CP            |            1.000 |       1.000 | 0.425 |  0.401 |            1.000 |       1.000 | 0.056 |  0.052 |\n|          | CP+ClassLabel |            0.972 |       1.000 | 0.088 |  0.050 |            0.999 |       1.000 | 0.027 |  0.023 |\n|          | CP+ClassScore |            0.972 |       1.000 | 0.088 |  0.050 |            0.999 |       1.000 | 0.027 |  0.023 |\n|          | SCAL          |            0.039 |       0.000 | 0.005 | -0.036 |            0.039 |       0.000 | 0.005 |  0.001 |\n|          | SD-training   |            1.000 |       1.000 | 1.000 |  1.000 |            1.000 |       1.000 | 1.000 |  1.000 |\n|          | SD-sampling   |            1.000 |       1.000 | 1.000 |  1.000 |            1.000 |       1.000 | 1.000 |  1.000 |\n|          | AutoStop      |            0.803 |       0.000 | 0.811 |  0.803 |            0.684 |       0.000 | 0.794 |  0.793 |\n|          | Target        |            0.882 |       0.500 | 0.092 |  0.054 |            0.808 |       1.000 | 0.073 |  0.069 |\n| Dataset  | model         | recall           | reliability | cost  | excess | recall           | reliability | cost  | excess |\n|    TR    | CP            |            1.000 |       1.000 | 0.059 |  0.054 |            1.000 |       1.000 | 0.056 |  0.052 |\n|          | CP+ClassLabel |            0.999 |       1.000 | 0.028 |  0.023 |            0.999 |       1.000 | 0.027 |  0.023 |\n|          | CP+ClassScore |            0.999 |       1.000 | 0.028 |  0.023 |            0.999 |       1.000 | 0.027 |  0.023 |\n|          | SCAL          |            0.903 |       0.647 | 0.144 |  0.140 |            0.761 |       0.676 | 0.107 |  0.103 |\n|          | SD-training   |            1.000 |       1.000 | 1.000 |  1.000 |            1.000 |       1.000 | 1.000 |  1.000 |\n|          | SD-sampling   |            0.936 |       0.794 | 0.779 |  0.778 |            0.896 |       0.735 | 0.690 |  0.689 |\n|          | AutoStop      |            0.953 |       0.941 | 0.766 |  0.765 |            0.885 |       0.912 | 0.754 |  0.753 |\n|          | Target        |            0.900 |       0.706 | 0.069 |  0.064 |            0.844 |       0.882 | 0.069 |  0.065 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper describes a small focussed contribution. Therefore it fits as a short paper. \n- At the same time, the authors test their approach on a large variety of different datasets and show that it works for their setup.", "reasons_to_reject": "- A more meaningful comparison might be good. The work is just compared to one approach (except for one dataset).\n- A more detailed analysis which provides more insights (e.g. in the context of the classifier), might be very valuable.", "questions_to_authors": "There are some aspects unclear:\n\n- In line 114 you write 'if d has been labelled as relevant', and then in line 122 you say that the counting process might not provide reliable results when e.g. only a small number of documents have been manually examined. So how do you define in your setup something as relevant? Is this a manual process in general? Somebody would analyse the first set of documents and label them? And in your case, would this be given, as the data is already labelled?\n\n- About the classifier: You use the first n samples with their labels and train a classifier on them. Generally, it would be interesting to know how well the classifier performs and how it performs then on the unexamined documents. I would assume that the label distribution changes between examined to unexamined documents, which might strongly influence the classifier's performance. Also, is there some kind of correlation between your approach's performance and the classifier's reliability? These would certainly be nice aspects to explore in more detail, to improve the work....\n\n- Excess cost evaluation is not clear to me. Can you say a bit more about cost(method) and cost(optimal)?\n", "paper_topic_and_main_contributions": "In the context of information retrieval, many documents are often returned. Therefore, it is difficult to analyze all documents, but it is important to identify the most relevant documents depending on the use case. The paper presents a method in the context of technology-assisted review (TAR) to reduce the number of relevant documents. Related work presented a stopping rule based on counting processes. This work extends this counting process by using a text classifier.\n\nContributions:\n- a method to integrate text classification into the stopping rule in the context of technology-assisted review/information retrieval\n- testing of a novel approach on a large range of different datasets\n- showed the benefit of their model\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- There might be an error in reference of Sadri and Cormack, 2022? Please check the reference part at the end.\n- line 248: This should probably start with a capital 'e'?"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers share some similar concerns, especially in the lack of traditional baseline comparison. According to the rebuttal information provided by the authors, they have performed related experiments and will add these materials in the extra page if accepted. I therefore believe the authors have answered most of the reviewers' concerns.
Decision: Accept-Findings