DOI: 10.5281/zenodo.17755086
title: Background data for: Random forests in corpus research: A systematic review

description: This dataset records the result of a systematic review on the use of random-forest models in corpus-based research. It is associated with the following paper:

Sönning, Lukas. 2026. Random forests in corpus research: A systematic review. OSF Preprints.

It consists of a UTF-8-encoded, tab-separated data table and a PDF file with a PRISMA flow chart (Page et al. 2021) outlining the study selection process.

 

A systematic review of random-forest modeling in (corpus-)linguistic research

To better understand how corpus linguists use random-forest models in their research, I conducted a systematic review. The report pool includes n = 6,972 research articles published between 2012 and 2024 across 20 linguistic journals. The selection of journals, which are listed in the table below, aims to cover a broad range of subdisciplines and research styles. The database was stored locally and analysed with the qualitative data analysis software MAXQDA 2022 (VERBI Software 2021). I used the search terms “random forest”, “random forests” and “random-forest” to identify relevant papers. This returned 130 articles, which were then screened for whether an RF model was used for data analysis; only 79 studies actually did so. The study selection process is outlined in the PRISMA flow chart (Page et al. 2021) that is part of this data post (file: fig02_PRISMA_2020_flow_diagram.pdf)

The following table lists, for each journal,



Total  the total number of studies in the report pool,

N  the number of articles using a random-forest model for data analysis, and

%  the percentage of articles (N/Total*100) using a random-forest model


Journals are sorted based on the right-most column (%). 






Journal
Total
N
%


International Journal of Learner Corpus Research
94
6
6.4%


Language Variation and Change
177
11
6.2%


Corpus Linguistics and Linguistic Theory
190
11
5.8%


English World-Wide
154
6
3.9%


International Journal of Corpus Linguistics
251
7
2.8%


English Language and Linguistics
295
6
2.0%


Corpora 
181
3
1.7%


Journal of Sociolinguistics 
224
3
1.3%


World Englishes 
457
4
0.9%


Linguistics 
451
4
0.9%


Studies in Second Language Acquisition 
394
3
0.8%


Lingua
1149
7
0.6%


Language Learning 
408
2
0.5%


Applied Psycholinguistics 
624
3
0.5%


Applied Linguistics 
464
2
0.4%


Cognitive Linguistics 
255
1
0.4%


Language 
277
1
0.4%


Linguistic Inquiry 
386
0
0.0%


Natural Language and Linguistic Theory
425
0
0.0%


Research in Corpus Linguistics
116
0
0.0%


Total
6,972
 
 





 

Within this pool of n = 79 articles, I then identified all RF models that were reported, which yielded a list of 147 items. These 147 models were then coded for a range of features, which are listed below. Given the focus of the current review on corpus-based research, I screened the candidate models for the type of data underlying the analysis; 21 models were excluded because they did not rely on corpus data. I adopted a relatively broad notion of “corpus-based”: An RF model qualified for inclusion if the method of data collection classified as observational, naturalistic and/or uncontrolled; most notably, sociolinguistic interviews were considered “corpus data”. One further RF model was excluded because its application was – in my view – nonsensical.  This yields a final pool of 125 RF models drawn from 69 research articles. While most studies reported a single analysis (n = 46, 67%), several papers included multiple RF models. 



article_id  article identifier

doi  persistent digital object identifier

journal  name of journal

year  of publication

month  month of publication

exclude  whether the RF model should be excluded from this review (because model was nensensical)

corpus_data  whether the data are drawn from a corpus

other_methods  other methods that were used to analyze the data set

software  software used to fit the RF model

package  R package used to fit the RF model

purpose  whether the random forest model served as the "primary" means of answering the research question, or whther it played an "ancillary" role

n_pred  number of predictor variables in the forest

n_pred_pars  number of predictor parameters in an equivalent main-effects-only regression model

clust_var_levels  if a clustering variable was included as a predictor in the model, number of levels

n_terms  number of terms in the forest; differs from "n_pred" if interactions are manually coded and included 

mtry  setting of the mtry parameter, which specifies the number of candidate variables (or terms) to sample at each splitting occasion

mtry_tuned  whether the choice of mtry was based on some type of optimality criterion

n_trees  number of trees in the random forest

n_obs  number of observations (i.e. tokens/data points) in the dataset

var_importance  the type of variable importance measure used ("standard", "conditional", "unclear")

interpretation  list of strategies used for model interpretation; see [1] below

interactions_incl  whether interaction between predictors were manually specified and included

tree_type  classification tree (categorical response variable) vs. regression tree (continuous response variable)

n_levels  for classification trees (categorical outcomes), number of response categories

ordinal  for categorical outcomes with three or more levels, whether these are ordered (i.e. an ordinal response variable)

alternation  whether the outcome variable can be considered as representing some sort of alternation

outcome_variable  short description of the outcome variable

clust_vars  whether clustering variables are present in the data, and if so, which ones (speaker/text/item)

clust_vars_aggr  clustering variables aggregated into "test" (speaker, author, text) and "item" (word, lexeme, lexical item, etc.)

clust_vars_incl  whether (and if so, which) clustering variables (e.g. speaker/text/item) were included in the forest

clust_vars_incl_aggr  whether (and if so, which) clustering variables (aggregated label) were included in the forest

clusterlevel_preds  whether the forest included cluster-level variables alongside the clustering variable

clust_vars_info  information about the relative importance of clustering and cluster-level variables

baseline  for classification forests, the proportion of the most common class/level (benchmark for prediction accuracy)

type_pred_specified  whether the type of prediction (ordinary vs. out-of-bag) underlying the assessment of predictive performance was specified

pred_accuracy  for classification forests, the prediction accuracy (proportion of correctly classified tokens) reported, without indication of the type (training data/ordinary vs. test data/out-of-bag)

pred_accuracy_oob  for classification forests, the out-of-bag prediction accuracy (proportion of correctly classified tokens)

c_index  concordance index or, equivalently, area under the curve (AUC) reported for the model, without indication of the type (training data/ordinary vs. test data/out-of-bag)

c_index_oob  out-of-bag concordance index or, equivalently, area under the curve (AUC) reported for the model


[1]

"varimp" = variable importance scores"tree" = representative tree"pdp" = partial dependence plots"preds" = predictions from the RF model"gsm" = global surrogate model"predictive performance" = model was used for genuine prediction task)

 

References

Page, Matthew J, Joanne E. McKenzie, Joanne E. Bossuyt, Isabelle Boutron, Tammy C. Hoffmann, Cynthia D. Mulrow, Larissa Shamseer, Jennifer M. Tetzlaff, Elie A. Akl, Sue E. Brennan, Roger Chou, Julie Glanville, Jeremy M. Grimshaw, Asbjørn Hróbjartsson, Manoj M. Lalu, Tianjing Li, Elizabeth W. Loder, Evan Mayo-Wilson, Steve McDonald, Luke A. McGuinness, Lesley A. Stewart, James Thomas, Andrea C. Tricco, Vivian A. Welch, Penny Whiting & David Moher. 2021. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. British Medical Journal 372: n71. https://doi.org/10.1136/bmj.n71 

VERBI Software. 2021. MAXQDA 2022 [computer software]. Berlin, Germany: VERBI Software. Available from maxqda.com.