# The Multi-Feature Tagger of English (MFTE)

Elen Le Foll, Muhammad Shakir

► To cite this version:

Elen Le Foll, Muhammad Shakir. The Multi-Feature Tagger of English (MFTE). Research in Corpus Linguistics, 2025, 13 (2), pp.63-93. 10.32714/ricl.13.02.03. hal-04827143

HAL Id: hal-04827143

https://hal.science/hal-04827143v1

Submitted on 24 Jan 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# The *Multi-Feature Tagger of English*
(MFTE): Rationale, description and
evaluation

Elen Le Follª – Muhammad Shakirᵇ

University of Cologneª / Germany

University of Münsterᵇ/ Germany

**Abstract** – The *Multi-Feature Tagger of English* (MFTE) provides a transparent and easily adaptable open-source tool for multivariable analyses of English corpora. Designed to contribute to the greater reproducibility, transparency, and accessibility of multivariable corpus studies, it comes with a simple GUI and is available both as a richly annotated *Python* script and as an executable file. In this article, we detail its features and how they are operationalised. The default tagset comprises 74 lexico-grammatical features, ranging from attributive adjectives and progressives to tag questions and emoticons. An optional extended tagset covers more than 70 additional features, including many semantic features, such as human nouns and verbs of causation. We evaluate the accuracy of the MFTE on a sample of 60 texts from the BNC2014 and COCA, and report precision and recall metrics for all the features of the simple tagset. We outline how that the use of a well-documented, open-source tool can contribute to improving the reproducibility and replicability of multivariable studies of English.

**Keywords** – software; multivariable analysis; multivariate analysis; open source; corpus linguistics; corpus tool; multi-dimensional analysis; *Python*

## 1. INTRODUCTION

The addition of multivariable<fnref n="1" /> analysis methods to the linguist's toolbox has proven indispensable to shed light on the intricate interplay between diverse linguistic features and the situational/contextual factors that shape them. One of the earliest such methods is multi-dimensional (MD) analysis—a framework pioneered by Douglas Biber in the 1980s and first applied to the study of register variation (see Biber 1984; 1988). What MD analysis and most other multivariable methods used in linguistics have in common

<footnote n="1">¹ In quantitative linguistics, the terms ‘multivariable’ and ‘multivariate’ are frequently equated. Statisticians, however, differentiate between ‘multivariable’ methods in which several independent variables (predictors) are used to explain or predict a single outcome variable and ‘multivariate’ ones in which there are two or more dependent (or outcome) variables (Hidalgo and Goodman 2013). As all multivariate methods are, by definition, also multivariable, we use the term ‘multivariable’ throughout this article as an overarching term encompassing methods such as multivariable linear and logistic regression, factor analysis, cluster analysis, and machine learning methods.</footnote>
is that they rely on software capable of automatically identifying a large number of
linguistic features in many texts. Since the co-occurrences of these linguistic features need
to be identified and counted across hundreds or thousands of texts for such analyses to be
feasible,<fnref n="2" /> automatic feature taggers are needed. Indeed, they can be said to constitute the
backbone of such analysis frameworks. In the context of MD analysis, Biber (2019: 14)
stresses that “[a]lthough its importance is not widely recognized, the computer program
used for grammatical tagging provides the foundation for MD studies.” This failure to
recognise the crucial importance of the tools underlying most quantitative corpus-
linguistic analyses is detrimental to the reproducibility of quantitative corpus-linguistic
studies. Even when the corpus data are publicly available, if the tools used to process the
data are not freely accessible, the results cannot be independently verified. It is also very
difficult to test their robustness and generalisability on new data.

This is particularly problematic in the context of the ‘replication crisis’ (first named
as such in Pashler and Wagenmakers 2012), which having been first exposed in social
psychology has, over the past decade, spared almost no discipline, highlighting the
pressing need to improve both the reproducibility of scientific studies and the replicability
of their findings.<fnref n="3" /> Among the many causes of the replication crisis/crises, the lack of
published (or otherwise freely accessible) research data, code and software ranks high. It
undoubtedly constitutes a major barrier to both the reproducibility and replicability of
published research (see, e.g. John et al. 2012; Baker 2016; Gewin 2016). (Corpus)
linguists, too, are becoming increasingly aware of the implications of non-reproducible
methods, as evidenced, for example, by a special issue of *Linguistics* devoted to the
replication crisis (Sönning and Werner 2021) and several recent articles and monographs
that tackle the issue head-on (e.g., Porte and McManus 2018; Wallis 2021; In’nami et al.
2022; McManus 2024).

<footnote n="2">² Synthesising data from 161 MD studies, Goulart and Wood (2021: 119) report a mean size of corpora used in MD analyses of 5.5 million words (with a very large range from under 10.000 words to over 206 million words). In a meta-analysis of 23 MD studies, Egbert and Staples (2019: 132) report that the mean number of variables entered in MD analyses is around 60.</footnote>

<footnote n="3">³ Although the terms are sometimes used interchangeably, we use the term ‘reproducibility’ to refer to the ability to obtain the same results as an original study using the authors’ data and code, whilst we understand ‘replicability’ to be about obtaining compatible results with (more or less) the same method(s) but different data (see, e.g., Berez-Kroeker et al. 2018).</footnote>
## 2. LITERATURE REVIEW

In the present paper, we introduce a substantially revised and extended version of the *Multi-Feature Tagger of English Perl*: the *MFTE Python* (hereafter MFTE), designed to facilitate the reproducible, multivariable analysis of large English corpora. Although our tool can be used for any type of quantitative corpus analysis, we developed the MFTE primarily with multivariable analyses in mind as it is often possible to manually examine the accuracy of a tagger for one or just a few feature(s). Such a procedure, however, becomes entirely unfeasible when a large(r) number of linguistic variables are entered in an analysis. The MFTE was designed to facilitate large-scale multivariable analyses such as MD studies by aiming to largely eliminate the need for the manual correction of feature identification ('fix-tagging', see Section 2.1.3).

In the following, we explain our rationale for the development of a new open-source tagger written in a modern, object-oriented programming language. The rationale in Section 2.1 is followed by the specifications of the tagger in Section 2.2. In Section 2.3, we list the linguistic features it tags and counts and explain how they are operationalised in Section 2.4. In Section 2.5, we explain the tagger's usage and outputs and then report on a detailed evaluation of the tagger's output, before discussing its strengths and limitations and concluding with an outlook on future possible uses and developments.

### 2.1. *Rationale for a new multi-feature tagger of English*

Several factors motivated the development of the MFTE. They are best summarised as concerns about the reproducibility, transparency, and accessibility of the corpus tools currently most frequently used in multivariable analyses of English. It should be mentioned that these same reasons had originally motivated the development of the *MFTE Perl*, an earlier but considerably less powerful version of the MFTE.<fnref n="4" /> Its documentation (Le Foll 2021) outlined the methodological decisions involved in the selection of its features, details of their operationalisations, and the rationale behind the use of different normalisation units for the feature frequencies. However, no research article about the *MFTE Perl* was submitted as it quickly became apparent that an entirely reworked version of it, ported to *Python* and considerably extended, would fulfil better

<footnote n="4">⁴ https://github.com/elenlefoll/MultiFeatureTaggerEnglish (accessed 8 March 2024)</footnote>
the three-fold objectives of reproducibility, transparency, and accessibility. It is this new
*Python* version that we present and evaluate in the present paper.

2.1.1. Reproducibility

Using proprietary software for research constitutes a barrier to reproducibility as only a
limited number of researchers (i.e., those who have personal contacts with the
developer(s) or the (institutional) means to pay for a licence) can attempt to reproduce the
results of published studies. In theory, publishing detailed information about the inner
workings of a piece of software is an alternative to making software accessible to all for
free and/or publishing its source code. This is essentially what was done in what remains
the most cited MD analysis study to date, namely Biber (1988: Appendix II), which
includes detailed descriptions of the algorithms of the tagger now widely known as the
‘*Biber tagger*’ (Gray 2019). It was used to identify the features entered in Biber’s (1988)
seminal MD analysis. As a result, even though the *Biber tagger* is not available to the
wider research community, it is possible to reconstruct it based on this list of algorithms.
There is no doubt, however, that such a ‘reconstruction’ is a time-consuming process that
requires advanced programming skills, which not all corpus linguists possess.

Demonstrating that reconstruction is possible based on Biber's (1988: Appendix II)
list of algorithms, Nini (2014; 2019) successfully reproduced the functions of the 1988
version of the *Biber tagger* with only very minor differences. The resulting corpus tool,
labelled the ‘*Multidimensional Analysis Tagger*’ (MAT), was originally released as
freeware in 2013 and subsequently made available under an open-source licence on
GitHub in 2020.<fnref n="5" /> MAT allows researchers to conduct reproducible analyses using the
linguistic features described in Biber (1988: Appendix II).<fnref n="6" /> More recent MD studies that
rely on the *Biber tagger*, however, are not reproducible as the *Biber tagger* has
considerably evolved since the 1988 publication (Biber and Egbert 2018: 22; Gray 2019:

<footnote n="5">⁵ https://github.com/andreanini/multidimensionalanalysistagger (accessed 14 December 2023).</footnote>

<footnote n="6">⁶ It is worth noting that MAT does more than just tag and count the features used in Biber's (1988) analysis: it also calculates dimension scores on Biber's (1988) dimensions of *General Written and Spoken English*, outputs plots with mean values of the tagged texts/corpus against these dimensions, making it ideal for conducting additive MD analyses (Berber Sardinha et al. 2019) that rely on Biber's (1988) *Model of General Spoken and Written English* as a comparison baseline. MAT also assigns each text to one of the eight text types proposed in Biber's (1989) *Typology of English Texts*. Moreover, the GUI version (for *Windows* only) features a tool for visualising the features of an input text that load on a particular dimension.</footnote>
46).<fnref n="7" /> Our reproducibility motivation for the development of the MFTE was therefore to allow researchers to conduct reproducible analyses involving a larger number of English lexico-grammatical and semantic features that go beyond those of the 1988-version of the *Biber tagger*, and which users can flexibly and transparently adapt and amend according to their needs.

### 2.1.2. Transparency

Ensuring that the concrete operationalisations of the features of the MFTE would be as transparent as possible was a further motivating factor for the development of the MFTE. Reproducibility and transparency of research processes are inextricably linked. Not only does the lack of access to the source code of a tagger mean that results cannot be reproduced, but it also means that researchers conducting and evaluating studies that rely on such tools have few means of understanding exactly how the features entered into these analyses were identified. While this is true for many tools used in corpus studies, it is particularly problematic in the context of multivariable studies such as MD analyses, which rely on counting large numbers of linguistic features across many texts, for which manual spot-checking of counts is simply not feasible.

A further aim of the MFTE was therefore to make available both detailed textual explanations of its feature operationalisations, as well as easily accessible source code to be able to examine their concrete operationalisation and, if need be, adapt them to specific language varieties and/or registers, linguistic theories,<fnref n="8" /> and research questions. Whilst it is possible to scrutinise the exact operationalisations of each linguistic feature of MAT, its code structure is relatively complex due to its graphical interface and many additional functions (see footnote 6), which means that only linguists with a strong programming background are likely to be able to edit the source code to introduce customised features/feature operationalisations. In developing the MFTE, we opted for the simplest

<footnote n="7">⁷ During the second round of revisions for this paper, an anonymous reviewer helpfully pointed out that a new tagger for MD analysis is now available in beta stage. According to the documentation, it is being developed by Kristopher Kyle and colleagues at the Linguistics Department of the University of Oregon in consultation with Douglas Biber's lab in Northern Arizona University: https://github.com/kristopherkyle/LxGrTgr (9 September 2024).</footnote>

<footnote n="8">⁸ It is worth remembering that POS tagging is fundamentally a method of analysing grammar and morphology. As a consequence, the process inevitably implicitly reflects a specific approach or theory of grammar (McEnery et al. 2006; Lindquist 2009: 44–45; Gray 2019: 34)</footnote>
structure possible: it consists of a single script that can be easily edited by linguists familiar with regular expressions. This brings us to our final, major motivation for developing the *MFTE Python*, which is to contribute to the better accessibility of taggers that can readily be used to conduct multivariable analyses of English.

### 2.1.3. Accessibility

For some features, both the *Biber tagger* and MAT require (semi-)manual ‘fix-tagging’ procedures to reach high levels of tagging accuracy (Gray 2019: 59–61). In fact, it is recommended that the *Biber tagger* be used in combination with an interactive tag-checker, see Biber and Gray (2013) for details. Although this process allows for the inclusion of linguistic features that cannot reliably be annotated automatically, it requires trained annotators to perform time-consuming manual checks and corrections.<fnref n="9" /> For a tool to be accessible for research projects with little or no funding, we therefore believe that the need for fix-tagging should be reduced to a minimum. Given that most multivariable linguistic methods, including MD analysis, require many different linguistic features to be identified across large corpora, we aimed for high tagging accuracy without the need for human intervention. Moreover, we concluded that the tagger documentation should include a detailed evaluation of the accuracy of the tagging procedure on a representative sample of texts. These evaluation results should be transparently reported for researchers to be able to decide which feature operationalisations are accurate enough for their specific research objectives.

Most standard POS taggers require an additional script to count the number of occurrences of each tag in each text and to normalise these counts if the texts of the corpora are of different lengths. This process adds an extra step in the preprocessing of tagged corpus data for multivariable analyses. In contrast, MAT and the *MFTE Perl* are more accessible in that they output tables of normalised frequencies that can readily be used as input for statistical tools and functions.

Whilst the *MFTE Perl* was designed to be used without fix-tagging, its outputs include normalised frequencies of each feature per text, and its documentation includes

<footnote n="9">⁹ For instance, for the TOEFL iBT project (with a corpus of 3,839 texts totalling 543,000 words), Gray and Biber (2013: 18) report having recruited and trained ten fix-taggers in addition to two independent coders and a project research assistant for the manual corrections of problematic tags.</footnote>
detailed results of a thorough evaluation, it nonetheless failed to adequately meet our third aim of accessibility. This is because we believe that accessibility also entails ease of use. Regrettably, the *MFTE Perl* requires a separate installation of the *Stanford POS tagger* (Toutanova *et al.* 2003) which, even with detailed installation instructions, is likely to constitute a barrier for some linguists. This is not the case with the present version of the MFTE (see Section 2.5). Furthermore, the present version of the MFTE was written in *Python*, a more accessible programming language, with a large user base and hence many beginner-friendly tutorials and helper tools (e.g., *Anaconda* and *Anaconda Mini* for installation).

Finally, unlike MAT for *Windows*, the *MFTE Perl* also lacks a graphical user interface (GUI). We consider this to be an important aspect of making open-source tools accessible to the wider research community and argue that a genuinely accessible tagger ought to include a GUI for all major operating systems.

## 2.2. Specifications of the MFTE

Based on our triple motivation to improve the reproducibility, transparency, and accessibly of multivariable English corpus studies and our survey of the strengths and weaknesses of existing tools in these regards (see Section 2.1), we elaborated the tagger specifications for the MFTE by updating the specifications originally specified for the *MFTE Perl* (Le Foll 2021). These specifications are shown in Table 1.

<table><caption>Table 1: Tagger specifications for the MFTE</caption><thead><tr><th></th><th></th></tr></thead><tbody><tr><td>1.</td><td><strong>Identify a broad range of lexico-grammatical and semantic features of English</strong><br>a. that can each be meaningfully interpreted<br>b. to a satisfactorily high degree of accuracy (with precision and recall rates of > 90%)<br>c. without the need for human intervention<br>d. in a broad range of English registers<br>e. with standard American or British orthography.</td></tr><tr><td>2.</td><td><strong>Output</strong><br>a. the full tagged texts in plain text format for qualitative analyses of the tagger's accuracy<br>b. delimiter-separated values (DSV) files containing both raw and normalised feature counts per text.</td></tr><tr><td>3.</td><td><strong>Be available</strong><br>a. as source code under a GNU licence for researchers with programming skills to scrutinise, adapt, improve and re-use and<br>b. as a GUI with adequate documentation for researchers with basic computer skills to be able to run the programme in all major operating systems.</td></tr></tbody></table>
In what follows, we outline how we set out to meet these specifications. In Section 2.3, we list the linguistic features tagged by the MFTE and, in Section 2.4 we motivate their operationalisations, before explaining how to use the tagger and understand its outputs (Section 2.5).

## 2.3. Tagset

In line with the MFTE's intended application for descriptive linguistic analyses (such as multivariable analyses of register variation) as opposed to classification tasks, we explicitly focused on the identification of linguistic features that can be “meaningfully interpretable” (specification criterion 1a). By this, we mean that the “scale and values [of each feature] represent a real-world language phenomenon that can be understood and explained” (Egbert et al. 2020: 24).

When designing the feature portfolio of the *MFTE Perl*, Le Foll (2021) examined simplified Hallidayan system networks grammars (see, e.g., Bartlett and O'Grady 2017) in an attempt to minimise researcher bias in the selection of linguistic features. The *MFTE Perl* identifies and counts 75 linguistic features covering lexical density and diversity, fine-grained POS classes, verb tense and aspect, various frequent lexico-grammatical constructions, and a selection of verb semantic categories. The *MFTE Python* builds on this original set of features from the *MFTE Perl*, of which 74 were retained to form the 'simple tagset' of the new MFTE. These are listed in Table 2. In addition, the present version of the MFTE features an 'extended tagset' with more than 70 additional features, mostly semantic features inspired from Biber *et al.* (1999) and Biber (2006), operationalised on the basis of the features from the simple tagset. A full list of all features including examples and a description of their operationalisation can be found on the MFTE's GitHub repository. Note that the latest features can be found in the developmental branch of the repository.
<table><caption>Table 2: Features of the MFTE's simple tagset</caption><thead><tr><th>Feature</th><th>Tag</th><th>Example</th></tr></thead><tbody><tr><td>BE able to</td><td>ABLE</td><td>It should be <strong>able</strong> to speak back to you. Would you be <strong>able</strong> to?</td></tr><tr><td>Amplifiers</td><td>AMP</td><td>I am <strong>very</strong> tired. They were both <strong>thoroughly</strong> frightened.</td></tr><tr><td>Average word length</td><td>AWL</td><td>It's a shame that you'd have to pay to get that quality. [AWL = 42/12 = 3.5]</td></tr><tr><td>BE as main verb</td><td>BEMA</td><td>It was nice to just <strong>be</strong> at home. She's irreplaceable. It's best.</td></tr><tr><td>Coordinators</td><td>CC</td><td><strong>Instead</strong> of listening to us, he <strong>also</strong> told John and Jill <strong>but</strong> at least his parents don't know yet.</td></tr><tr><td>Numbers</td><td>CD</td><td>That's her number <strong>one</strong> secret. It happened on 7 February 2019.</td></tr><tr><td>Concessive conj.</td><td>CONC</td><td>Even <strong>though</strong> the antigens are normally hidden...</td></tr><tr><td>Conditional conj.</td><td>COND</td><td><strong>If</strong> I were you... Even <strong>if</strong> the treatment works...</td></tr><tr><td>Verbal contractions</td><td>CONT</td><td>I don't know. It isn't <strong>my</strong> problem. You'll have to deal.</td></tr><tr><td>Causal conjunctions</td><td>CUZ</td><td>He was scared <strong>because</strong> of the costume. Yeah, <strong>coz</strong> he hated it.</td></tr><tr><td>Demonstrative pronouns and articles</td><td>DEMO</td><td>What are you doing <strong>this</strong> weekend? Whoever did <strong>that</strong> should admit it.</td></tr><tr><td>Discourse/pragmatic markers</td><td>DMA</td><td><strong>Well</strong>, no they didn't say <strong>actually</strong>. Okay I guess we'll see.</td></tr><tr><td>DO auxiliary</td><td>DOAUX</td><td>Should take longer than it <strong>does</strong>. Ah you <strong>did</strong>. Didn't you?</td></tr><tr><td>Determiners</td><td>DT</td><td>Is that a new top? Are they <strong>both</strong> Spice Girls? On <strong>either</strong> side.</td></tr><tr><td>Downtoners</td><td>DWNT</td><td>These tickets were <strong>only</strong> 45 pounds. It's <strong>almost</strong> time to go.</td></tr><tr><td>Elaborating conjunctions</td><td>ELAB</td><td><strong>Similarly</strong>, you may, <strong>for example</strong>, write bullet points.</td></tr><tr><td>Emoji and emoticons</td><td>EMO</td><td>:-( :DD XD :)</td></tr><tr><td>Emphatics</td><td>EMPH</td><td>I do wish I hadn't drunk quite <strong>so</strong> much. Oh <strong>really</strong>? I <strong>just</strong> can't get my head around it.</td></tr><tr><td>Existential <em>there</em></td><td>EX</td><td><strong>There</strong> are students. And <em>there</em> is now a scholarship scheme.</td></tr><tr><td>Filled pauses and interjections</td><td>FPUH</td><td><strong>Oh</strong> noooooo, Tiger's furious! <strong>Wow</strong>! <strong>Hey</strong> Tom! Er I know.</td></tr><tr><td>Frequency references</td><td>FREQ</td><td>We should <strong>always</strong> wear a mask. He had found his voice <strong>again</strong>.</td></tr><tr><td>Going to constructions</td><td>GTO</td><td>I'm not <strong>gonna</strong> go. You're <strong>going</strong> to absolutely love it there!</td></tr><tr><td>Hedges</td><td>HDG</td><td>There seemed to be no <strong>sort</strong> of chance of getting out. She's <strong>maybe</strong> gonna do it.</td></tr><tr><td>HAVE got constructions</td><td>HGOT</td><td>He's <strong>got</strong> one. Has she <strong>got</strong> any?</td></tr><tr><td>Hashtags</td><td>HST</td><td><strong>#AcWri</strong> <strong>#BuyIGetIFree</strong></td></tr><tr><td>Prepositions</td><td>IN</td><td>The Great Wall <strong>of</strong> China is the longest wall <strong>in</strong> the world. There are towers <strong>along</strong> the wall.</td></tr><tr><td>Attributive adjectives</td><td>JJAT</td><td>I've got a <strong>fantastic</strong> idea! <strong>Cheap</strong>, <strong>quick</strong> and <strong>easy</strong> fix!</td></tr><tr><td>Predicative adjectives</td><td>JJPR</td><td><strong>That's right</strong>. <strong>One of the main advantages of being</strong> famous...</td></tr><tr><td>Lexical density</td><td>LDE</td><td>It's a <strong>shame</strong> that you'd have to <strong>pay</strong> to get that <strong>quality</strong>. [LDE = 3/14 = 0.21]</td></tr><tr><td>Like</td><td>LIKE</td><td>Sounds <strong>like</strong> me. And just <strong>like</strong> his father. I wasn't gonna <strong>like</strong> do it.</td></tr><tr><td>Modal <em>can</em></td><td>MDCA</td><td><strong>Can</strong> I give him a hint? You <strong>cannot</strong>. I <strong>can't</strong> believe it!</td></tr><tr><td>Modal <em>could</em></td><td>MDCO</td><td>Well, that <strong>could</strong> be the problem. <strong>Could</strong> you do it by Friday?</td></tr><tr><td>Modals <em>may</em> and <em>might</em></td><td>MDMM</td><td><strong>May</strong> I have a word with you? But it <strong>might</strong> not be enough.</td></tr><tr><td>Necessity modals</td><td>MDNE</td><td>I really <strong>must</strong> go. <strong>Shouldn't</strong> you be going now? You <strong>need</strong> not have worried.</td></tr><tr><td>Modal <em>would</em></td><td>MDWO</td><td><strong>Wouldn't</strong> you like to know? I'd like to think it works.</td></tr><tr><td><em>Will</em> and <em>shall</em> modals</td><td>MDWS</td><td>It <strong>won't</strong> do. Yes, it <strong>will</strong>. <strong>Shall</strong> we see?</td></tr><tr><td>Noun compounds</td><td>NCOMP</td><td><strong>the</strong> <strong>dungeon entrance</strong>; this rare <strong>winter phenomenon</strong></td></tr><tr><td>Total nouns (including proper nouns)</td><td>NN</td><td>on <strong>Monday 6 Aug</strong>, the <strong>U.S.</strong>, on the <strong>High Street</strong>, comprehension</td></tr></tbody></table>
<table><caption>Table 2: Features of the MFTE's simple tagset (Continuation)</caption><thead><tr><th>Feature</th><th>Tag</th><th>Example</th></tr></thead><tbody><tr><td>@mentions</td><td>NN</td><td><b>@gretathunberg</b> @MSF_france</td></tr><tr><td>BE-passives</td><td>PASS</td><td>He must have been <b>burgled</b>. They need to be <b>informed</b>.</td></tr><tr><td>Perfect aspect</td><td>PEAS</td><td>Have you <b>been</b> on a student exchange? He has been <b>told</b>.</td></tr><tr><td>GET-passives</td><td>PGET</td><td>He's gonna get <b>sacked</b>. She'll get me <b>executed</b>.</td></tr><tr><td>It pronoun reference</td><td>PIT</td><td><b>It</b> fell and broke. I implemented <b>it</b>. <b>Its</b> impact is unproven.</td></tr><tr><td>Place references</td><td>PLACE</td><td><b>It's</b> not <b>far</b> to go. I'll get it from <b>upstairs</b>. <b>It's</b> <b>downhill</b>.</td></tr><tr><td>Politeness markers</td><td>POLITE</td><td>Can you open the window, <b>please</b>? Would you <b>mind</b> giving me a hand?</td></tr><tr><td>s-genitives</td><td>POS</td><td>the world's two most populous country; my parents' house</td></tr><tr><td>Reference to the speaker/writer and other(s)</td><td>PP1P</td><td><b>We</b> were told to deal with it <b>ourselves</b>. It's not <b>ours</b> either.</td></tr><tr><td>Reference to the speaker/writer</td><td>PP1S</td><td><b>I</b> don't know. It isn't <b>my</b> problem. Nor is it <b>mine</b>.</td></tr><tr><td>Reference to addressee(s)</td><td>PP2</td><td><b>If</b> <b>your</b> model was good enough, <b>you</b>'d be able to work it out.</td></tr><tr><td>Single, female third person</td><td>PP3f</td><td><b>She</b> does tend to keep to <b>herself</b>, doesn't <b>she</b>?</td></tr><tr><td>Single, male third person</td><td>PP3m</td><td><b>He</b> is beginning to form <b>his</b> own opinions. I trust <b>him</b>.</td></tr><tr><td>Other personal pronouns</td><td>PPother</td><td><b>One</b> would hardly suppose that your eye was as steady as ever.</td></tr><tr><td>Progressive aspect</td><td>PROG</td><td>He wasn't <b>paying</b> attention. I'm <b>going</b> to the market.</td></tr><tr><td>Quantifiers</td><td>QUAN</td><td><b>Such</b> a good time in <b>half</b> an hour. She's got <b>all</b> these ideas. It happens <b>every</b> time.</td></tr><tr><td>Quantifying pronouns</td><td>QUPR</td><td>She said addressing <b>nobody</b> in particular. <b>Somebody</b> will.</td></tr><tr><td>Question tags</td><td>QUTAG</td><td>Do they? Were you? It's just it's repetitive, isn't it?</td></tr><tr><td>Other adverbs</td><td>RB</td><td><b>Unfortunately</b>, that's the case. <b>Exactly</b> two weeks.</td></tr><tr><td>Particles</td><td>RP</td><td><b>I'll</b> look it <b>up</b>. It's coming <b>down</b>. When will you come <b>over</b>?</td></tr><tr><td>So</td><td>SO</td><td>She had spent <b>so</b> many summers there. <b>So</b>, there you go.</td></tr><tr><td>Split auxiliaries/infinitives</td><td>SPLIT</td><td><b>I</b> would actually drive. You <b>can</b> just so tell. I <b>can't</b> imagine it.</td></tr><tr><td>Stranded prepositions</td><td>STPR</td><td>We've got more than can be accounted <b>for</b>. Open the door and let them <b>in</b>.</td></tr><tr><td>Subordinator that omission</td><td>THATD</td><td>I mean <b>[THATD]</b> you'll do everything. I thought <b>[THATD]</b> he just meant our side.</td></tr><tr><td>That relative clauses</td><td>THRC</td><td><b>I'll</b> just run a cable <b>that</b> goes from here to there.</td></tr><tr><td>That subordinate clauses</td><td>THSC</td><td>Did you know <b>that</b> the calendar we use today was started by Julius Caesar?</td></tr><tr><td>Time references</td><td>TIME</td><td><b>It</b> will <b>soon</b> be possible. <b>Now</b> is the time. I haven't it <b>yet</b>.</td></tr><tr><td>Reference to more than one non-interactant and single they reference</td><td>TPP3t</td><td>The text allows readers to grapple with <b>their</b> own conclusions. Do you see <b>them</b>?</td></tr><tr><td>Lexical diversity</td><td>TTR</td><td><b>It's</b> a shame <b>that</b> you'd have to <b>pay</b> to get that <b>quality</b>. [TTR = 12/14 = 0.85]</td></tr><tr><td>URL and e-mail addresses</td><td>URL</td><td><b>www.faz.net</b> <a href="https://twitter.com smith@gmail.au">https://twitter.com smith@gmail.au</a></td></tr><tr><td>Past tense</td><td>VBD</td><td><b>It</b> fell and broke. I implemented <b>it</b>. If I were rich.</td></tr><tr><td>Non-finite verb -ing forms</td><td>VBG</td><td>He texted me <b>saying</b> no. He just started <b>laughing</b>.</td></tr><tr><td>Non-finite -ed verb forms</td><td>VBN</td><td>These include cancers <b>caused</b> by viruses. Have you read any of the books <b>mentioned</b> there?</td></tr><tr><td>Imperatives</td><td>VIMP</td><td><b>Let</b> me know! In groups, <b>share</b> your opinion and <b>take</b> notes.</td></tr><tr><td>Present tense</td><td>VPRT</td><td><b>It's</b> ours. Who <b>doesn't</b> love it? <b>I know</b>.</td></tr><tr><td>Direct WH-questions</td><td>WHQU</td><td><b>What</b>'s happening? <b>Why</b> don't we call the game off? <b>How</b>?</td></tr><tr><td>WH subordinate clauses</td><td>WHSC</td><td>I'm thinking of someone <b>who</b> is not here today. Do you know <b>whether</b> the banks are open?</td></tr><tr><td>Negation</td><td>XX0</td><td><b>Why</b> <b>don't</b> you believe me? There is <b>no</b> way. <b>Nor</b> am I.</td></tr><tr><td>Yes/no questions</td><td>YNQU</td><td>Have you thought about <b>giving up</b>? Do you mind?</td></tr></tbody></table>
The MFTE performs feature extraction in several steps. First, each text is tagged with basic POS labels using the POS tagger from the stanza Python library (Qi et al. 2020). Then, several loops of rule-based algorithms are run to refine some of the analyses of the POS tagger and to identify further linguistic features on the basis of syntactic patterns defined by a combination of regular expressions and dictionary lists. The lists are based on Biber (1988), Biber et al. (1999), Biber (2006), and the COBUILD (Sinclair et al. 1990).

Whilst many of the features of the simple tagset may look superficially like the 1988 version of the *Biber tagger* (and hence also to the MAT), their operationalisations often differ substantially. In particular, the algorithms used to capture verb tense, aspect, and voice are fundamentally different. For example, rather than tag the perfect aspect onto the auxiliary *HAVE* as in MAT, the MFTE assigns the ‘perfect aspect tag’ (PEAS) to the past participle form of the verb, whilst the auxiliary is marked for tense with either the VBD or the VPRT tag. Similarly, the ‘passive voice tag’ (PASS) is assigned to the past participle form rather than to the verb *BE*. These new operationalisations make possible the creation of a distinct, linguistically meaningful, VBN variable, which only includes non-finite uses of past participle forms. As the MFTE also identifies verbs in the progressive aspect (PROG) and the *going*-to construction (GTO), non-finite uses of present participle forms can also be accounted for in a separate variable (VBG). Moreover, these verb feature operationalisations are optimised for a range of syntactic patterns, e.g., by allowing for various combinations of intervening adverbs, negation, and paralinguistic sounds in verb phrases involving auxiliaries. Other lexico-grammatical features from the MFTE’s simple tagset that are not typically found in lexico-grammatical taggers include question tags (QUTAG), imperatives (VIMP), and emojis or emoticons (EMO).

Various strategies were employed to deal with multifunctional/polysemous lexical items: in many cases, they were included in the semantic categories corresponding to their most frequent functions (e.g., *only* is assigned to the category of downtoners rather than hedges). When this proved too error-prone, infrequent items were excluded. Moreover, for two highly frequent multifunctional words (*so* and *like*) separate feature categories were created to capture all potentially problematic cases, i.e., all occurrences of *so* and all occurrences of *like* tagged as a preposition or adjective by the stanza POS tagger. Both words are frequently used as discourse markers and fillers in conversation. These uses are
very difficult to automatically disambiguate. Consequently, if they are not excluded, they
run the risk of severely distorting the frequencies of adverbs and prepositions in
conversational texts. More recent versions of the *Biber tagger* make use of different
algorithms and probabilities depending on the mode/register of the texts to be tagged to
(partially) circumvent this issue (Gray 2019: 46). We believe that such a procedure bears
the risk of predetermining patterns of occurrences and have therefore opted to relegate
these items to separate feature categories for *like* (when not tagged as a verb) and *so*,
instead. While this is by no means a perfect solution, it allows users of the MFTE to
decide —depending on the resources available to them— whether to perform manual fix-
tagging to assign the correct functional tag to each occurrence of *so* and *like*, or to exclude
these tokens by not entering the counts of the SO and LIKE variables in their analyses.

When using POS taggers, Brezina (2018: 192) suggests removing paralinguistic
sounds (e.g., *um*, *er*, *mhm*, *mm*) from the spoken data as these are frequently misidentified
as nouns, thus potentially vastly overestimating the noun count in natural conversation
corpora in which these sounds have been transcribed. Rather than remove these tokens as
part of the pre-processing of the corpus texts, we opted to retain them as they constitute
a defining characteristic of natural spoken language. The MFTE counts these as a self-
contained linguistic variable (FPUH), which, like all the linguistic features counted by
the MFTE, the user is free to either include or exclude from their analyses, depending on
their texts (considering, for instance, how reliably these were transcribed across of the
texts under study) and research questions.

A final major departure from the feature extraction principle applied by some
lexico-grammatical taggers that is worth mentioning concerns the treatment of multiword
items. The 1988 version of the *Biber tagger* and MAT (Nini 2014; 2019), for example,
tag the first token of the multiword *on the other hand* as a conjunct and assigns NULL
tags to the remaining tokens, as illustrated in (1) below. This means that, when tagged
with these taggers, the string *on the other hand* counts as just one conjunct. In contrast,
the MFTE assigns the conjunct tag (CONJ) in addition to the usual preposition,
determiner, adjective, and noun tags, as shown in (2). NULL tags were only retained for
a select few multiword units that largely resist compositional analysis: *of course* (tagged
as a discourse marker, DMA), *all right* (DMA) and *no one* (tagged as a quantifying
pronoun, QUPR).
(1) CONJ the_NULL other_NULL hand_NULL

(2) on_IN CONJ the_DT other_JJ hand_NN

The three semantic verb categories employed in Biber (1988) are only used in the MFTE for the identification of the *that*-omissions in subordinate clauses (THATD). Instead, the MFTE's extended tagset adopts, with some minor corrections, the semantic categories described in Biber *et al.* (1999) and Biber (2006). As a result, in addition to ten semantic verb features, the extended tagset also includes fine-grained semantic categories for adverbs, nouns, and adjectives, as well as combined grammatico-semantic features such as ‘*to* clauses preceded by verbs of desire’ (ToVDSR) or ‘stance nouns without prepositions’ (NSTNOther). The extended tagset also includes comparative and superlative constructions (see GitHub repository for details).

The user should be aware that the tables of counts generated when using the extended tagset include some composite features. These consist of aggregates of individual feature counts, e.g., the variable PASSall contains the sum of the counts of BE-passives (PASS) and GET-passives (PGET), two features from the simple tagset. It goes without saying that, to avoid redundant correlations, these aggregate counts should not be entered in MD analyses together with their respective individual feature counts. In proposing this large number of lexical, grammatical, and semantic features, we are not suggesting that it makes sense to enter them all in multi-feature analyses of English, but rather that, with the help of our detailed documentation, users can make informed, linguistically motivated choices as to which features are relevant and reliable enough (see evaluation results in Section 4.3) for their use cases.

## 2.4. Usage

Since the alpha version was first released in April 2023, the MFTE has been accessible under an open-source GPL-3.0 licence. It can be downloaded from a dedicated GitHub repository at https://github.com/mshakirDr/MFTE. Detailed installation and usage instructions can be found on the main page of the GitHub repository.

The MFTE relies on commonly utilised *Python* dependencies whose installation are not particularly involved. To expand its accessibility to non-programming linguists, we have also released the programme as an executable file for *Windows*. Once this executable file has been downloaded, the GUI can be run without installing *Python* or any
dependencies and is therefore the most user-friendly version of the MFTE (currently only for *Windows*; see Figure 1).

<figure><img src="image_2.png" /><figcaption>Figure1: Screenshot of the downloaded MFTE GUI executable for *Windows* (MFTElection.exe), alongside the running MFTE GUI application</figcaption></figure>

For *Mac OS*, *Linux*, and *Windows*, both the command-line version and the GUI version can be run from a terminal. The command-line version is straightforward (see Figure 2), requiring only four arguments:

1. the path to the folder containing the text files to be tagged;

2. the number of words used to calculate the type-token ratio (TTR)—as in the MAT, the default is 400 but this should be set lower if any texts are shorter than 400 words;

3. an option to use the extended tagset (the default is True);

4. an option to tag multiple files in parallel (this reduces running time for large corpora significantly but may also cause high CPU usage, hence the default setting is False).
<figure><img src="image_3.png" /><figcaption>Figure 2: Screenshot of the command line version of the MFTE (example command highlighted in red)</figcaption></figure>

## 2.5. Outputs

The outputs of the MFTE are “recorded and encoded in the annotated corpus” such that they are “explicit and recoverable” (McEnery et al. 2006: 31). To this end, the MFTE creates two versions of each processed text files:

1. the texts as tagged using the underlying POS tagger (in a subfolder labelled 'POS_Tagged') and

2. the texts as subsequently tagged by the MFTE (in a subfolder labelled 'MFTE_Tagged').

This ensures that it is possible to trace back the origin of tagging errors and to rectify them by either modifying the code of the tagger or manually fix-tagging the tagger output.

The MFTE also outputs three tables of feature counts as comma-separated text files. The filenames include the name of the folder in which the original texts are stored as a prefix. The folders are named following this pattern:

1. `[prefix]_normed_100words_counts.csv

2. `[prefix]_normed_complex_counts.csv

3. `[prefix]_raw_counts.csv
Each CSV file consists of a data matrix in which each row corresponds to a text file from the tagged corpus and each column to a linguistic feature. In all three tables, the first five columns of the count tables are identical: the first column lists the filenames, the second the total number of words in each text as used for word-based normalisation (excluding fillers, see online appendix). The remaining columns correspond to the linguistic features listed in Table 2 for the simple tagset followed by those of the extended tagset (see online appendix), if this option was selected.

The difference between the three tables that the MFTE outputs is that the first contains feature frequencies normalised per 100 words. In the second table, ‘complex’ normalised feature frequencies are calculated based on the three normalisation baselines listed in the fifth column of the table in the appendix. This means that, in this table, the present tense variable (VPRT) represents the percentage of finite verbs in the present tense. As such, it can range from zero, i.e., texts in which no single verb is in the present tense to 100, i.e., texts that are exclusively in the present tense, and therefore does not violate the assumptions of the binomial distribution required for many statistical methods commonly used in corpus linguistics research (Wallis 2020: 56). For details on the normalisation units used for each feature and the rationale behind these choices, see Le Foll (2021: 20–23; 2024: 120–124). Lastly, the MFTE outputs a table of raw counts. Unless the text samples of the examined corpus are all the same length, this table should not be used as is for any statistical analyses; however, it may be used by researchers who wish to implement their own normalisation baseline(s). We have also found it very useful for test purposes, i.e., to check how the tagger deals with certain strings.

### 3. EVALUATING TAGGER ACCURACY

Given that taggers provide “the foundation for MD studies” (Biber 2019: 14) and other multivariable studies, their accuracy is crucial for drawing reliable conclusions that can contribute to building cumulative knowledge. In a comparison of the accuracy of just four linguistic features as identified by three different taggers, Picoral et al. (2021) reported differences large enough to lead to significantly different conclusions. Despite this, Goulart and Wood (2021: 123) concluded that tagger accuracy is underreported in the majority of published MD studies. This is not surprising, as the large number of linguistic features typically entered in such analyses makes comprehensive evaluations of tagger
accuracy an arduous task. In the context of most small- to mid-scale projects, it is simply not feasible. This is why we believe that, to meet our aim for providing the research community with an accessible tagger, it is necessary to conduct and publish the results of a comprehensive evaluation of the accuracy of the MFTE.

When evaluating the performance of a tagger, several accuracy measures can be used, as shown in Table 3 below. On the one hand, the number of correct tags, i.e., true positives, can be counted and, on the other hand, the number of incorrect tags, i.e., false positives. The simplest measure of accuracy is ‘precision’: the ratio of true positives to all tags assigned by the tagger. ‘Recall’, by contrast, is the ratio of true positives to all instances of a given tag in the data. It takes into account both true positives and false negatives (instances where a particular tag should have been assigned by the tagger but was not). In other words, recall provides a measure of the tagger’s ability to correctly identify and classify all instances of a particular feature. While both precision and recall provide valuable information about a tagger’s performance, they only give a partial picture of its accuracy. If precision is high but recall is low, the tagger may be too conservative, tagging only those instances about which it is particularly confident. Conversely, a tagger with low precision but high recall will assign some tags too liberally.

<table><caption>Table 3: Summary of the terminology frequently used in tagger performance evaluations</caption><thead><tr><td>Term</td><td>Definition</td></tr></thead><tbody><tr><td>True positive</td><td>Feature correctly identified as X</td></tr><tr><td>True negative</td><td>Feature correctly identified as not X</td></tr><tr><td>False positive</td><td>Feature incorrectly identified as X</td></tr><tr><td>False negative</td><td>Feature incorrectly identified as not X</td></tr><tr><td>Precision</td><td>True positive count / (true positive count + false positive count)</td></tr><tr><td>Recall</td><td>True positive count / (true positive count + false negative count)</td></tr><tr><td>F1 score</td><td>2 * (precision * recall) / (precision + recall)</td></tr></tbody></table>

In an ideal world, a tagger would have 100 per cent precision (i.e., all assigned tags are correct) and 100 per cent recall (i.e., all features are labelled with all the correct tags). In practice, however, attempts to increase precision on any one feature will usually result in lower recall rate for that particular feature (and many others) and vice versa. An important aspect of tagger development therefore consists in striking the appropriate balance between precision and recall. If it is feasible to complement automatic tagging with a manual fix-tagging phase, then it makes sense to prioritise recall. If, however, a tagger is to be used without any manual intervention, both precision and recall are important. This
is why a third accuracy measure is often calculated: the F1 score, which combines
precision and recall (see Table 3). It ranges from 0 (entirely inaccurate) to 1 (perfectly
accurate) and provides a single number that can be used to compare the performance of
different taggers.

A common strategy in POS tagger evaluations is to report one overall measure of
per-token accuracy across all tags (typically slightly over 97%, e.g., Manning 2011). This
is common practice in NLP but invariably leads to misleadingly inflated results. Indeed,
some of the most frequent tokens, in particular punctuation markers and determiners, are
both highly frequent and extremely easy to ‘get right’ (e.g., *the* and *I*). By contrast, per-
sentence accuracy rates of POS taggers tend to be considerably more modest (hovering
around 50–57%) and considerably lower rates for non-standard varieties and registers for
which there is little training data (Manning 2011).

Although it is crucial for users of a tagger to be aware of its per-feature accuracy
to gauge the reliability of its annotation, few taggers have documentation that includes
detailed results of a comprehensive accuracy evaluation study. The *Biber tagger*
constitutes an exception. We are aware of two comprehensive evaluations of its tagging
accuracy: Biber and Gray (2013: 16–18) and Gray (2015). The first, however, only reports
post-fix-tagging precision and recall rates (Biber and Gray 2013: Appendices C and D).
In other words, users of the *Biber tagger* who do not have access to the fix-tagging scripts
used as part of this project cannot expect similar accuracy levels when tagging their own
texts. Moreover, in Biber and Gray (2013: 18) the same 5 per cent sample of texts that
were originally manually checked for tagging errors was analysed to calculate the
reported precision and recall rates after fix-tagging. This procedure entails the risk of
inflating the tagger’s accuracy metrics as the fix-tagging scripts may have been
constructed based on errors specific to the sample. By contrast, Gray (2015) reports both
“initial reliability rates” of the uncorrected *Biber tagger* output and “final reliability rates
after [fix-tagging] scripts” (Gray 2015: Appendix B) on excerpts of a random sample of
15 research articles across different disciplines and registers. On close inspection of the
initial reliability rates, it transpires that not all of the features of the *Biber tagger* (or rather
the version used in this particular project) seem suitable for use without some manual or
semi-automatic ‘fix-tagging’ procedure, as recall and precision rates for some features
are well below 90 per cent.
An anonymous reviewer drew our attention to the fact that several recent PhD projects conducted at Northern Arizona University (the ‘home’ of the *Biber tagger*) include small-scale evaluations of the accuracy of more recent versions of the *Biber tagger* for selected features. Goulart (2022: Appendix A) reports recall and precision rates for 29 features in L1 and L2 academic writing. Wood (2023: 59–63) does so for 28 features in a sample of statutory texts, while Dixon (2022: 71–72) evaluated the tagger’s accuracy for 20 features in 17 texts representing gaming language. Before fix-tagging, Goulart (2022: Appendix A) reports precision and recall rates well above 90 per cent for all the selected features except nominalisations. In Wood’s (2023) corpus of statutory language, on the other hand, accuracy is much lower for many of the selected features (e.g., noun and adjective complement clauses, agentless passives, clausal and phrasal coordinated conjunctions). These results confirm that the accuracy of taggers varies widely across different linguistic features and depends on the type of texts being tagged. While it is inevitable that such labour-intensive evaluations cannot realistically be carried out on large samples, it is problematic that most of these evaluations do not report the number of tags on which recall and precision were calculated. This means that confidence intervals around the reported accuracy metrics cannot be computed (see Section 4.2). Dixon’s (2022: 72) evaluation constitutes an exception: the figures reveal that some of the reported accuracy metrics were calculated on as few as three or four occurrences, thus confirming that the results of such small-scale, project-specific evaluations cannot meaningfully be used in other projects.

The documentation of the *MFTE Perl* (Le Foll 2021) includes a detailed evaluation of the tagger on a stratified random sample from the *British National Corpus 2014* (BNC2014; Brezina et al. 2021). Samples of 24 texts covering written and spoken British English from a range of registers were tagged using the *MFTE Perl*. The resulting 31,311 tags were manually annotated by two linguists. For each tag, the annotators had three options: 1) mark it as correct, 2) mark it as incorrect and assign the correct tag, or 3) mark it as an unclear or ambiguous context/token for which no tag could reliably be assigned. Most of the 75 features of the *MFTE Perl* achieved a “satisfactorily high degree of accuracy” (Le Foll 2021: 14) with high rates of both recall and precision. F1 scores below 90 per cent were reported for 10 features. These include two rare features for which Le Foll (2021: 45) warns that accuracy rates are unreliable because they are based on very few occurrences.
## 4. EVALUATION

In the following, we report on the accuracy of the simple tagset of the MFTE on British and American English texts from a range of registers. Given how misleading overall accuracy rates may be (see Section 3), we report recall, precision, and the combined F1 measure for each feature of the simple tagset, thus allowing users to make an informed decision as to which set of features they wish to include in their analyses and which they would like to exclude depending on their research questions and the characteristics of their data. Whenever possible, we also report 95 per cent confidence intervals around these accuracy metrics.

### 4.1. *Data*

In line with the specifications of the MFTE (see Section 2.2), for the evaluation of the *MFTE v.1.0*, we chose a stratified random sample of 30 texts from the BNC2014 and 30 texts from the *Corpus of Contemporary American English* (COCA; Davies 1990) representing diverse registers, as shown in Table 4. To maximise the potential for text/topic-specific tagging errors, we analysed samples of roughly 1,000 tokens from the longer texts in our evaluation sample. This was motivated by an observation from the evaluation of the *MFTE Perl* in which most tagging errors were found to be text/topic-specific and therefore highly clustered (see Le Foll 2021: 25–43). The sampled texts from the spoken subcorpus of the BNC2014 were pre-processed to remove the anonymisation tags and metadata following the procedure documented in Le Foll (2021: 28).
<table><caption>Table 4: Evaluation data</caption><thead><tr><th>Corpus</th><th>Subcorpus</th><th>Number of texts</th><th>Number of tags</th></tr></thead><tbody><tr><td>BNC2014</td><td>Academic writing</td><td>3</td><td>2,617</td></tr><tr><td>BNC2014</td><td>E-Language: Blogs</td><td>3</td><td>2,092</td></tr><tr><td>BNC2014</td><td>E-Language: E-Mails</td><td>2</td><td>1,964</td></tr><tr><td>BNC2014</td><td>E-Language: Forums</td><td>2</td><td>2,822</td></tr><tr><td>BNC2014</td><td>E-Language: Reviews</td><td>2</td><td>3,291</td></tr><tr><td>BNC2014</td><td>E-Language: Social Media Posts</td><td>3</td><td>3,263</td></tr><tr><td>BNC2014</td><td>E-Language: Text Messages</td><td>1</td><td>396</td></tr><tr><td>BNC2014</td><td>Fiction</td><td>3</td><td>4,144</td></tr><tr><td>BNC2014</td><td>News: Magazines</td><td>2</td><td>2,363</td></tr><tr><td>BNC2014</td><td>News: Newspapers</td><td>6</td><td>5,036</td></tr><tr><td>BNC2014</td><td>Spoken: Conversation</td><td>3</td><td>5,312</td></tr><tr><td>COCA</td><td>Academic writing</td><td>3</td><td>2,877</td></tr><tr><td>COCA</td><td>E-Language: Blogs</td><td>3</td><td>3,337</td></tr><tr><td>COCA</td><td>E-Language: Web Pages</td><td>3</td><td>2,416</td></tr><tr><td>COCA</td><td>Fiction</td><td>3</td><td>3,129</td></tr><tr><td>COCA</td><td>News: Magazines</td><td>3</td><td>2,202</td></tr><tr><td>COCA</td><td>News: Newspaper Articles</td><td>4</td><td>2,342</td></tr><tr><td>COCA</td><td>News: Newspaper Opinion Pieces</td><td>4</td><td>2,993</td></tr><tr><td>COCA</td><td>Spoken: Conversation</td><td>4</td><td>6,015</td></tr><tr><td>COCA</td><td>Spoken: TV/Movies</td><td>3</td><td>2,541</td></tr><tr><td>Totals</td><td></td><td>60</td><td>61,154</td></tr></tbody></table>

## 4.2. Methodology

The 60 files tallied in Table 4 were tagged using the *MFTE Python v.1.0* with its simple tagset. The resulting tagged text files were then converted to a spreadsheet format for manual evaluation.<fnref n="10" /> Each tag was marked as either correct, unclear, or incorrect. In the case of an incorrect tag, a corrected version of the tag was added to the corresponding column. In addition, tags were added where they were missing.<fnref n="11" /> These 60 spreadsheet files were subsequently processed and merged using custom *R* functions.

<footnote n="10">¹⁰ The evaluation was performed by the first author and her research assistant, Tatjana Winter, whom we thank for her meticulous work.</footnote>

<footnote n="11">¹¹ For details of the procedure, see Le Foll (2021: 29–33) and Le Foll and Shakir (2023).</footnote>
Statistics and data visualisations were computed in *R* and *Python* (see GitHub
repository for data and code). Bootstrap simulation was used to calculate 95 per cent
confidence intervals (CI) for the precision, recall and F1 measures of each feature based
on the results of 1,000 bootstrapped samples (in a procedure inspired by Picoral *et al.*
2021). We performed this task in *Python* and applied it to all tags for which there were
more than 100 occurrences in the 60 evaluation files (see Table 4).

4.3. *Results*

Of the 61,154 manually reviewed tags, 294 (0.48%; 95% CI [0.43–0.54%]) were deemed
by the human annotators to be ‘unclear’ due to misspellings, text processing errors,
ambiguous contexts, or fragmented sentences (particularly in the COCA data where 5%
of each text was removed for copyright reasons). A further 8,506 tags were excluded from
the evaluation metrics because they are not tallied in the tables of frequencies generated
by the MFTE. As these include all the tags denoting punctuation marks, as well as foreign
words, symbols, and other non-word tokens, they would also have considerably inflated
the overall accuracy metrics. Excluding these, the number of correctly assigned tags
across the 74 linguistic features of the MFTE simple tagset was 51,139. This corresponds
to an overall precision of 97.13 per cent (95% CI: 96.99–97.23%) across the 60 evaluation
files. Figure 3 shows the recall, precision and F1 measures for each feature of the simple
tagset with at least 100 occurrences in the evaluation corpus. The error bars correspond
to bootstrapped 95 per cent CI. The colours indicate how many times each tag occurred
across the 60 evaluation files (note that the colour scale is logarithmic).

Twelve features from the simple tagset did not meet the precision and/or recall rates
of at least 90 per cent stipulated in the tagger specifications (see Section 2.2), although
five of these do have F1 scores above 90 per cent. The least accurate features are stranded
prepositions (STPR), verbs in the imperative (VIMP), *that* omission (THATD), WH-
questions (WHQU), non-finite past participle forms (VBN), and *GET*-passives (PGET).
<figure><img src="image_4.png" /><figcaption>Figure 3: Precision, recall, and F1 score with bootstrapped 95 per cent CI for features with ≥100 occurrences across the 60 evaluation files</figcaption></figure>
Figure 4 displays the most frequent tagger errors as clusters of red points. The y-axis lists the tags as assigned by the MFTE, whilst the x-axis corresponds to those assigned by the two human annotators. For the sake of readability, only the most frequent tags are included in Figure 4. The code provided in the repository may be used to compute and examine the full matrix. The figure shows that notable clusters of errors involve the confusion of infinitives (VB) vs. imperatives (VIMP), attributive (JJAT) vs. predicative (JJPR) adjectives, non-finite past participles (VBN) vs. finite verbs in the perfect aspect (PEAS), and WH-questions (WHQU) vs. WH-subordinate clauses (WHSC). We can also see that several clusters of red points involve the tag ‘NONE’. NONE is not part of the tagset. We used this tag as a placeholder to indicate that the MFTE assigned an unwarranted second or third-order tag to this token (e.g., when two adjacent nouns were incorrectly identified as a noun compound by the tagger), or to indicate that a necessary tag was omitted (e.g., if *am* in *I am happy* was assigned the present tense tag, but not the one for *BE* as a main verb).

<figure><img src="image_5.png" /><figcaption>Figure 4: Confusion matrix showing mismatches between the MFTE output (Tag) and the human annotators' assessments (TagGold) in red</figcaption></figure>
As was to be expected, the MFTE performs slightly less well on internet and spontaneously spoken registers than on professionally written and edited texts such as those typically found in academic writing and fiction (see Table 5). For most purposes, however, these differences are unlikely to be significant. The code provided in the repository also allows for the calculation of per-feature accuracy measures for each of the six broad register categories of the BNC2014 and COCA.

<table><caption>Table 5: Overall precision of the MFTE across the broad register categories</caption><thead><tr><td>Register</td><td>Accuracy</td><td>Lower CI</td><td>Upper CI</td></tr></thead><tbody><tr><td>Academic writing</td><td>97.83</td><td>97.37</td><td>98.23</td></tr><tr><td>E-language</td><td>96.56</td><td>96.27</td><td>96.83</td></tr><tr><td>Fiction</td><td>97.89</td><td>97.49</td><td>98.24</td></tr><tr><td>News writing</td><td>97.77</td><td>97.50</td><td>98.02</td></tr><tr><td>Spoken</td><td>96.62</td><td>96.28</td><td>96.94</td></tr></tbody></table>

## 5. DISCUSSION

Based on the results of the evaluation, we now examine the extent to which the MFTE meets the tagger specifications elaborated in Section 2.2. We further discuss the limitations of both the evaluation and the tagger itself.

With a total of 74 ‘simple’ and over 70 additional ‘extended’ features covering fine-grained part-of-speech classes, tenses and aspects, frequent lexico-grammatical constructions, and numerous semantic categories, the MFTE’s feature portfolio meets the first criterion of identifying a broad range of lexico-grammatical and semantic features of English (see Table 1 in Section 2.2). Aside from the SO and LIKE features (see Section 2.4) they can readily be meaningfully interpreted (criterion 1a). Of the features in the simple tagset, all but twelve features reached “a satisfactorily high degree of accuracy” (see Table 1), with both precision and recall rates of > 90 per cent (criterion 1b). That said, it is worth remembering that some of the per-feature precision and recall rates presented in this study are based on relatively very few data points (see section 4.3).

Some of the most frequently mistagged features are among the problematic features for which some users of even the more recent versions of the *Biber tagger* have reported performing (semi-)automatic fix-tagging (e.g., features involving various uses of *that* and non-finite *-ed* and *-ing* clauses, see Goulart 2022 or Wood 2023). Low accuracy metrics
for some of these features may be the price of meeting specification criterion 1c: feature
identification “without the need for human intervention” (see Section 2.2). For some
features, we consider some of relatively low accuracy rates to be necessary to meet
criterion 1a: if linguistic features are to be “meaningful” (in the sense of functionally and
linguistically interpretable), it is crucial to distinguish, for example, non-finite past
participle forms (VBN) from finite verbs in the perfect aspect (PEAS), and imperatives
(VIMP) from infinitives (VB). Many existing taggers do not do this and therefore achieve
higher accuracy rates overall, but the features are less readily interpretable. In sum, the
results of the evaluation suggest that, at least for the simple tagset, most feature counts
can be entered into multivariable analyses “without the need for human intervention”
(1c).

While the MFTE struggled with e-language most (especially text messages and
social media posts), the results of our evaluation nevertheless confirm that the MFTE
performs sufficiently well across “a broad range of English registers” (thus meeting
criterion 1d) with texts in “standard American or British orthography” (thus meeting
criterion 1e). We recommend that researchers carry out additional evaluation tests if they
intend to use the MFTE for other registers and/or varieties of English. Particular care
should be taken when applying the extended tagset of the MFTE, as the accuracy of these
features has not been subject to such a systematic evaluation.

The outputs of the MFTE also conforms to the tagger specification. Not only does
the MFTE produce a table of raw counts (thus allowing researchers to apply their own
normalisations) and two tables of normalised counts (criterion 2b), but it also saves the
tagged texts for detailed examination of the texts themselves (criterion 2a). This is
important to ensure full transparency of the tagging process and to verify the accuracy of
the tagger’s output.

To meet the final requirement of the tagger specification, the MFTE source code
and all the additional evaluation materials are available under a GPL-3.0 licence for use
and scrutiny by the research community (criterion 3a). Criterion 3b is also satisfied with
the publication of step-by-step instructions on how to install and run the MFTE on the
landing page of its GitHub repository, together with the present article describing the
development and evaluation of the tagger.
## 6. CONCLUSION

In this article, we have presented the *Multi-Feature Tagger of English* (MFTE), an open-source tool designed for multivariable analyses of English. Characterised by transparency, adaptability, and accessibility, the MFTE offers promising avenues for future research endeavours in line with the principles of Open Science. Unlike ‘standard’ POS taggers such as the *Stanford tagger* (Toutanova et al. 2003) or CLAWS (Leech et al. 1994; Rayson and Garside 1998), its output does not require any additional processing (i.e., it outputs tables of counts with different normalisation options) and aims to tag only linguistically meaningful, functionally relevant features. As a free tool, the MFTE can contribute to making multivariable analysis of English more accessible to researchers and students from institutions with fewer resources.

While MAT and the initial version of the MFTE are in *Perl*, the new MFTE runs in *Python*, a programming language increasingly familiar to linguists. We hope that both the use of an accessible, object-based language and of an open-source licence will encourage colleagues not only to make use of the tagger, but also to contribute improvements. Our hope is that, as the tool gains traction within the research community, collaborative efforts will lead to further enhancements, expanding the tagset and refining the tagger’s performance across varieties and text types. Compared to the *MFTE Perl*, this new *Python* version benefits from being written in an actively developed language with a large user base, thus facilitating updates and improvements from developers (e.g., the integration of additional features using native parser libraries in *Python*).

In addition, the transparency and accessibility of the MFTE may also inspire linguists to develop similar taggers for languages other than English. Indeed, although MD analysis is, in theory, applicable to any language, in practice, most MD studies to date have examined varieties of English, which we believe is in part attributable to the lack of open-source taggers of lexico-grammatical features for languages other than English.

In discussing the results of the extensive evaluation of the MFTE’s simple tagset, we have also acknowledged the limitations of the tagger. Its accuracy will undoubtedly vary with respect to registers, topic domains, and varieties of English not included in the evaluation corpus. The extended tagset has not yet been systematically evaluated, and its reliance on dictionary lists for the semantic features is clearly a limitation. Refining and
updating these lists will be essential for the continued accuracy of the tagger. Future
studies could explore the generation of tailored test data using Large Language Models
(LLMs) as a means of evaluating the precision and recall rates of infrequent linguistic
features. Finally, although the results of the evaluation are considerably more detailed
than those of most linguistic taggers, they should nonetheless be interpreted with an
awareness of the inherent challenges of making objective, categorical judgments when
interpreting complex and often ambiguous linguistic phenomena. It would be misleading
to suggest that these judgments are theory-free. As Gray (2019: 45) points out in an article
focusing on the tagging and counting of linguistic features for MD analysis, “conflicting
POS categorisation reflects a different grammatical interpretation or theory of the nature
of this word.”

With these considerations in mind, we also hope that the MFTE will not only make
a significant contribution to multivariable corpus linguistics research, but also stimulate
ongoing methodological discussions on the transparency, validity, and reliability of the
tools and methods used in corpus linguistics research. Ultimately, we hope that, in the
near future, making research materials, data, and code available alongside linguistics
publications will no longer be the exception (Wieling et al. 2018; Bochynska et al. 2023),
but the norm.

REFERENCES

Baker, Monya. 2016. 1,500 scientists lift the lid on reproducibility. *Nature* 533/7604: 452–454.

Barlett, Tom and Gerard O'Grady eds. 2017. *The Routledge Handbook of Systemic Functional Linguistics*. London: Routledge.

Berber Sardinha, Tony, Marcia Veirano Pinto, Cristina Mayer, Maria Carolina Zuppardi and Carlos Henrique Kauffmann. 2019. Adding registers to a previous multidimensional analysis. In Tony Berber Sardinha and Marcia Veirano Pinto eds. *Multidimensional Analysis: Research Methods and Current Issues*. New York: Bloomsbury, 165–188.

Berez-Kroeker, Andrea L., Lauren Gawne, Susan Smythe Kung, Barbara F. Kelly, Tyler Heston, Gary Holton and Peter Pulsifer. 2018. Reproducible research in linguistics: A position statement on data citation and attribution in our field. *Linguistics* 56/1: 1–18.

Biber, Douglas. 1984. *A Model of Textual Relations within the Written and Spoken Modes*. California: University of Southern California dissertation.

Biber, Douglas. 1988. *Variation across Speech and Writing*. Cambridge: Cambridge University Press.

Biber, Douglas. A typology of texts. *Linguistics* 27: 3–43.
Biber, Douglas. 2006. *University Language: A Corpus-based Study of Spoken and Written Registers*. Amsterdam: John Benjamins.

Biber, Douglas. 2019. Multidimensional Analysis: A historical synopsis. In Tony Berber Sardinha and Marcia Veirano Pinto eds. *Multi-Dimensional Analysis: Research Methods and Current Issues*. London: Bloomsbury Academic, 11–26.

Biber, Douglas and Jesse Egbert. 2018. *Register Variation Online*. Cambridge: Cambridge University Press.

Biber, Douglas and Bethany Gray. 2013. Discourse characteristics of writing and speaking task types on the TOEFL IBT test: A lexico-grammatical analysis. ETS Research Report Series 2013/1. https://doi.org/10.1002/j.2333-8504.2013.tb02311.x.

Biber, Douglas, Stig Johansson, Geoffrey Leech, Susan Conrad and Edward Finegan. 1999. *The Longman Grammar of Spoken and Written English*. Harlow: Longman.

Bochynska, Agata, Liam Keeble, Caitlin Halfacre, Joseph V. Casillas, Irys-Amélie Champagne, Kaidi Chen, Melanie Röthlisberger, Erin M. Buchanan and Timo B. Roettger. 2023. Reproducible research practices and transparency across linguistics. *Glossa Psycholinguistics* 2/1. https://doi.org/10.5070/G6011239.

Brezina, Vaclav. 2018. *Statistics in Corpus Linguistics: A Practical Guide*. Cambridge: Cambridge University Press.

Brezina, Vaclav, Abi Hawtin and Tony McEnery. 2021. The written British National Corpus 2014 – design and comparability. Text & Talk 41/5–6: 595–615.

Davies, Mark. 1990. *Corpus of Contemporary American English (COCA)*. https://www.english-corpora.org/coca/.

Dixon, Daniel Hobson. 2022. *The Language in Digital Games: Register Variation in Virtual and Real-World Contexts*. Flagstaff: Northern Arizona University dissertation.

Egbert, Jesse, Tove Larsson and Douglas Biber. 2020. *Doing Linguistics with a Corpus: Methodological Considerations for the Everyday User*. Cambridge: Cambridge University Press.

Egbert, Jesse and Shelley Staples. 2019. Doing multi-dimensional analysis in SPSS, SAS, and R. In Tony Berber Sardinha and Marcia Veirano Pinto eds. *Multi-Dimensional Analysis: Research Methods and Current Issues*. London: Bloomsbury Academic, 125–144.

Gewin, Virginia. 2016. Data sharing: An open mind on open data. *Nature* 529/7584: 117–119.

Goulart, Larissa. 2022. Communicative Text Types in University Writing. Flagstaff: Northern Arizona University dissertation.

Goulart, Larissa and Margaret Wood. 2021. Methodological synthesis of research using multi-dimensional analysis. *Journal of Research Design and Statistics in Linguistics and Communication Science* 6/2: 107–137.

Gray, Bethany. 2015. Linguistic Variation in Research Articles: *When Discipline Tells only Part of the Story*. Amsterdam: John Benjamins.

Gray, Bethany. 2019. Tagging and counting linguistic features for multi-dimensional analysis. In Tony Berber Sardinha and Marcia Veirano Pinto eds. *Multi-Dimensional Analysis: Research Methods and Current Issues*. London: Bloomsbury Academic, 43–66.

Gray, Bethany and Douglas Biber. 2013. Lexical frames in academic prose and conversation. *International Journal of Corpus Linguistics* 18/1: 109–135.

Hidalgo, Bertha and Melody Goodman. 2013. Multivariate or multivariable regression? *American Journal of Public Health* 103/1: 39–40.
In'nami, Yo, Atsushi Mizumoto, Luke Plonsky and Rie Koizumi. 2022. Promoting computationally reproducible research in applied linguistics: Recommended practices and considerations. *Research Methods in Applied Linguistics* 1/3. 100030. https://doi.org/10.1016/j.rmal.2022.100030.

John, Leslie K., George Loewenstein and Drazen Prelec. 2012. Measuring the prevalence of questionable research practices with incentives for truth telling. *Psychological Science* 23/5: 524–532

Le Foll, Elen. 2021. *Introducing the Multi-Feature Tagger of English (MFTE)*. Perl. Osnabrück University. https://github.com/elenlefoll/MultiFeatureTaggerEnglish.

Le Foll, Elen. 2024. *Textbook English: A Multi-Dimensional Approach*. Studies in Corpus Linguistics 116. Amsterdam: John Benjamins.

Le Foll, Elen and Muhammad Shakir. 2023. *Introducing a New Open-Source Corpus-Linguistic Tool: The Multi-Feature Tagger of English (MFTE)*. Paper presented at the 44th International Computer Archive of Modern and Medieval English Conference. NWU Vanderbijlpark: South Africa.

Leech, Geoffrey, Roger Garside and Michael Bryant. 1994. CLAWS4: The tagging of the British National Corpus. In *Proceedings of the 15th conference on Computational Linguistics*. Kyoto: Association for Computational Linguistics, 622–628.

Lindquist, Hans. 2009. *Corpus Linguistics and the Description of English (Edinburgh Textbooks on the English Language – Advanced)*. Edinburgh: Edinburgh University Press.

Manning, Christopher D. 2011. Part-of-speech tagging from 97% to 100%: Is it time for some linguistics? In Alexander F. Gelbukh ed. *Computational Linguistics and Intelligent Text Processing*. Berlin: Springer, 171–189.

McEnery, Tony, Richard Xiao and Yukio Tono. 2006. *Corpus-Based Language Studies: An Advanced Resource Book*. London: Taylor & Francis.

McManus, Kevin. 2024. Replication and open science in applied linguistics research. In Luke Plonsky ed. *Open Science in Applied Linguistics*. Applied Linguistic Press, 148–165.

Nini, Andrea. 2014. *Multidimensional Analysis Tagger (MAT)*. http://sites.google.com/site/multidimensionaltagger.

Nini, Andrea. 2019. The multi-dimensional analysis tagger. In Tony Berber Sardinha and Marcia Veirano Pinto eds. *Multi-Dimensional Analysis: Research Methods and Current Issues*. New York: Bloomsbury, 67–96.

Pashler, Harold and Eric-Jan Wagenmakers. 2012. Introduction to the special section on replicability in psychological science: A crisis of confidence? *Perspectives on Psychological Science* 7/6: 528–530.

Picoral, Adriana, Shelley Staples and Randi Reppen. 2021. Automated annotation of learner English: An evaluation of software tools. *International Journal of Learner Corpus Research* 7/1: 17–52.

Porte, Graeme and Kevin McManus. 2018. *Doing Replication Research in Applied Linguistics*. Milton Park: Routledge.

Qi, Peng, Yuhao Zhang, Yuhui Zhang, Jason Bolton and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. *arXiv*. https://doi.org/10.48550/arXiv.2003.07082.

Rayson, Paul and Roger Garside. 1998. The CLAWS web tagger. *ICAME Journal* 22: 121–123.

Sinclair, John McH., Gwyneth Fox, Stephen Bullon, Ramesh Krishnamurthy, Elisabeth Manning and John Todd eds. 1990. *Collins Cobuild English grammar: Helping learners with real English*. Glasgow: Harper Collins.
Sönning, Lukas and Valentin Werner. 2021. The replication crisis, scientific revolutions, and linguistics. *Linguistics* 59/5: 1179–1206.

Toutanova, Kristina, Dan Klein, Christopher D. Manning and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Marti Hearst and Mari Ostendorf eds. *Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology*. Edmonton: Association for Computational Linguistics, 173–180.

Wallis, Sean. 2020. *Statistics in Corpus Linguistics Research: A New Approach*. London: Routledge.

Wieling, Martijn, Josine Rawee and Gertjan van Noord. 2018. Reproducibility in computational linguistics: Are we willing to share? *Computational Linguistics* 44/4: 641–649.

Wood, Margaret. 2023. Communicative Function and Linguistic Variation in State Statutory Law. Flagstaff: Northern Arizona University dissertation.

### *Corresponding author*

Elen Le Foll

University of Cologne

Department of Romance Studies

Universitätsstraße 22

50937 Cologne

Germany

E-mail: elefoll@uni-koeln.de

received: December 2023
accepted: September 2024